Speaker
Description
Scientific datasets rarely stand alone. Raw data, processed derivatives, and experimental metadata are deeply related, but most data services store them as if they aren't. At the Advanced Light Source, we've been working on closing that gap.
We use Tiled, from the Bluesky project, as our primary catalog and data service. As we built applications on top of it, embedding URLs in metadata fields proved too brittle — links are invisible unless you already know they're there, and per-application link schemas don't support scalable search.
We built splash_links to address this: a data service that stores Entities and Links, forming a queryable persisted graph over our data holdings. Entities hold references to Tiled nodes; Links relate them to each other.
An interesting engineering question turned out to be what to build it on. We'll walk through our evaluation of relational databases, native graph databases, and Triplestore databases — including RDF serialization and SPARQL-based semantic queries — and discuss the trade-offs that informed our final choice.
