Speaker
Description
Benefitting from being a greenfield site, ESS uses NeXus exclusively for data capture. Files are templated using a JSON schema, and populated using data consumed from a Kafka instance via bespoke software 'kafka-to-nexus'. This system provides a common, repeatable, and parallel system for data capture and retention.
Although no spallation neutrons have yet been produced on site, the data system has been tested extensively using randomly-generated neutron events and component values that data analysis and instrument teams can use already to work on tools using the data structure that will match the files produced at beam lines from day 1. This continuous data system runs 24/7, and has already uncovered multiple issues that would have affected data capture, but have instead been fixed without production downtime.
This talk will focus on the choices made in this filewriting system, the advantages of such a pipeline, problems seen when putting it into practice, and future work developing the system's resilience.
