Speaker
Description
The European XFEL is a megahertz repetition-rate facility producing extremely bright and coherent pulses of femtosecond duration. With its AGIPD, LPD and DSSC detectors specifically built to operate at these repetition rates, the amount of data generated in the context of user experiments can exceed hundreds of gigabits per second, resulting in tens of petabytes stored every year. These rates and volumes pose significant challenges both for facilities and users thereof. In fact, if unaddressed, extraction and interpretation of scientific content is hindered, and investment and operational costs quickly become unsustainable.
These challenges are addressed by the facility on both a policy as well as a technical level. A new Scientific Data Policy has been adopted to clarify the individual responsibilities in data retention and curation. Data reduction becomes a required and early step in the data lifecycle of experiments. Here, the facility aids its users by developing data reduction pipelines, tools and by providing dedicated support.
In this work, we present the adopted data reduction strategy and its integration into the existing facility systems and workflows. These range from end-to-end solutions for established techniques to extendable frameworks in case of unique requirements. Depending on the use case, reduction may happen for streamed data before hitting the disk or after comprehensive analysis.
