Speaker
Description
Effective data management infrastructure is essential to keep research data accessible, efficiently processed, and reusable.
This contribution presents the design and implementation data management solution adopted for European XFEL,
which delivers advanced data services through a four layer storage architecture, each addressing specific data handling challenges.
The Online storage layer serves as a high speed cache, managing extreme data rates of up to 15 GB/s per instrument during experiments.
The High-Performance Storage layer supports both near real time data processing and post experiment analysis.
These layers are connected via a 4.4 km, 1 Tb/s InfiniBand link between the European XFEL experiment hall and the DESY computing centre, enabling fast data transfer.
The Mass Storage layer extends capacity for mid term access and in-depth analysis, while the Tape Archive ensures secure long term preservation with a minimum retention period of 10 years.
Together with connected computing clusters, the infrastructure supports processing of up to 2 PB of data per day, demonstrating its scalability and reliability.
