Speaker
Description
At European XFEL, scientific data is managed in a hierarchical storage system that combines high-performance storage (GPFS), a dCache mass storage system, and a tape archive. In this architecture, dCache serves as an intermediate layer between GPFS and tape systems, enabling seamless access to raw data from the Maxwell compute cluster throughout the custody period defined by policy. As data volumes generated by multiple instruments continue to grow rapidly, the pressure on the dCache limited storage capacity increases, making large-scale data management and storage planning more challenging. This situation highlights the need for a dedicated analytical tool capable of monitoring and analyzing data access patterns to distinguish between frequently and rarely accessed datasets, and supporting decisions to free up space efficiently for better storage utilization.
To address this need, we developed a data management analytics layer that leverages filesystem events to capture access patterns and usage statistics of raw data stored in dCache. The approach operates through a multi-stage pipeline: dCache billing records are first cleaned and normalized, then processed through ETL-based modeling workflows to derive meaningful metrics, which are subsequently persisted in an InfluxDB time-series data lake. On top of this stack, purpose-built Grafana dashboards provide intuitive monitoring and analysis tailored to specific operational use cases.
This solution provides detailed visibility into data-access patterns across instruments, proposals, and runs, allowing read frequencies to be tracked with fine granularity. Overall, the solution strengthens the data management strategy by fostering transparency and supporting storage allocation decisions with concrete evidence. Looking ahead, the framework is designed to extend to GPFS storage, with the goal of delivering a unified view of access patterns in raw and processed datasets across multiple levels of the storage hierarchy.
