Speaker
Description
Online analysis refers to data analysis pipelines based on processing of live data streams in near-real-time during an experiment. Such analysis provides rapid feedback, helping in understanding outcomes and optimizing steering of the experiment. At European XFEL, the high data rates generated by MHz detectors make live data processing challenging in terms of both computational intensity as well as network transfer. To tackle these challenges, we must utilize an edge computing approach - moving data processing and reduction upstream closer to the source rather than leaving processing of full data to downstream external monolithic analysis software.
Existing pipelines developed for real-time detector data correction provide an ideal integration point for upstream analysis processing. These correction pipelines run within the Karabo control system and are deployed on dedicated compute nodes which are connected via fast InfiniBand fabric and typically have several high-performance GPUs available.
To exploit excess compute capacity left over after data corrections, we provide a number of programmable integration points within these pipelines. These allow for custom kernels for processing the data as well as for making data reduction decisions on the fly. This opens the door for comprehensive online analysis on the full data streams; the results can be visualized within Karabo or streamed to external analysis and visualization suites.
Karabo integration is an additional benefit of these integration points: Moving analysis parameters and (optionally) visualization inside the control system eases the overhead associated with deployment, setup, and configuration - and allows for future integration with other development projects such as automation and optimization.
We present an overview of the architecture described above along with examples of analysis pipelines already developed and used for various experimental techniques.
