Speaker
Description
I will describe our system for fully real-time processing of X-ray serial crystallography data streamed directly from the detector, with no intermediate disk storage. Based on the CrystFEL software [1] in combination with the ASAP::O high-performance data framework developed at DESY [2], the system is capable of indexing and integrating more than 1000 frames per second using only a single compute node. It has been deployed at the synchrotron light source PETRA III, where it has been reliably used in a series of user experiments [3]. By finding and removing the largest performance bottlenecks in the software and its environment, we achieved a speed-up of around 50 times since the beginning of the project. Further speedups have since been made by measures such as addressing lock contention and reducing memory allocations.
A prerequisite for real-time processing is that the experimental geometry is accurately known before the experiment. The methods so far employed for determining this geometry have been too slow to run very frequently, or have had bias problems. However, an algorithm in particle physics exists to solve an analogous problem with much higher performance [4]. This algorithm, named Millepede, has been successfully transferred from particle physics to X-ray crystallography [5]. Even when fitting across tens or hundreds of thousands of diffraction patterns, the Millepede method calculates an updated detector model in a negligible amount of time, allowing frequent calibration updates to be a part of our high-speed data processing system.
[1] T. A. White, R. A. Kirian, A. V. Martin, A. Aquila et al. J. Appl. Cryst. 45 (2012) p335.
[2] https://asapo.pages.desy.de/asapo/
[3] T. White, T. Schoof, S. Yakubov, A. Tolstikova, et al. IUCrJ 12 (2025) p97.
[4] V. Blobel, Nuclear Instruments and Methods in Physics Research A 566 (2006) p5.
[5] T. A. White, J. Appl. Cryst. 59 (2026) p594.
