The Advanced Photon Source (APS) at Argonne National Laboratory (ANL) has completed a major upgrade that included replacement of the storage ring delivering greatly increased brightness, coherence, and high-energy x-ray capabilities, enabling transformative new scientific opportunities across the APS experimental program. These advances, together with new state-of-the-art high-bandwidth commercial detectors and enhanced beamline instrumentation, are changing how experiments are performed and are creating new requirements for networking, controls and data acquisition, automation, computing, workflows, data reduction and analysis tools, including AI approaches, and data management.

The APS has continued to make significant progress developing and deploying key elements of its scientific computing strategy. These advances include upgrades to networking infrastructure, deployment of modern experiment control software at beamline instruments, expanded capabilities and use of common data management and workflow tools and science portals, use of Argonne Leadership Computing Facility supercomputers for large-scale and on-demand data processing and analysis, development of high-speed and highly parallel data processing and analysis software, and the application of novel mathematical and AI methods to challenging data reduction and analysis problems. Looking forward, the APS must continue evolving these capabilities into reliable, operations-ready infrastructure that can support the full breadth of the upgraded facility. This includes strengthening automation, workflow orchestration, monitoring, portability across computing resources, and collaboration with other light sources, experimental facilities, large-scale computing facilities, and the APS user community.

*Work supported by U.S. Department of Energy, Office of Science, under Contract No. DE-AC02-06CH11357.

The European XFEL, a world-leading X-ray Free Electron Laser situated in the Hamburg-area, Germany, offers a unique 4.5 MHz burst pulse structure that enables users from across the globe to conduct scientific measurements during typically one-week-long beam times. These measurements are supported by facility staff throughout, and each of the 7 instruments can be reconfigured significantly for each beam time.

In this contribution, we provide an overview of the services and infrastructure provided by the European XFEL Data Department to facilitate and support the scientific operation of the facility. These services include the regular ingestion, reduction, processing, and archival of data volumes exceeding 1 PB per week, at peak rates of 20, and soon 40 GB per second, online processing and metadata extraction, experiment automation and steering, and the provision of bespoke X-ray imaging detectors capable of resolving the facility’s MHz burst rate. The EuXFEL Data Operation Center serves as a central support entity, providing support for these services 24/7 during the facility’s operation.

We also discuss future prospects for higher data rates, advanced automation and data reduction techniques tailored to specific experimental techniques, and the integration of agentic AI to enhance the operation and support of the facility.

Modern detectors typically consist of multiple modules to achieve large areas and generate increasingly high data rates. Synchronization across these modules, together with efficient real-time data handling, is therefore essential for enabling responsive and data-driven experiments.

ASAP::O, a high-performance streaming framework designed for low-latency data transport and scalable processing, has been developed and deployed at DESY. It decouples data acquisition from analysis via a producer–consumer architecture, enabling parallel data streaming, online processing, and distributed access. Because of its built-in synchronization mechanisms, ASAP::O provides unified and consistent access to data originating from multi-module detector systems, as well as from multiple detectors.

At DESY, ASAP::O is used for live detector data streaming to analysis nodes, integration with automated processing pipelines, and implementation of feedback-driven experimental workflows. These capabilities enable near real-time monitoring and support adaptive experiment control under high-throughput conditions.

Our results demonstrate that ASAP::O provides a robust and scalable backbone for next-generation data acquisition, enabling efficient workflows and supporting the transition toward autonomous beamline operation in preparation for the PETRA IV upgrade.

Abstract

Ultra-fast time-resolved synchrotron computed tomography (CT) can acquire up to 1000 tomographs per second, enabling the study of highly dynamic processes but producing data streams that challenge transmission, reconstruction, analysis, and storage. To support smart experiments and rapid feedback at the TOMCAT beamlines of the Swiss Light Source, we developed a live reconstruction pipeline that sustains detector-rate processing in real time.

The system integrates high-speed acquisition, remote direct memory access (RDMA) streaming, and multi-GPU reconstruction. On the data acquisition node, 12-bit GigaFRoST camera output is converted to 16-bit format, grouped into 3D chunks, and stored in shared-memory buffers. Metadata are exchanged with ZeroMQ, while image payloads are transferred with Unified Communication X (UCX). On the reconstruction node, data are buffered in host memory, partitioned into chunks, and processed on four NVIDIA H100 GPUs. Host-to-device and device-to-host transfers are overlapped with computation through CUDA streams, and the reconstruction workflow is implemented on top of Tomocupy.

The pipeline achieves up to 15 GB/s reconstruction throughput in 16-bit format, matching the full detector streaming rate of 7.7 GB/s at 12-bit output. This enables continuous reconstruction with sufficiently low latency for downstream analysis and online decision-making. The resulting capability provides the basis for adaptive data acquisition strategies that can suppress redundant data, preserve critical events, and feed reconstructed information back to the beamline control system. These results show that live reconstruction is a practical enabling technology for smart ultra-fast synchrotron CT and is well aligned with the performance requirements of the Swiss Light Source 2.0 upgrade.

Acknowledgements

This project is funded by the Swiss Data Science Center (SDSC), No. C23-02L.

Event Formation Unit - a unified data acquisition pipeline at ESS

M.J. Christensen, T. Bukovics, M. Ipsen, M. Christiansen

The European Spallation Source (ESS) will initially consist of 15 scientific
instruments. These cover diverse scientific scopes—such as neutron spectroscopy,
imaging, diffraction, and reflectometry—each requiring completely different
instrument designs and specialized detector technologies.

The Experiment Control and Data Collection (ECDC) group has developed a unified
data acquisition pipeline to support this heterogeneous system. Within our
readout system, raw analog signals are first processed by digitizers alongside
precise timestamping. The core computational engine of our pipeline is the Event
Formation Unit (EFU). The EFU consists of base software and shared algorithmic
libraries common to all instruments, as well as detector-specific code. This
approach significantly reduces maintenance overhead, ensures consistent logging
and reporting, and enables centralized monitoring.

The EFU is a high-performance C++ application whose primary esponsibilities are
to receive the digitized readout data over UDP, extract and reconstruct distinct
"neutron events", and stream these to a distributed network storage (Apache
Kafka) using Google Flatbuffers for serialisation. These reconstructed events
are fundamentally defined by a relative timestamp and a pixel ID representing
the detector position. Kafka serves as a robust buffer, distributing the live
data to downstream clients, including instrument control systems, live
visualization tools, and the file writer mechanism that produces the final NeXus
scientific data files. This empowers researchers to fully leverage the data
capabilities of ESS.

The Karabo data acquisition (DAQ) at the European XFEL is capable of data reduction based on external information like beam conditions and shutter positions. To cope with the storage challenges imposed by the next generation of the AGIPD detector, i.e. almost 1 GB/s for each of its 56 modules, it will be necessary to base the decision to store or reject data frames on near-realtime data analysis like hit finding for the SFX experiments this detector will be used for. This requires to stream data at the full rate from the C++-based DAQ to the Python-based calibration and data analysis algorithms. The infiniband network infrastructure at the European XFEL offers the required bandwidth, but the TCP/IP-based Karabo data pipelines do not cope reliably enough with the required data rate. To overcome this shortcoming, Karabo data exchange options have been extended with Remote Direct Memory Access (RDMA).

This contribution presents design choices, lessons learned, achieved data
rates, and future development directions of the RDMA communication between the Karabo DAQ, calibration and online analysis.

The MAIQMag project aims to utilize AI trained on multiple modalities to provide faster and more unique understanding of Hamiltonian parameters for Quantum Magnets. These modalities include, single crystal and powder inelastic neutron scattering data, Resonance Inelastic Xray scattering data and Magnetization. The neutron results are from Oak Ridge National Laboratory and the X-ray results are from Stanford Linear Accelerator, Argonne National Laboratory and Brookhaven National Laboratory. Furthermore simulations using the Su(n)ny.jl and EDRIX packages provide the necessary parameterized data for training. For this data to be accessed for training and interrogating an efficient data broker is needed. Tiled has proven to be an efficient solution for our use case. It provides a python scripting interface to allow quick searching of the metadata, and it allows extracting only the portions of multi-dimensional data that is needed for the processing. A hierarchical schema has been deployed and tests have been run as more data is ingested and have demonstrated query responses in a few ms. Currently there are 165,823 artifacts and 47,637 entities in the data base and it is ever growing.

Beamline scientists spend a disproportionate amount of their time answering the same
questions and running the same routine procedures for users. "Where is the sample
stage right now?", "how do I switch to flyscan mode?", "what does this PV do?" All of
this work is essential, but rarely the best use of an expert who is also responsible for
the experiment. BAIT (Bluesky AI Tools) offloads this layer to a small set of cooperating
LLM agents designed with the goal to leverage the Bluesky ecosystem.

BAIT is a Python package that builds on work by the BCDA (Beamline Controls & Data
Acquisition) group at the APS (Advanced Photon Source), in particular the BITS
(Bluesky Instrument Template Structure) package. BITS has enabled uniform
deployments across beamlines through a standardized layout, which BAIT leverages so
the same agents can operate across different beamline installations with minimal
beamline configuration.

The implementation, currently being deployed across APS beamlines, exposes several
agents behind a single chat interface: a router that classifies incoming questions, a
documentation agent that performs retrieval augmented search over beamline manuals
and IOC notes, a device agent that reads live Ophyd values and allows for writes with
human confirmation before any hardware moves, an IOC level diagnostics agent, and a
recipe execution agent that turns scientist written procedures (e.g. "align the beam")
into Bluesky plans.

The talk will cover the architecture of the package, how it leverages BITS and the wider
Bluesky stack, the safety model for LLM driven control, and lessons learned from
deploying and operating BAIT at working beamlines.

Bluesky and Tiled together provide an ideal platform for AI agents to interact with user facility data. Bluesky captures the context around measurements and the relationships between data streams in an experiment. Tiled empowers scientists and their agents to search all that context and selectively access data of interest---whether the data came from Bluesky or another system. When an LLM is given access to Tiled, it has been shown to support text-based inquiries about the data.

Tiled is more than a data catalog: it enables clients to remotely slice into data and optionally transcode the format into one suitable for the application. Streaming capabilities, added within the last year, can push data to interested subscribers before it touches disk. Tiled also recently added the ability to send notifications, which can be used to launch workflows or synchronize updates with external systems.

Agents can utilize Tiled's features fluently because Tiled uses web APIs that are well represented in the models' training data (OpenAPI REST, WebSockets, Web Hooks). Tiled's security and access control features, which have been externally vetted, make it possible to grant agents access that is appropriately sandboxed with fine-grained permissions.

Bluesky and Tiled are open source projects governed by an international, multi-facility collaboration. Tiled alone has 60 code contributors representing over a dozen institutions.

Large synchrotron datasets present a growing challenge: data volumes increasingly outpace human capacity for systematic exploration and discovery. We present an agentic AI framework that couples LangGraph-based multi-agent orchestration with Tiled/bluesky for exploring and searching data. Specialized agents autonomously browse Tiled node hierarchies, retrieve datasets and metadata, apply analysis routines, and synthesize findings across large collections of experimental runs — enabling natural-language-driven data exploration without requiring users to write bespoke analysis scripts for each query.
Beyond dataset navigation for light source, we deploy deep research agents to build and enrich scientific knowledge graphs. These agents traverse literature, experimental metadata, and prior analysis results to construct structured representations of materials knowledge, linking experimental observables to processing conditions and published findings. The combined system enables queries that span both raw data and accumulated scientific context.
We demonstrate the approach on SAXS/GISAXS datasets at the Advanced Light Source, where agents identify structural trends, flag anomalous scans, and surface candidate datasets contextualized against the broader materials literature.
This work raises practical questions around trust, reproducibility, and human oversight that the facility software community will need to address collectively.

The high brilliance of modern X-ray and neutron sources enables temporally and spatially resolved measurements with high acquisition rates, resulting in vast amounts of data. Given the achieved experimental performance, the data analysis becomes a substantial bottleneck on the way to new discoveries. This applies, inter alia, to surface-sensitive scattering techniques, such as grazing-incidence diffraction (GID) / grazing-incidence wide-angle scattering (GIWAXS). Moreover, the analysis of 2D GID/GIWAXS data is inherently challenging due to the high data dimensionality and various parasitic contributions. To address these challenges, we develop an mlgid data analysis pipeline [1], which divides the task into different stages. After the initial data reduction, an object detection ML model detects Bragg peaks in GID patterns. After additional peak refinement, the extracted peak parameters can be used for further analysis, e.g., probabilistic ML-based matching to known crystalline structures or determination of unit cell parameters. The pipeline achieves performance comparable with the typical data acquisition rates and was also shown capable of on-the-fly analysis during in situ GID experiments. These findings pave the way for more efficient experiments, allowing the experimental plan to be adjusted based on analysis results obtained directly during the experiment. They ultimately have the potential for automated surface-scattering experiments.

We acknowledge funding via DAPHNE4NFDI (DFG grant no. 460248799) and OSCARS (Horizon Europe grant No. 101129751).

1. D. Lapkin et al., (2026, in preparation)

At light sources like the European X-Ray Free-Electron Laser (EuXFEL), the high pulse repetition rates generate vast quantities of data that are challenging to track and analyze in real-time. Traditionally, users have relied on manual, error-prone spreadsheets to maintain run tables and record metadata. This presentation introduces the Data And Metadata iNspection Interactive Thing (DAMNIT), a specialized tool designed to automate the creation of experiment overviews. By executing user-defined Python "context files," DAMNIT extracts scientific metadata and computes primary results as soon as experimental data is available. Integrated with facility-wide services for event triggers and distributed processing, DAMNIT provides a unified interface via both a Qt-based desktop client and a modern web frontend, that enables researchers to monitor, plot, and programmatically access their results through a dedicated Python API.

The presentation will focus on concrete use cases demonstrating how DAMNIT has been deployed across diverse scientific instruments at EuXFEL to streamline complex workflows.

Deploying GPU-accelerated AI workflows from photon-source beamlines to remote HPC facilities requires navigating incompatible job schedulers, GPU architectures, CUDA versions, and container runtimes. Users and beamline staff must maintain separate deployment procedures for each facility, limiting portability and slowing adoption of new methods.

We present a facility-agnostic pipeline framework and dashboard, developed at SSRL, that enables single-action submission of AI workflows to heterogeneous compute systems. The framework currently orchestrates jobs across five compute platforms spanning national laboratory HPC, an on-premises Kubernetes cluster, and commercial cloud, unified through a common REST API that abstracts away differences in job schedulers and GPU architectures. Globus handles data transfer between the beamline and remote compute. We have demonstrated full container portability between the Kubernetes and cloud targets, with containerization of the remaining HPC targets underway. Beamline staff select a target facility and launch a workflow without modifying the underlying job scripts.

We demonstrate the framework with a GPU-accelerated tomography pipeline that performs reconstruction and zero-shot 3D segmentation using SAM3, a vision foundation model, processing 38 GB of raw synchrotron data in under 30 minutes. We discuss container build strategies for facilities without system CUDA installations, dtype portability challenges in foundation model inference, and our approach to building a catalog of reusable, facility-portable AI workflows for photon science.

The Extensible Workflow System (Ewoks) is a meta-workflow system that provides a unified interface for defining and executing scientific workflows with built-in data provenance, supporting FAIR data practices. It targets use cases at large-scale facilities, from manual and automated data processing to the orchestration of fully automated experiments.

Ewoks leverages existing workflow technologies adapted to different contexts, including interactive desktop use, online integration via message brokers, usage through facility data portals, execution on HPC clusters, and conditional workflows for expert systems. By integrating these approaches, it enables unified, reproducible, scalable, and interoperable data processing at facilities such as the ESRF where it is operational at 38 beamlines.

Tofu image processing toolkit has been used with a great success for nearly a decade to reconstruct microCT data acquired at imaging beamlines of the KIT Light Source in Karlsruhe, Canadian Light Source, and P23 beamline of DESY as well as laminography data acquired at ESRF ID19 and neutron-based CT scans done at Institut Laue-Langevin. The architecture, functionality, and performance of tofu have been described in J. of Synchrotron Radiation by T. Faragó, S. Gasilov, et al. in vol. 29, pp. 916-927, 2022. Here, we present in greater detail the features of tofu ez - a PyQt-based graphical user interface for generation of ufo-launch/tofu data reconstruction pipelines. Tofu ez permits one to interactively create image processing pipelines composed of up to 10 steps that encompass all essential operations encountered when dealing with microCT data acquired at synchrotrons, such as the removal of hot pixels, suppression of artifacts stemming from scintillator defects, phase retrieval, ring removal, and denoising. All of that is applicable to the half acquisition mode (360° tomography with the off-centered rotation axis to effectively double the horizontally field of view) combined with multiple vertical scans per sample. In such data sets, the horizontal and vertical overlaps between CT projections can be estimated automatically so that one obtains fully stitched data cubes in the end. Tofu ez produces a formatted bash script for as many CT data sets as it finds in the input directory. These scripts can be executed locally or submitted to a cluster as a slurm job. Recently, we have installed tofu on Maxwell cluster and added a parser of h5 files to enable reconstruction of data acquired at Hereon imaging stations at P05 and P07 beamlines.

SIRIUS is a 4th generation synchrotron facility operated by LNLS/CNPEM, with multiple beamlines currently in operation and others under commissioning. As the number and complexity of beamlines grow, there is an increasing need for scalable, maintainable, and reusable software solutions.

To address these challenges, SIRIUS has been evolving its high-level control and orchestration ecosystem around modular services and a client-server architecture. A central component of this effort is the SIRIUS Ophyd and Bluesky Utilities (sophys) ecosystem, including sophys-server, sophys-clients (web, desktop and CLI) and beamline specific integration, which provide a unified way to execute experiments, manage queues, and interact with the experiment. It integrates data acquisition routines based on Bluesky, enabling a consistent interface across different beamlines. This approach has enabled code reuse, reduced maintenance overhead, and led to faster delivery of new features across multiple beamlines.

The architecture incorporates extensible authentication and authorization integrated with institutional systems. Additional applications such as sophys-CLI expand usability for different user profiles.

Recent developments include web-based real-time visualization of control system process variables and support for asynchronous acquisition modes such as fly-scanning. Ongoing work also explores solutions for more complex experimental scenarios, including multiple concurrent execution queues for parallel and in situ experiments.

Looking forward, we discuss the adoption of Tiled as a potential layer for structured data access and experiment planning, and its integration with workflow orchestration tools such as Prefect. These directions aim to further improve reproducibility, automation and interoperability across beamlines and to provide the flexibility needed to reach a better user experience and more efficient use of beamtime.

The ROCK-IT project has developed an operando catalysis automated and remote-operable experimental workflow at DESY's Petra-III P65 beamline. We will present here our control system based on tandem operation of two Bluesky Queueserver workers orchestrating purpose-developed Tango device servers via Ophyd-Async. Experimental results are serialized and streamed to an ASEP::O analysis pipeline for live processing and data visualization. Metadata is collated by Bluesky and ingested into a SciCat data catalogue. Remote operation and raw data visualization is accomplished by authenticated access to an ESRF Daiquiri browser-based graphical interface with an added compatibility layer to interface with the Bluesky Queueserver.

At the previous NOBUGS conference, the Sardana community presented the outcomes of a collaborative workshop on continuous scans (SOLARIS 2023) and outlined a midterm roadmap to address emerging experimental requirements. Since then, it has been refined through annual Sardana workshops (MAX IV 2025, ALBA 2026), reinforcing a collaborative, incremental process driven by real use cases across facilities. This contribution reports on the application of these developments in real environments.

Several previously identified challenges—such as synchronized shutter control and multi-technique experiments combining detectors operating at different rates—have seen significant progress, with some already deployed in production. In particular, the integration of Blissdata into Sardana, together with multiple synchronization mechanisms and native shutter control, is being introduced in beamlines at ALBA and MAX IV to enable more advanced data acquisition and composition workflows. These developments also foster collaboration between facilities, showing how Blissdata, originally developed within the Bliss framework at ESRF, enables sharing client tools for data writing, processing, and analysis across facilities.

In parallel, improvements include enhancements to mesh scans, evolution of the scan API, a new experiment configuration GUI, and native support for complex trajectories control, moving towards more flexible and interoperable workflows.

Elettra, the Italian synchrotron facility in Trieste, is undergoing a major upgrade that will result in a substantial increase in X-ray brilliance and coherence. To effectively exploit such improvement, each beamline requires cutting-edge infrastructure and control instruments to manage a vast ecosystem of diverse tasks, ranging from the photon transport to data collection and processing. The Software Per Esperimenti (SPE) team has designed and developed a set of software and hardware solutions that will be used to face the specific needs of every Elettra 2.0 beamline. The main goal is standardization: each tool has been designed to be flexible and reusable. This paper presents a review of a set of these solutions: GeCo, the beamline equipment protection system; BACS, the beamline personnel protection system; and DonkiOrchestra, the data collection engine. Furthermore, the software frameworks chosen for instrument control and graphical interfaces are presented and discussed.

AI-assisted coding is gaining significant traction at the EuXFEL. Among all LLM use cases facilitated by the facility’s enterprise foundation model provider, coding-related tasks have the highest token usage. However, there are still unanswered questions and evolving guidelines regarding AI-assisted and agentic coding. These include determining where we should embrace, tolerate, and avoid AI-generated code, how much we expect AI-coding to become the new norm, and how we can integrate tools that can quickly produce hundreds, if not thousands, of lines of code into code review and testing processes.

In this contribution, we will report on a condensed AI code camp scheduled for June. This intensive, hands-on exercise will involve a small cross-functional team with diverse experience in AI-driven coding. The team will examine, address, and challenge the aforementioned questions. The objective is to further develop a behavior tree-driven graphical sequencer for the facility, which is currently in a prototype stage. The task is to accomplish this within five days, taking advantage of agentic coding opportunities as much as possible.

At SPring-8, scheduling accepted user proposals has traditionally required extensive manual coordination by beamline staff, who must consider user availability, staff availability, beamline operation cycles, and compatibility of experimental setups. To address this challenge, we developed an automated proposal scheduling system based on 0-1 integer programming and deployed it for practical beamline operation.
The scheduler formulates proposal allocation as a combinatorial optimization problem and generates feasible schedule candidates under multiple operational constraints. At BL19B2, the system reduced schedule coordination time from more than one full day of manual work to a few hours, while improving consistency and reducing dependence on individual experience.
We also integrated the scheduler into a broader digital workflow for proposal management. Accepted proposal information, coordination results, and finalized schedules can be managed in a more continuous manner, and linkage with the User Support Database supports downstream registration and sharing of finalized schedules. This integration helps beamline staff manage proposal information more efficiently and provides a basis for future user-facing functions, such as web-based availability input.
This work demonstrates that proposal scheduling at synchrotron facilities can be treated not only as an optimization problem, but also as part of an integrated workflow for scalable and maintainable user experiment support.

The experiment data acquisition and sequencing system BLISS was introduced on the first ESRF beamlines in 2020 and is today, in 2026, driving experiments on all 46 ESRF beamlines. Data acquisition sequences are implemented for more than 100 scientific techniques.
BLISS is programmed in Python, to allow easy sequence programming for scientists and easy integration of scientific software. BLISS offers: Configuration of hardware and experimental set-ups, a generic scanning engine for step based and continuous data acquisition, live data display, frameworks to handle 1D and 2D detectors, spectrometers, monochromators, diffractometers (HKL) and regulation loops. External clients can access BLISS via the available REST-API.
The BLISS data interface separates data acquisition from any client application accessing experimental data. A standardized H5Py like API allows clients a transparent access to on-line or off-line data. All necessary experimental meta data is made available for fast on-line data display and analysis or saving to off-line HDF5 data files. The BLISS data portal is available with SARDANA and BLUE SKY, thus allowing exchangeable scientific client software.
A missing part in the BLISS data interface is the fast on-line access to heavy data (for example 2D and 1D data) produced at very high data rates (kHz). As part of the system infrastructure, we need large, scalable data buffers to implement data reduction and vetoing without file saving. Filtering of relevant data for in-situ experiments or rapid feedback to the running experiment, directly from on-line data, are other use cases.

Silx (https://github.com/silx-kit/silx) is an open-source Python toolkit and tool (silx view) for viewing HDF5/Nexus files used in many sites world-wide.

silx provides not only a NeXus-HDF5 data visualization tool but also a comprehensive collection of Python packages to support the development of data assessment, reduction, and analysis applications at synchrotron radiation facilities worldwide.

This talk will present the current state of silx after a decade of evolution which started in 2016:

• a suite of Qt widgets to browse and visualize multidimensional data, offering a matplotlib and an OpenGL backend.
• OpenCL-based processing for high-performance computing
• Input / output tools for different file formats like NeXus-HDF5

It has been 10 years since silx was awarded for the Best Poster prize at NOBUGS 2016 in Copenhagen. But how far are we from the initial expectation?

During this journey, a project—and its contributors—evolves, adapts, and re-imagines itself. Growth brings lessons and now is the time to look in the rear view mirror: what went well, what didn't, what could be improved, what do we foresee for the next decade?

For over two decades, the dCache project has provided open-source software to meet ever more demanding storage requirements.
More than 80 sites worldwide rely on dCache to provide storage service for LHC experiments, Belle-II, EuXFEL, and many others.
This can be achieved only with a well-established software development process, from a whiteboard, where ideas are created, through development,
packaging, and testing. During this presentation, we will present how dcache developers ensure the software quality, the challenges they met,
and the techniques that were used to solve them.

Instrument scientists and scientific users desire a unified, flexible and accessible interface to control their experiments: one place from which to select the sample, compose and run scripted scan sequences, and view and plot live data with the possibility for multi-user sessions and remote access. The technical staff managing the control system, on the other hand, will want support for a specific underlying control protocol.

NICOS is a cross-platform, integrated control software designed to meet these requirements. It is written in Python with a network-based architecture able to control devices via EPICS, Tango, SECoP or CARESS.

The graphical Qt-based client combines device control, scan orchestration, and plotting of live (or historic) data into a rich, scientist-facing GUI. It can be extended with additional panels such as an electronic logbook or graphical instrument views. Users can create complex scans in the built-in Python-based script editor, validate them through dry-runs and monitor their execution process.

The NICOS server consists of a set of networked daemons, each providing different functionality: from script execution and archiving to electronic logbook integration and condition monitoring (watchdog). These processes communicate via a common cache with each other, which limits the necessary hardware polling.

NICOS has originally been developed at Heinz Maier-Leibnitz Zentrum (MLZ) and is free and open source software. Today, it is actively used and contributed to by MLZ, the European Spallation Source (ESS) and Paul Scherrer Institute (PSI) as well as other facilities worldwide.

In this contribution, the NICOS software, recent developments, and its typical use at ESS instruments will be presented on behalf of the NICOS collaboration and the Experimental Control and Data Collection group at ESS.

Darfix is a GUI software to treat Dark-field X-ray Microscopy (DFXM) data at the ID03 beamline at ESRF.

Since its inception in 2019, Darfix specifications have evolved through successive revisions; a major change was the upgrade of the data acquisition format from EDF to HDF5. As a result, the Darfix codebase began to grow and became overly complex. An incomplete abstraction attempted to handle the two formats, as well as data either fully loaded into RAM or dynamically loaded. This resulted in four distinct use cases and led to an unmanageable amount of conditional code. The first challenge was to reduce this complexity.

On the other hand, as with most ESRF applications, dataset number and size continue to grow each year. Darfix datasets can range from 2 GB to 80 GB and involve computationally expensive operations such as curve fitting across the entire dataset. This led to a second challenge: memory footprint and performance are critical to keeping Darfix attractive for scientists.

How should legacy features be handled? How can one decide whether to refactor, redesign, or remove them?

How can the right balance be achieved between performance, memory usage, and code complexity?

This poster explores these common programming challenges through the case of Darfix software.

RESEDA (Resonance Spin Echo for Diverse Applications) is a spin-echo spectrometer at the MLZ and offers sub-µeV energy resolution and an exceptional dynamic range of ~8 orders of magnitude. One of its operation modes is the MIEZE (Modulation of IntEnsity with Zero Effort) mode, which enables the measurement of the intermediate scattering function, S(Q, τ), in depolarizing sample environments, such as under high magnetic fields. For the efficient reduction of the complex MIEZE dataset, we develop MIEZEPY -- an open-source Python-based software package with a friendly and highly intuitive user interface. In this contribution we will present a step-by-step demonstration of MIEZEPY’s application as well as implementation of the NeXus format as the standard file format for data acquired at the RESEDA instrument.

With the upgrade from SLS to SLS 2.0, a fourth-generation synchrotron, the macromolecular crystallography (MX) beamlines have undergone significant hardware and software enhancements, enabling fully automated data collection workflows. By leveraging the on-the-fly data processing capabilities of Jungfraujoch (Leonarski, F. et al., 2023), together with integrated machine learning and artificial intelligence, automation now spans the entire experimental pipeline. This includes robotic sample exchange, machine learning assisted automated loop centering, dose estimation, diffraction data acquisition, and data processing. A real time diffraction viewer (Jungfraujoch Viewer) was developed for monitoring data collection on the fly. Alongside optimizing established beamline operations, we are exploring new modes of interaction between users, samples, and instrumentation to improve consistency and efficiency.

This presentation will describe the current beamline configuration and software suites (Aare Suite). Uses Beamline and Experiment Control (BEC) (Wakonig, K., et al. 2024) as a foundation, our team updated the graphical user interface (AareGUI) and data acquisition workflow (AareDAQ), developed database (AareDB) and investigated different ways of working with fine tuning YOLO models. In addition, we will outline our roadmap for further incorporating AI into the local contact workflow, e.g. automatic beamline recovery, ideally ensuring more restful nights for our local contacts.

1. F. Leonarski, J. Nan, Z. Matěj, et al., IUCrJ, 10, 729–737 (2023). https://doi.org10.1107/S2052252523008618
2. Wakonig, K., Appel, C., Ashton, A., Augustin, S., Holler, M., Usov, I., Wyzula, J. & Yao, X. (marie) (2024). Proceedings of ICALEPCS2023 https://doi.org/10.18429/JACOW-ICALEPCS2023-MO2AO02.

The National Synchrotron Radiation Centre SOLARIS, a third-generation light source, is the only synchrotron facility in Central-Eastern Europe, located in Poland. The SOLARIS Centre, equipped with seven fully operational beamlines and three under construction, serves as a key research hub for a wide array of scientific disciplines. The Centre requires advanced software tools to support the analysis of experimental data and has developed a specialized application to assist scientists.

Motors Setup is a GUI app dedicated to beamlines where precise setup of motors is crucial for optimal work and conducting experiments.
App offers motors grouping and creating profiles to easily manage different beamlines' operational modes. Configuration is saved in dedicated JSON file. This feature simplifies operators' work due to automatic restoring all motors’ positions, saved in config files, at once.
Other features include creating history for a selected profile and indicating which version should be set as primary - it eliminates the need to create new versions.
GUI has also implemented functionalities to add or edit description for config files (so-called snaps) to make easier snap's identification.
Proposed solution have been implemented in Python-based Tango ecosystem with Taurus and Sardana frameworks.

Pypleem is an advanced data analysis application dedicated for spectroscopic data stored with PEEM and LEEM microscopes. This approach focuses on addressing challenges related to increasing volume and complexity of datasets by providing an environment with wide range of tools for data postprocessing and interpretation. The framework supports multiple experimental modules, including X-ray absorption spectroscopy (XAS) and X-ray magnetic circular dichroism (XMCD), and enables a unified workflow for handling data structures. Additional features such as three-dimensional visualization of magnetic vector fields support the analysis of spatially complex magnetic structures.

Knowing your data is trustworthy brings confidence and peace of mind to the subsequent data analysis step(s). This is reliant on the provision of thorough, well-configured data correction steps. Over the last decade, we have achieved this through implementation of a modular data correction system, which allows a comprehensive data correction graph to be constructed for a given instrument or experiment type. The first implementation in Java in the Data Analysis WorkbeNch (DAWN) has long been producing vast quantities of data on a range of instruments, using a correction graph that is universally applicable to all types of samples. It is now time for a refactored version.

With the involvement of several interested parties, the foundation for the new Python-based implementation has been laid and a functional prototype is available. Features include:
- native support for propagating multiple uncertainty estimates
- units-aware operations
- command-line and web-based APIs
- a full traceability chain can be included in the processed files
- graph output and interfaces are available for visualisation and (future) graphical configuration tools
- minimal dependencies on external libraries

This talk will introduce MoDaCor and encourage collaboration and adoption.

The High Energy Photon Source (HEPS) generates massive amounts of highly heterogeneous experimental data, placing significant pressure on storage, computing, and network resources. Data compression is an effective way to mitigate data growth, but a single compression method cannot meet the diverse requirements in the HEPS context. To address this issue, this work first studies a compression method recommendation framework for different beamline experiments, enabling adaptive selection of compression methods based on data characteristics. Second, for data that are difficult to compress, deep learning techniques are used to further explore the potential of lossless compression. Finally, lossy compression methods for scientific computing scenarios are explored, with different loss quantification designs tailored to different computing tasks. This work provides systematic solutions and key technical support for data compression at HEPS.

Ptychography is a lensless, high-resolution advanced imaging technique widely used at modern synchrotron facilities. The integration of deep learning methods has partially addressed the issues of high computational cost and strict data overlap requirements associated with traditional algorithms. However, existing discriminative learning models still face limitations in reconstruction accuracy and training stability. In this talk, we present PtychoDiffusion, a physics-guided diffusion posterior sampling framework for high-fidelity ptychographic reconstruction. The framework leverages the strengths of diffusion models by learning the denoise process from random noise to target sample. After that, we innovatively incorporate physical principles into the model by embedding classical iterative projection algorithms into the denoising steps, achieving a combination of generative priors and physical constraints, thereby significantly enhancing reconstruction reliability. Numerical experiments using simulated ptychographic datasets demonstrate that PtychoDiffusion delivers higher reconstruction accuracy and more consistent image details under various conditions, with particularly notable advantages in recovering detail structures and handling low-overlap data. Moreover, preliminary validation on experimental ptychography data further suggests the potential of the framework for practical beamline reconstruction workflows. These results indicate that physics-guided diffusion framework offers a principled and effective paradigm for unsupervised ptychography reconstruction and shows promising application potential.

Estimating the latent capacity required to model high-dimensional data is important for representation learning, compression, model design, and downstream control. We formulate latent capacity estimation as a sequential measurement problem over an ordered family of nested representations induced by prefix-masked autoencoders. This turns latent capacity selection from a costly architectural search into a one-dimensional decision variable, allowing effective representational dimensionality to emerge during training within a single shared model.

Beyond reconstruction, the learned latent space provides a compact control space for inverse problems and adaptive optimization. We show that latent representations learned from simulated point spread function data and beamline datasets can be used to control the underlying system efficiently, enabling fast adaptation toward target outputs without expensive search in the original high-dimensional parameter space.

Empirically, adaptive allocation focuses training on the most informative capacity regimes, giving lower reconstruction error, reduced variance, and more stable dimension estimates than fixed schedules under matched budgets. Experiments on synthetic manifolds, structured image datasets, simulated PSF data, and beamline control benchmarks show that effective representational capacity can be identified reliably within a single training run while supporting fast latent-space adaptation for scientific instrument control.

Many existing scientific workflows for segmentation-based data acquisition rely on intensive manual labour of domain experts to label samples as out-of-box Machine Learning models do not generalise well. Different frameworks adopt different data formats, evaluation metrics, and configuration interfaces, resulting in slower experiment setups and need for intermediary steps for result comparison.

We introduce a Pipeline to simplify and accelerate the finetuning and evaluation process to produce custom benchmarks and models for 2D image datasets for instance segmentation. The Pipeline is a unified tool for standardised data handling, experiment configuration, finetuning, inferencing and evaluation, supporting three state-of-the-art segmentation frameworks – Detectron2, Ultralytics YOLO (v8/v11), and the Segment Anything Model (SAM). Each stage of the pipeline can be executed via command line interfaces, with seamless switching between models. Designed with modularity and extensibility, the Pipeline allows integration of new models and datasets without restructuring the entire pipeline. We validate the Pipeline with a case study using a multi-magnification level CryoEM dataset. Overall, this work promotes reproducible benchmarking and enables more transparent and efficient model comparison.

Two-dimensional pixel detectors used at X-ray diffraction beamlines
provide planar cuts through the reciprocal-space representation of the
X-ray scattering function. With the advent of today’s high-brilliance
X-ray sources, the resulting very high signal-to-noise data demand
analysis pipelines that minimize both systematic errors and approximation
artifacts.

We present a new open-source software framework for converting
two-dimensional X-ray scattering images into normalized one- and
two-dimensional representations, $I(Q)$ and $I(Q,\phi)$. The framework
supports the combination of multiple detectors acquired simultaneously
and/or multiple images at different detector setting (e.g., at
different detector-sample distances).

While radial integration is straightforward if the sample scattered
symmetrically, any non-symmetric experiment, which due to polarization
and sample geometry is almost every experiment at these S/N ratio,
failure to account for the scattering symmetry operator when coupled
with an incomplete detector surface due to either masked pixels or
physical detector construction, results in bias data and artifacts.
This code provide a robust way to calculate the symmetry components for each
$Q$-bin and then correctly weight the pixel contributions in the
radial integrals to compensate for the missing non-symmetric pixel units

We demonstrate how this method improves data smoothness and scaling,
and significantly reduces Fourier artifacts in pair distribution
functions (PDFs) derived from such datasets. The software is designed
to be modular and easily embedded into existing data-analysis
environments, and is already integrated into real-time processing
workflows for ultrafast pump–probe experiments
[``Real-Time Data Sorting and Interaction at Ultrafast Pump-Probe
Experiments'' submitted by John Bekx].

Modern beamlines need a flexible way to run everything from simple motor moves to multi-dimensional, hardware-triggered acquisitions - often orchestrated through web-based user interfaces rather than bespoke scripts. At the Australian Synchrotron (ANSTO), we have implemented a Generic Scan library on top of Bluesky, making Bluesky declarative: it specifies what should be measured, not how the scan should be executed, enabling an almost generic web GUI to drive experiments while the framework generates executable Bluesky plans. Scans are described as a hierarchy of nested step levels, each containing one or more positioners and supporting absolute or relative trajectories. Data collection is encapsulated in reusable measurement blocks that standardise detector configuration, triggers, exposure settings, and acquisition modes with multiple stopping conditions. Pre- and post-plan hooks at both step-level and measurement level enable insertion of alignment, calibration, metadata injection, and safe cleanup logic without modifying core scan code. This talk outlines the schema, execution plan generation, and practical beamline use cases.

At the Australian Synchrotron (ANSTO), we see the future of beamline control in increased automation and composable workflows: modular building blocks that can be rearranged as science use cases evolve, and as ML/AI-driven decision points become part of routine experiments. We have adopted Prefect as a workflow engine to orchestrate end-to-end beamline pipelines in a single system, spanning acquisition through to data handling and processing. Prefect flows are used to queue and run Bluesky plans, then trigger downstream tasks such as metadata capture, data aggregation, and automated processing/reduction, with clear state tracking, retries, and beamline staff visibility across the whole chain. By treating these steps as one observable workflow, we reduce ad-hoc glue code and make it easier to standardise operational behaviour across beamlines while still supporting beamline-specific logic. In this presentation we show how Prefect is used at our beamlines, including MX3 and ADS, and demonstrate patterns for combining user-driven acquisition with automated post-acquisition processing. We discuss lessons learned on reliability, scheduling/queuing, and user experience, and outline how a workflow-first approach supports remote operation and scalable data handling.

Bridging the gap between a visiting user's X-ray Photon Correlation Spectroscopy (XPCS) experiment concept and its technical feasibility at Argonne National Laboratory's Advanced Photon Source (APS) often requires extensive consultation with beamline scientists. An integrated AI platform at APS Beamline 8-ID is introduced, consisting of multiple components. An agentic AI system autonomously explores publication records across global research center databases, selects publications by evaluating paper content for relevance to XPCS, and submits them to a human review queue before ingestion. Approved publications are embedded using SciBERT and stored in a Qdrant vector database, powering a retrieval-augmented generation (RAG) system that enables users to evaluate experiment feasibility through a conversational interface grounded in curated XPCS literature rather than general LLM knowledge. A document ranking system allows beamline scientists to assign priority weights to ingested publications, ensuring responses reflect the most relevant sources for the beamline. Users can refine hypotheses and inquire about experiment feasibility across multiple turns, such as whether 8-ID can maintain sub-angstrom resolution during high-temperature metallic glass aging over extended timescales, and receive scientifically grounded responses with source attribution. By providing users with on-demand access to beamline-specific expertise, the platform enables better experiment preparation, deepens users' understanding of experiment feasibility, and reduces burden on beamline scientists.

As part of the upgrade project of the Swiss Light Source (SLS) to a fourth-generation synchrotron, a unified solution for controlling and orchestrating experiments has been developed. The Beamline and Experiment Control system (BEC) leverages community tools but embraces a service-oriented approach to facilitate fast and maintainable developments in a rapidly evolving scientific environment.
In this contribution, we present the experience gained during the development and rollout of BEC across the SLS beamlines over the past years. We discuss decisions related to deployment strategies, monitoring infrastructure and user interface design as well as the challenges of establishing a common experiment control framework across beamlines that previously relied on independent and highly customized solutions. In addition, we reflect critically on approaches that proved successful in operation as well as decisions that later required redesign or simplification. By sharing both stories of success and shortcomings, we aim to highlight areas that may be relevant for other facilities facing similar organizational structure and technical challenges.

Feedback from real users who rely on software daily is one of the most valuable inputs for developers. In scientific facilities using distributed control systems, large-scale GUIs must handle hundreds of process variables event notifications and polling requests, often affected by hardware delays and timeouts. Users expect reliable, fast data delivery, and startup time is the first hurdle: poor initial performance can quickly undermine user confidence.

Taurus is a Python framework for building graphical user interfaces that support multiple control systems and data sources. It is a community-driven, open-source project used for over a decade in scientific facilities, including synchrotrons (ALBA, DESY, MAX IV, SOLARIS), laser laboratories (MBI Berlin), and observatories (ESO).

Taurus originally suffered from slow GUI startup times when dealing with large number of process variables. An optimization effort initiated four years ago has reduced startup times by up to 70%, first by addressing polling inefficiencies and later improving event subscription mechanisms (for TANGO). A third phase now targets runtime performance optimizations as conditions evolve. Beyond performance, Taurus continues to expand its community. The outcomes of a recent workshop at SOLEIL will be presented, including developments, user feedback, and the roadmap.

Taranta is a modern web application built on the TANGO control system framework, providing interactive dashboards for hierarchical device visualization, live attribute monitoring, and remote command execution. This talk presents the work carried out at DESY Photon Science to deploy and integrate Taranta into the DESY infrastructure, enabling secure and scalable remote access to beamline control systems.
We describe the integration of OAuth2Proxy into Taranta, allowing seamless external authentication via Keycloak, and the development of a custom Door authentication plugin for Traefik that enforces beamtime-limited user access with fine-grained authorization. Dashboard sharing is implemented through DESY LDAP and Keycloak groups, enabling role-based collaboration across beamline teams.
On the deployment side, we present our Helm charts for running Taranta on the DESY Kubernetes cluster, incorporating Consul for service discovery and dynamic configuration management across the cluster. We further discuss the integration of Daiquiri, a unified web-based user interface for beamline control software, providing a consistent front-end experience on top of Sardana, Bluesky, and BLISS frameworks alongside Taranta's TANGO-based dashboards.
Together, these components form a robust, production-ready stack that brings modern, secure, and well-integrated remote access capabilities to beamline operations at DESY.

Sardana and Bluesky are two widely adopted frameworks for experiment control and data acquisition at synchrotron beamlines. While Bluesky's queue server provides a modern interface for remote experiment orchestration, many beamlines continue to rely on Sardana macros as their primary control workflow. This creates a challenge for facilities seeking to modernize their interfaces without disrupting established Sardana-based operations.
We have modified the bluesky-queueserver to enable Sardana macros to be executed as native queue items. Sardana macros are wrapped into Bluesky-compatible callable objects and integrated into the queue server's execution model, allowing beamline scientists to submit and manage Sardana macros through standard Bluesky interfaces such as Daiquiri — without requiring migration away from Sardana. This enables users to remotely submit macros into the queue via the Bluesky queue server, bringing modern remote access and workflow orchestration capabilities to beamlines that rely on existing Sardana macro libraries.

The OSCARS project PaN-Finder explores how recent advances in artificial intelligence, particularly large language models (LLMs) and vector embeddings, can be leveraged to enhance FAIRness of open data in the European photon and neutron community. We developed a retrieval-augmented generation (RAG) system that improves data findability and relevance in the search results by enabling both expert-driven, domain-specific queries and more general, non-expert searches.
This work provides new insights into data curation and search methodologies, revealing shared patterns and variety present in open data across facilities and scientific disciplines. It also demonstrates how AI assisted methods can enrich metadata, supporting more accurate relevance ranking and improving scientific data reuse. The PaN-Finder challenges traditional top-down curation models, and promotes a more holistic, data-driven approach better aligned with the current rapidly evolving landscape of data and technology.
In this talk, we present the project’s journey, from conceptual design to implementation, and discuss the key choices that help shaping the current PaN-Finder tool, highlighting lessons learned and future directions in FAIR data.

The Magnetic Materials Group Toolbox is a collaborative project across the Magnetic Materials Group of beamlines at Diamond Light Source. Its purpose is to bring common data-analysis tools together into a single Python module, enabling users to write simple analysis scripts and Jupyter notebooks for routine measurements. Common functionality include reading scan data from NeXus files, automatic normalisation, peak fitting, creating detector integration regions of interest, analysis of XMCD spectra, absorption corrections and much more.

DiamondLightSource/mmg_toolbox

The repository includes scripts and notebooks for automated data analysis workflows, designed to work on current and future Diamond software infrastructure. It provides a well defined python environment that can be containerised for various cloud-based deployments.
The MMG Toolbox also includes a beamline configurable graphical user interface for viewing experimental data. The viewer reads NeXus files and takes advantage of the format’s automatic plotting features to display the appropriate axes and detector images. It also presents beamline specific metadata and can be configured to suit user needs. The viewer supports a range of simple data-analysis tasks, including peak fitting, plotting multiple scans (as multi-line plots, 2D images, or 3D surfaces), and generating Python scripts and Jupyter notebooks to provide flexibility in data manipulation.

Next-generation photon sources face increasing demands for handling high data throughput and highly diverse experimental methodologies. We have developed a modular workflow management system (WMS) designed to streamline data processing pipelines for multi-disciplinary beamlines.

The system is built upon our "Daisy" software framework, which adopts architectural patterns inspired by established frameworks like Gaudi and Mantid. This foundation allows for a clear separation between algorithm development and execution logic. Key features of the WMS include a hierarchical encapsulation mechanism that enables scientists to share and reuse specialized methods, and an internal engine that supports graphical orchestration and distributed task execution. Furthermore, by modeling the full lifecycle of data elements, the system enhances the reproducibility and verifiability of scientific results.

The framework's utility has been demonstrated through pilot implementations in automated data reduction and analysis for typical synchrotron experiments. At NOBUGS 2026, we hope to share our practical experiences in building this system and discuss how such a workflow-driven approach can better support user groups in managing complex experimental data.

We present the NRS Agent, a data analysis agent system for Nuclear Resonant Scattering (NRS) experiments conducted at synchrotron light sources. The NRS Agent is a multi-agent assistant designed for end-to-end Nuclear Resonant Scattering workflows aimed at reducing manual effort and accelerating complex analyses. The agent is designed specifically for data recorded by the High Energy Photon Source (HEPS), a high-energy synchrotron light source under construction in China, although it may also be applicable to other light sources. This agent is developed using the Dr.Sai framework, an AI-driven multi-agent system designed to automate end-to-end physics analysis workflows. The major NRS analysis domains covered by this project are the Coherent Nuclear Resonant Scattering and Nuclear Resonant Inelastic X-ray Scattering (NRIXS) experiments. To provide a comprehensive analytical framework, the NRS Agent integrates the standard analysis packages CONUSS and PHOENIX, which are used for coherent scattering modeling/fitting and extracting material phonon properties from NRIXS data, respectively. The agent automates the entire analysis workflow from inspection of raw measured data and data type conversion to create analysis-ready datasets to generation of structured configurations and execution of CONUSS and PHOENIX pipelines. Broad parameter control, reproducible workflow management, high-throughput parallel analyses across multiple datasets and physical configurations, and wise iterative fitting are among the features that contribute to the agent's utility for end users. This agent serves as a powerful assistant for scientists, significantly enhancing both the speed and accuracy of their analyses.

We investigate the use of an AI Copilot to support beamline staff during experiments at the European XFEL. This is part of an effort to leverage generative AI to improve operations. The results of an initial survey of beamline staff indicated that interactions with the data acquisition (DAQ) system during beamtimes are a common source of problems, and are thus a focus of the initial prototype.
The AI Copilot consists of an anomaly detection model, whose outputs are forwarded by a retrieval-augmented generation (RAG)–based knowledge assistant to the Zulip chat client, enabling scientists to communicate effectively with the Copilot through a familiar interface. In this contribution, we present the survey results and initial findings from classifying control system data accessible to the model. This data, from all scientific instruments, was recorded over a period of multiple weeks. We also introduce a Markov chain–based anomaly detection approach, evaluated on synthetic as well as real data from the control system.

The Advanced Photon Source is transitioning to using the American Science Cloud (AmSC) for data analysis. The American Science Cloud is a centralized platform provided by the U.S. Department of Energy to accelerate scientific discovery by leveraging artificial intelligence. As a part of initial AmSC demonstrations, the APS engaged in two development efforts as an infrastructure partner team. The first was to develop standard reusable stages for the APS Data Management workflow engine to submit data transfer and processing tasks to the National Energy Research Scientific Computing Center (NERSC) and the Argonne Leadership Computing Facility (ALCF) using AmSC APIs. A new workflow was developed for the Meta AI Segment Anything Model (SAM 3) and applied for tomographic segmentation of data from APS beamline 2-BM. This demonstration was developed in partnership with the Stanford Synchrotron Radiation Lightsource and the Advanced Light Source. The second development effort was integration of the AmSC APIs into the Ptychodus application to support AI/ML model training and inference using PtychoPINN as well as conventional reconstructions using pty-chi. This demonstration was developed in partnership with the Linac Coherent Light Source. Future efforts will include operationalizing the AmSC demonstrators and transitioning existing workflows for additional X-ray data analysis tools.

Recently a comprehensive REST and socket.io API on top of the BLISS controls system [1] has been developed allowing for remote control and monitoring of core beamline components as well as launching of scans and macros. The REST API has opened up new possibilities for creating simple User Interfaces (UIs) for beamline monitoring. To this extent blisswebui has been created, a light-weight web based monitoring UI framework for BLISS released as a python package and built on top of @esrf/daiquiri-lib [2]. blisswebui is configured directly from within the BLISS configuration utility beacon, and is simple to install on a per beamline basis via pip without the need to interact with the javascript tool chain. @esrf/daiquiri-lib was created as a shared javascript UI component library, extracting the core widgets from daiquiri [3] and making them available to other projects. The aim is to provide a coherent User Experience (UX) with the acquisition UI daiquiri [2] so that users can move seamlessly between the two applications.

In addition a client library on top of the REST API, blissclient [4] is also provided. This is a small python library that can be used to remotely control bliss and provide meaningful feedback in case of errors. It also integrates with blissdata [5] to retrieve scan data from redis. The library avoids the need to make direct REST API calls and provides a coherent API from python.

Further information can be found at https://bliss.gitlab-pages.esrf.fr/bliss/master/blissapi.html

[1] BLISS: https://bliss.gitlab-pages.esrf.fr/bliss/master/
[2] daiquiri-lib: https://www.npmjs.com/package/@esrf/daiquiri-lib
[3] daiquiri: a web-based user interface framework for beamline control and data acquisition, Fisher et al., J. Synchrotron Rad. (2021). 28, 1996-2002
[4] blissclient: https://gitlab.esrf.fr/bliss/blissclient
[5] blissdata https://gitlab.esrf.fr/bliss/blissdata

Recent advancements in high-repetition-rate x-ray sources and experimental complexity at large-scale facilities, such as FLASH (Free electron LASer in Hamburg), have placed increasing demands on data acquisition and analysis systems. While HPSS (High Performance Storage Systems) and HPC (High Performance Computing) clusters provide efficient resources for processing the vast data generated, many users face steep learning curves in accessing and analyzing this data efficiently. We introduce fab (Flash Analysis for Beamtimes), a Python-based library designed to streamline data loading and analysis at FLASH. fab automates the data retrieval and synchronization processes, handles interactions with the cluster workload manager, and provides an accessible platform for users with varying levels of data science experience. By simplifying the handling of device-specific formats and HPC configurations, fab enables researchers to focus on experiment-specific analyses without needing extensive technical expertise.

To support Phase II of CSNS and overcome key limitations of existing data management system (DMS), a new generation DMS tailored for AI4Science has been designed. A new metadata standard for photon and radiation experiments will be applied to enable unified management across the entire experiment lifecycle and multiple facilities. The system federates NoSQL and vector databases, supporting both keyword-based and semantic-based metadata search. Tiered storage employs a peer-to-peer mechanism to manage data transfer at the file base between tape and disk, with on-demand compression during file movement and storage. AI agents leveraging locally deployed large language models (LLMs) are under development: one extracts and completes metadata from diverse data sources, another generates automated experiment analysis reports, and a third, an experiment planner, uses retrieval-augmented generation (RAG) and simulation to assist in experiment design. A new data portal will integrate these capabilities, offering users a unified interface for experiment planning, simulation, data search, visualization, analysis, and retrieval.

DNS is a multipurpose diffuse neutron scattering instrument. It supports polarized powder and single-crystal diffraction studies and also enables neutron time-of-flight (TOF) spectroscopy measurements when an optional disk chopper is inserted into the neutron beam between the polarizer and the sample. The instrument is particularly powerful for investigating magnetic correlations in complex magnetic materials and exotic quantum magnets, as well as low-energy magnetic excitations when operated in a time-of-flight mode.

Data reduction is an essential stage of the DNS data processing pipeline. It involves removing contributions from the instrument and its environment from the measured raw data, followed by the transformation of the corrected data into physically meaningful $(Q, E)$ space in terms of the scattering vector $Q$ and energy transfer $E$.

In this work, we present DNS Reduction - a new graphical interface that has recently been fully integrated into the open-source Mantid software project. The interface enables instrument scientists and users of the DNS instrument at MLZ to reduce their data in a user-friendly and reproducible way. It implements a set of data-reduction algorithms for the main types of DNS experiments, including polarized powder diffraction, TOF inelastic scattering on powder samples, and polarized single-crystal diffraction. It is intended to replace the legacy DNSplot software at the instrument.

The OSIRIS backscattering indirect geometry spectrometer at ISIS Neutron and Muon Source is undergoing a major upgrade with the addition of a silicon crystal analyser and position sensitive detectors, complementing the existing graphite analyser and scintillation detectors, respectively. This will extend the accessible timescale from picoseconds to nanoseconds, opening new scientific opportunities in areas such as ionic conductivity in battery materials and slow diffusion processes in solid-state systems. The position sensitivity of detectors will enable to measure momentum transfers in the vertical direction, which will help mostly the single-crystal community with access to full four-dimensional S(Q, ω) spectra.

A central part of the software work involved updating the Instrument Definition File (IDF) — an XML-based description of the instrument geometry that Mantid uses to drive coordinate transformations, detector and pixel grouping, and visualisation across the framework. By adding accurate silicon analyser positions and new parameter files covering energy fixed values and resolution for each silicon reflection, the new analyser became immediately visible in Mantid's Instrument Viewer and correctly handled by the framework's geometry engine without bespoke changes. Building on this, the ISISIndirectEnergyTransfer algorithm and the Indirect Transmission tools were extended to support silicon reflections, with unit and system tests added to cover both analysers.

On the interface side, the prior refactoring of the Indirect Data Reduction GUI into an MVP architecture significantly reduced the cost of adding silicon analyser support: users can now select the silicon analyser and individual reflections directly in the GUI, and both analysers can be run alternately.

This contribution describes the software engineering approach, the design decisions that eased the extension, and lessons relevant to other facilities planning similar instrument upgrades.

At the European XFEL (EuXFEL), modern photon science experiments generate high-rate data streams requiring low-latency processing and precise timing. Such constraints demand novel approaches to data processing that go beyond conventional centralized architectures and enable computation closer to the data source.

In this work, an FPGA-based edge computing platform integrated into Karabo, the Supervisory Control and Data Acquisition (SCADA) system used at the EuXFEL, is presented. The platform is based on a Xilinx Zynq system-on-chip (SoC) architecture, integrating programmable logic (PL) and an embedded processing system (PS). This enables dynamic partitioning of data processing tasks between hardware and software to optimize latency and throughput, while supporting real-time interaction with experimental signals via configurable interfaces.

Integration is achieved by deploying Karabo components directly on the embedded system, enabling the platform to operate as a native element of the control and data acquisition framework. Communication with FPGA resources is performed via a Python-based environment using PYNQ, enabling direct control and data exchange between PS and PL. The modular design allows new data processing methods to be integrated with minimal effort, supporting flexible and adaptive processing workflows.

The effectiveness of the proposed approach has been demonstrated using a liquid sample droplets injection system at the SPB instrument, where a photodiode signal is digitized via an analog-to-digital converter (ADC) and processed in real time to estimate the temporal delay between the sample and the X-ray beam, enabling precise temporal alignment. The system achieves reliable real-time performance and scalable data processing, supporting high data acquisition rates in the 100 kHz range.

Note: Please see attachment, it has figure and references

Data-driven methods are rapidly transforming experimental science [1], not only through advances in artificial intelligence but also by enabling scriptable access to scientific instruments [2]. Providing users with programmable control over instrumentation opens new opportunities for statistical analysis, adaptive experimentation, and more efficient materials discovery [3, 4, 5]. However, modern electron microscopy systems remain constrained by fragmented, vendor-specific hardware interfaces [4,5], limiting programmability and flexibility despite their substantial capital cost (often up to $10 million) [6].

In this talk, I will present a modular, vendor-agnostic framework for coordinating heterogeneous electron microscope hardware, where individual subsystems such as stage, scan, and detectors are abstracted as independent device-level services and orchestrated through a unified PyTango-based[6] control layer. This approach enables instruments to be accessed programmatically through scripting interfaces, allowing users to construct flexible, data-driven experimental workflows beyond traditional GUI-based operation. Figure 1 shows the schematic of the interface architecture and its realization in our university on a ThermoFisher spectra 300 microscope. Eventually our plan is to connect our interface with the BlueSky framework[8].

Code availability: https://github.com/pycroscopy/asyncroscopy

At European XFEL, the in-house developed Supervisory Control and Data Acquisition (SCADA) system, Karabo [1], has enabled scientific experiments at the photon beamlines since free-electron laser operations began in 2017. Karabo 2, released in 2016, was developed to meet the requirements of day-one operations, delivering essential functionality while defining core concepts intended to scale to facility-wide usage.

Building on almost a decade of operational experience, Karabo 3 sharpens proven concepts, refines the system architecture, and removes design elements that did not stand the test of practice. The release was scheduled to coincide with the six-month long European XFEL Long Installation and Maintenance Period (LIMP) in 2025, providing a unique opportunity to migrate more than 400 software repositories that are part of the Karabo eco-system.

This contribution discusses the lessons learned from operating Karabo at a large research infrastructure, which motivated the main architectural improvements introduced in Karabo 3, and describes the strategy adopted for the software repository migration.

[1] S. Hauf et al., "The Karabo distributed control system," Journal of Synchrotron Radiation 26(5), 1448–1461, 2019. https://doi.org/10.1107/S1600577519006696

From the very early days of Karabo, the SCADA system in use at European XFEL (EuXFEL) for instruments and photon systems, interaction with other control systems has been a key requirement. The interoperability with DOOCS have been particularly important, as this is the control system in use for the EuXFEL accelerator at DESY. Hence, different interfaces were developed for bi-directional communication between Karabo and DOOCS. Later on, we followed the same approach (i.e., a generic interface) to develop interfaces to other well-established control systems, namely TANGO and EPICS. The purpose was to ease the integration of new hardware devices, including user-provided ones, but also to encourage the use of Karabo outside the EuXFEL facility.
On top of that, we developed the Karabo WebProxy, an HTTP server to expose parts of a Karabo ecosystem to any other software, and its companion Karabo-Proxy, an open source Python client to interact with it.
In this contribution, we will provide an overview of the existing possibilities to operate Karabo together with different control systems.

At the European XFEL, the in-house developed Supervisory Control and Data Acquisition (SCADA) system, Karabo [1], has supported scientific experiments at the photon beamlines since user operations began in 2017. The Karabo GUI, the main graphical user interface of the system, was developed as a multipurpose desktop application based on Python [2] and Qt [3] and serves as the primary entry point to the control system. KIWI (Karabo’s Interactive Web Interface) extends Karabo by bringing panel visualization to the web. Built on the same concepts as the Karabo GUI, which stores operator panels in a database using an SVG-based scene format, KIWI is a modern React [4] and TypeScript [5] application that renders operator panels directly in the browser, including on mobile and tablet devices. This enables new workflows through platform-independent access without local installation, reducing deployment overhead while improving accessibility, portability, and scalability across teams. This contribution presents the motivation, architecture, current status, and lessons learned from the development of KIWI.

References
[1] S. Hauf et al., "The Karabo distributed control system," Journal of Synchrotron Radiation 26(5), 1448–1461, 2019. https://doi.org/10.1107/S1600577519006696
[2] Python Documentation. https://docs.python.org/3/
[3] Qt Framework. https://www.qt.io/
[4] React. https://react.dev/
[5] TypeScript Documentation. https://www.typescriptlang.org/docs/

At European XFEL, scientific data is managed in a hierarchical storage system that combines high-performance storage (GPFS), a dCache mass storage system, and a tape archive. In this architecture, dCache serves as an intermediate layer between GPFS and tape systems, enabling seamless access to raw data from the Maxwell compute cluster throughout the custody period defined by policy. As data volumes generated by multiple instruments continue to grow rapidly, the pressure on the dCache limited storage capacity increases, making large-scale data management and storage planning more challenging. This situation highlights the need for a dedicated analytical tool capable of monitoring and analyzing data access patterns to distinguish between frequently and rarely accessed datasets, and supporting decisions to free up space efficiently for better storage utilization.
To address this need, we developed a data management analytics layer that leverages filesystem events to capture access patterns and usage statistics of raw data stored in dCache. The approach operates through a multi-stage pipeline: dCache billing records are first cleaned and normalized, then processed through ETL-based modeling workflows to derive meaningful metrics, which are subsequently persisted in an InfluxDB time-series data lake. On top of this stack, purpose-built Grafana dashboards provide intuitive monitoring and analysis tailored to specific operational use cases.
This solution provides detailed visibility into data-access patterns across instruments, proposals, and runs, allowing read frequencies to be tracked with fine granularity. Overall, the solution strengthens the data management strategy by fostering transparency and supporting storage allocation decisions with concrete evidence. Looking ahead, the framework is designed to extend to GPFS storage, with the goal of delivering a unified view of access patterns in raw and processed datasets across multiple levels of the storage hierarchy.

Imaging detectors play a key role in X-ray experiments at synchrotron facilities, and their continuous increase in data generation capabilities keeps opening opportunities for new science in many domains. The integration of such fast instruments is a challenge in terms of data acquisition (DAQ) [1], processing and storage. Consequently, the ESRF has developed LIMA2 [2], a scalable framework for high-throughput 2D DAQ and real-time processing, as part of its data strategy for the EBS [3]. LIMA2 targets edge computing for low-latency online data analysis and feature extraction, providing fast feedback to the experiment and advanced data reduction techniques like veto and rejection. The pre-processed output is available to the downstream EWOKS stage [4] through Bliss-Data, metadata and HDF5, resulting in a complete and coherent solution for the ESRF. In production since 2023, LIMA2 is installed at seven ESRF beamlines for controlling PSI Jungfrau, Dectris Eiger2 & Pilatus4, and ESRF Smartpix detectors. In this work we update on the status of the project, with the latest developments like the Fast Azimuthal Integration (FAI) pipeline, based on pyFAI OpenCL code [5]. A new feature in the core processing library is the support of frame accumulation in multi-receiver topology, which is particularly challenging with dynamic frame dispatch. Accumulation can be used during long exposures to ensure a low latency in the algorithm verifying that the photon flux does not go beyond a safe threshold, critical for protecting against radiation damage. All these developments are exposed to the user through a consolidated Bliss and Bliss-Data support [6]. The deployment of new detectors like the Rigaku XSPA is also presented, along with the plans for the next expected detector integrations. Finally, we summarize the anticipated project evolution in the short- and medium-term and share our vision for the long-term.

Operating a large-scale scientific facility such as the European XFEL requires reliable and efficient on-call duty (OCD) management to support 24/7 operations. Initially, these processes relied on manual spreadsheet-based workflows, which were error-prone, non-scalable, and inefficient, consuming effort that should instead be dedicated to development and operational tasks, such as assessing staff availability and tracking working hours, including nights, weekends, and public holidays. To address these limitations, we developed the On-Call and Data Operations Center (DOC) Manager (OCDM), a web-based platform that supports on-call and shift scheduling, incident tracking, and the management of staff availability and personal constraints. The system is built on a layered Java architecture (MVC) following a client–server model. In continuous operation since 2018 and currently used daily by more than 120 active users, OCDM has, within an 18-month audited log window, recorded more than 16,000 access events, including more than 4,800 scheduling operations and 3,000 automated notification emails. This demonstrates the crucial role that OCDM has gained within the data department of the European XFEL.

Between PDF analysis on 1D powder data and 3D-ΔPDF on single crystal data, there is an unexplored area called PDF analysis on 2D data for samples that exhibit fiber-texture and whose scattering signals are cylindrically symmetric in reciprocal space. To measure such kind of samples, the preferred method is collect a series of diffractograms with densely spaced incidence/tilting angles, before reconstructing into 3D reciprocal space data. The measurement takes time in the same order of magnitude as what is needed for a 3D-PDF on a single crystal. In a typical in situ experiment, tilting the sample to large incidence angles is not possible. On the other hand, taking one image at a fixed sample tilting angle leads to the so-called missing-wedge problem. Significant artifacts appear in real-space after Fourier transform of the scattering data with missing-wedge, making it impossible to recognize useful feature for real-space analysis. No effective way has been reported to address such an issue. Here, we developed an iterative algorithm to minimize the artifact. This method makes it possible to extract reliable real-space information for fiber-textured sample with one-shot measurement, potentially to enable in situ experiments on such kind of samples. A thin film platinum sample is employed to validate the method.
Along the core method we developed, we show a few techniques that facilitate the data reduction: correct for sample rotations/positions based on scattering signals, ray-tracing style absorption correction, and minimizing the foot-print of shallow angle incidence by deconvolution with varying kernels.

Synchrotron X-ray diffraction (XRD) experiments are a versatile tool in understanding material properties and processes in a wide range of applications. However, advanced synchrotron diffraction data analysis has traditionally required strong physics expertise, limiting broader adoption of diffraction methods. In addition, faster experiments and the increasing use of in situ environments require rapid data processing and feedback during beamtime to enable efficient experimental decision making.
To address these challenges, we present pydidas [1], an open source Python based diffraction data analysis suite developed at Helmholtz Zentrum Hereon. pydidas is designed to improve the data processing for existing users and broaden the potential user base for our XRD experiments by delivering a user-friendly and fast processing tool. It is designed to inherently use community-standard data formats like NeXus and make use of parallelization.
pydidas aims to integrate all essential steps of XRD data processing—including data browsing and visualization, experiment calibration, workflow setup, processing, and result visualization—into a single software environment with an intuitive graphical user interface.
To accommodate diverse analysis requirements, pydidas uses a modular, plugin based workflow architecture. Core processing functionality includes commonly required steps such as azimuthal integration (using pyFAI), corrections, and fitting. In addition, custom plugins can be easily integrated to support specialized or experiment specific workflows beyond the provided generic functionality. pydidas is actively developed and used in routine beamline operation, with ongoing extensions driven by user feedback and emerging analysis needs.
[1] http://pydidas.hereon.de

The increasing demand for open, reproducible, and FAIR-compliant research has made robust data management and persistent identification of data, samples, instruments and processing workflows essential in modern scientific large-scale infrastructures. This work presents the current implementation of dataset publication and DOI minting within SciCat at DESY. Datasets are curated, enriched with metadata, and assigned persistent identifiers to support citation, accessibility, and long-term preservation. The DOI minting workflow enables seamless linkage between datasets and associated publications, promoting transparency and reuse of scientific data.
Complementing this effort, we introduce the BlissData Ingestor, a schema-driven tool designed to bridge the gap between raw experimental data and structured data management pipelines at DESY beamlines. The ingestor processes scan data available in BlissData and applies LinkML-based mappings and validation to ensure semantic consistency and data quality prior to ingestion.
A key capability of the presented tool is its support for live metadata handling in ongoing data acquisitions campaigns streaming and retrospective data curation. Leveraging BlissData and it’s events handling of live scans in near real-time, enables integration with online monitoring workflows, while historical scans can be selectively replayed for reprocessing without repeating experiments. Once validated, the data is automatically ingested into SciCat, making it immediately discoverable and accessible through the scientific data catalogue.
By integrating schema-driven validation, automated ingestion, and DOI minting, this work advances towards a more scalable, efficient, and user-friendly data management ecosystem. Together, these developments reduce manual intervention, improve data reliability, and strengthen the overall research data lifecycle from acquisition to publication.

OpenHKL is software for the reduction of single crystal diffraction data, written in C++ and with a modern Qt graphical interface. Given a sequence of diffraction images captured at incremented sample rotation angles, it produces a list of indexed peaks with integrated intensities. It is open-source, has a Python scripting interface, and natively handles neutron diffraction data. Developed at the Heinz-Meier Leibnitz institute (MLZ), it will handle data reduction for the BioDiff, HEIDI and POLI diffractometers, and can be easily extended to other instruments with different data collections strategies and detector geometries.

In 2020, ESRF installed and commissioned the EBS storage ring, which significantly increased X-ray flux. Taking advantage of the scientific opportunities presented by the EBS beam, ESRF has simultaneously invested heavily to equip beamlines with progressively larger and faster detectors

In anticipation of the data deluge from the EBS upgrade, ESRF has created a forward-looking, comprehensive software strategy built on modularity, scalability, and open science principles. This strategy positions the facility to not only manage the enormous immediate data output from EBS but also to adapt to future increases in flux and detector capabilities. Within this strategy, LIMA2 handles image acquisition with scalable detector control and early data processing. BLISS coordinates beamline synchronization and data acquisition. EWOKS processes the data through workflows that BLISS can trigger online or users can run offline. The ESRF data catalogue and portal store and manage all experimental data and metadata. Blissdata ties everything together as the central layer connecting data producers and consumers. Tango provides the backbone of distributed control for the accelerator and beamlines, while Qt and web-based solutions (silx, flint, h5web, Daiquiri, MXCuBE) handle data display and graphical control interfaces in various contexts. Data and software interoperability is achieved through standardized data formats (hdf5, NeXus) and consistent metadata annotation practices.

The result is a future-proof infrastructure that supports both traditional experiments and emerging modes of data-intensive, open data and collaborative science across our user community. The status of the ESRF software ecosystem will be presented as well as current and future developments. We will also analyze some perspectives into the future such as, for example, the role of DRAC and domain portals to enable science beyond traditional schemes or the new perspectives opened by Artificial Intelligence.

Title: Toward Autonomous Beamline Operation: Early Integration of a Beamline Agent in BLISS at PETRA P25

Abstract:
Automation remains a central objective in modern synchrotron beamline operation, promising improved efficiency, reproducibility, and user accessibility. At the P25 beamline of PETRA III, we are taking initial steps toward this goal by introducing a beamline agent designed to operate within the BLISS control environment.

This contribution presents the current status of the beamline agent, which is still in an early development phase but already demonstrates the ability to perform nontrivial experimental tasks. In particular, the agent can autonomously execute procedures such as slit alignment, combining domain knowledge with adaptive decision-making. These capabilities illustrate the potential of agent-based approaches to move beyond scripted automation toward more flexible and intelligent control strategies.

We provide an overview of how the beamline agent is integrated into the BLISS ecosystem, including its interaction with hardware abstractions, command interfaces, and data acquisition workflows. Emphasis is placed on architectural choices that allow the agent to operate transparently alongside existing control mechanisms while remaining extensible for future capabilities.

While the system is not yet production-ready, these early results highlight a promising direction toward the long-term vision of fully automated beamlines. This work aims to share insights, challenges, and lessons learned with the NOBUGS community, and to encourage discussion on how agent-based solutions can reshape beamline software and experimental practice.

Why does the development of scientific software often deviate from robust software engineering standards? Reflecting on the motivations behind each reveals the tacit route to what is deemed successful software. Industry software is typically designed for scalability, stability, and broad applicability, while scientific software prioritizes adaptability, exploration, and domain-specific insight for a smaller, expert user base. Reduction sits at a crossroads where strict standards must be met to achieve facility goals, but the process is still required to grow with the community.

This poster will examine the merits of each approach when developing a standardized software for facility end users. It will assert where standard engineering practices should remain firm--reproducibility, provenance, and transparency--and where flexibility is required to not stifle the scientific process, such as workflow flexibility, and facets for exploration. Drawing on experience with SNAP reduction, I highlight real examples how each approach either succeeded or failed to achieve facility goals.

I further argue that many recurring “software problems” are fundamentally social, arising from fragmented collaboration and lack of consensus rather than technical limitations. Rather than proposing a global solution, I present practical design strategies for building systems that remain robust and usable in the presence of ongoing methodological divergence.

Large-scale research facilities such as the European XFEL rely on precise real-time tuning of numerous interdependent subsystems to ensure stable and optimal operation. Automating these procedures can significantly reduce the workload of operators while improving the efficiency and reliability of beamtime usage. In this work, we will present ongoing efforts aimed at automating various elements of the photon system of the European XFEL. In particular, we will focus on the alignment of the photon beam transport, achieved by adjusting multiple optical components such as mirrors and lenses, using precise and constrained actuation. Furthermore, we will discuss the automation of the alignment and calibration of an X-ray spectrometer. By replacing manual interventions with algorithmic control, these developments demonstrate the potential to streamline the alignment of photon systems while allowing operators to focus on high-level experimental optimization.

After 31 years of operations the Italian third generation synchrotron light source Elettra will be replaced by Elettra 2.0, a fourth-generation one. The upgrade project includes the construction of at least five new beamlines, bringing the total to 32. We are using the dark period to replace the existing beamline Personnel Safety System with the new Beamline Access Control System BACS 2.0. Based on PILZ fail-safe PLC, it controls the user access to the radiation hutches of Elettra 2.0. The access to each hutch is based on individual permissions using a personal RFID key. BACS 2.0 is modular in both hardware and software: its architecture has been designed in order to be scalable and capable of managing up to eight hutches. A TCP modbus Tango device and a dedicated web interface have been developed for monitoring the system. The new PSS has successfully been installed on four beamlines and six more installations are planned for 2026.

Operando synchrotron experiments require coordinated control of complex sample environments and beamline measurements over several days. These experiments are often run manually or through monolithic scripts, limiting flexibility and making live adaptation during beamtime difficult. Within the ROCK-IT project, we developed a Bluesky-based orchestration framework that represents experiments as machine-readable stages combining sample environment conditions with X-ray measurement routines. The framework uses two independent Bluesky Queueservers: an outer server controls experiment progression and sample environments, while an inner server executes editable beamline measurement loops. This hierarchical design allows measurements to be modified during operation without interrupting the experiment.

The framework was deployed for operando catalysis experiments at multiple facilities. We present methods for queue inspection, graphical recipe preview, and validation before execution, enabling safer unattended operation, live adaptation, and improved reproducibility.

High Energy Photon Source (HEPS) experiments generate large volumes of image sequences, whose wide dynamic range, non-negligible noise, and complex temporal redundancy make it difficult for existing lossless compression methods to reduce data volume effectively. We propose a two-stage lossless compression method based on the Mamba state space model. The original images are first transformed into token sequences by inter-frame differencing and Flag quantization. A stage-one Mamba model predicts tokens position by position, and prediction residuals are computed in the physical-value domain. Since these residuals have a more concentrated distribution, a clipping-and-escape mechanism is used to preserve large residuals exactly, while a stage-two Mamba model performs entropy modeling and arithmetic coding on residual tokens. During decompression, residual decoding, stage-one re-inference, and token recovery reconstruct the original images exactly, with pixel-level verification ensuring full losslessness. Tests on real datasets show that our method achieves an average compressed-to-original size ratio of 46.52%, corresponding to a compression ratio of 2.1496x, which is 29%-81% higher than those of three other methods, JPEG-LS, JPEG-XR, and gzip level 9, whose compression ratios on the same dataset are 1.6563x, 1.6241x, and 1.1855x, respectively. These results demonstrate that two-stage residual modeling can effectively improve the lossless compression performance of synchrotron image sequences.

Operando spectroscopy plays a central role in catalysis research, but its full potential increasingly depends on efficient automation of experiments, control systems, and data handling. Within the project ROCK-IT (Remote, Operando Controlled, Knowledge-driven, and IT-based), we developed a framework to support automated in situ XAS measurements of catalytic samples at the CATACT beamline at KARA.
The project delivered upgrades in both sample environment control and online data processing. Integration of ASAP::O as a real-time streaming framework enables near real-time access to measurement results and automated conversion of raw SPEC data into metadata-rich NeXuS format. In parallel, a Tango-based device control module was implemented for the dedicated sample environment, allowing automated operation under controlled gas mixtures and temperature conditions. For automation of sample exchange a robot is integrated in the setup.
The framework was tested in a coordinated round-robin study of a Ni catalyst benchmark sample across three Helmholtz centers (KIT, DESY and HZB). The campaign served both as a validation of the technical developments and as a demonstration of interoperable, distributed operando workflows. This contribution presents the implemented infrastructure and discusses lessons learned toward more automated and reproducible operando spectroscopy experiments.

As a fourth-generation synchrotron radiation light source, HEPS generates BCDI data that far exceeds that of third-generation sources. The reading, processing, and saving of these data have become extremely challenging. Daisy-BCDI is a high-performance computing software developed based on openClaw. With the backend connected to the Gemma 4:31B open-source model API key, users can perform data reading, processing, and result saving of the BCDI software through both conversational and interface-based interactions. The software supports multi-GPU parallel computing, enabling the use of multiple GPU cards for parallel computation of multiple random seeds during phase retrieval, significantly accelerating the reconstruction speed.

EasyScience (https://github.com/easyscience) is a framework initiated by the European Spallation Source (ESS) to unify and streamline data analysis across neutron scattering techniques. Built with Python and QML, EasyScience aims to accelerate the analysis step in the data processing workflow by providing friendly graphical interfaces for new users and scripting options via Jupyter notebooks for advanced users.

The framework includes two main core components: EasyApplication, the front end, and EasyScience, the back end. The front end offers a collection of shared graphical interface elements for building user-friendly applications, while the back end enables model-dependent analysis through integration with established calculation engines.

Current active projects include EasyDiffraction, EasyReflectometry, EasyImaging, and EasyDynamics. EasyDiffraction and EasyReflectometry, which address neutron diffraction and reflectometry workflows, are the more established projects, while EasyImaging, for Bragg-edge imaging, and EasyDynamics, for quasielastic neutron scattering, were started more recently. All four projects are under active development, with additional functionality planned as they mature.

Two other recently started collaborative projects, EasyTexture and EasyShapes, aim to complement these analysis tools and adapt the EasyScience framework to data reduction workflows and to the preparation and execution of molecular dynamics simulation workflows, respectively.

This contribution provides an overview of the EasyScience framework, highlighting its current applications and its potential to enhance neutron scattering data processing.

The ESS data path for science instruments has a natural split into distinct paths for detector data and controls data. Four elements make up the detector data path (data path): Frontend electronics, readout master, event formation unit and Apache Kafka.

The data path consists of a combination of firmware (FPGA) in the first two elements and server software in the latter two elements. This allows high precision timestamping in firmware as well as flexibility in the processing of detector readouts.

The data distribution consists of a multi-port optical ring architecture in at the front ends and switched 100G Ethernet in the server infrastructure. Using Ethernet and IP addressing, data from different rings can be directed flexibly to single or multiple- servers and processes.

Finally, we use Apache Kafka which provides a fast, scalable (and ephemeral) storage.

The fundamental operation of the data acquisition system was designed between 2016 and 2017 and is expected to remain in its current form when ESS starts initial operations in early 2027.

BESSY II operates more than fifty beamlines supporting a broad range of experimental techniques across diverse scientific fields. With unique undulators, world-record photon energy resolution (e.g., <1 meV at 60 eV), femtosecond time resolution, and nanometer-scale spatial precision, the facility enables highly demanding experiments. These increasingly include operando and multimodal techniques that monitor samples and processes in real time. Meeting these requirements makes robust, reproducible, and interoperable data lifecycle management essential.

In this work, we present XAS round-robin studies initiated at the mySpot multimodal beamline of BESSY II. MySpot is a multi-modal beamline designed to perform different techniques—XRD, SAXS, XRF, EXAFS, and XANES—simultaneously at the sample position. Round-robin studies are widely used across scientific disciplines to ensure quality control, identify cross-platform inconsistencies, and advance understanding of experimental systems, systematic effects, and theoretical limitations. By comparing raw and processed spectra across BESSY II beamlines, the study aims to identify systematic differences in energy calibration, spectral amplitudes, and data-processing workflows.

Technically, at the heart of this work is NeXus, an interoperable data format; pynxools, a data converter for making experimental data FAIR though NeXus; nx5d for fine-grained management of data analysis parameters; and EWOKS, an extensible workflow management system. Preliminary results on automating data-evaluation tasks using these tools will be presented.

A 20-year-old codebase, an ancient and inefficient dependency, a weakness tangled up in 1.5 million lines of code, and the proper tools to finally restore order and performance.
The Mantid project is an international software collaboration to create a program suite for every need in neutron scattering data reduction workflows, first begun in 2007. Since that time, it has relied on the API released by the NeXus International Advisory Committee for reading neutron scattering data files. Multiple code audits showed Mantid had severe performance deficiencies in data loading. Users frequently complained about load times. However, data loading was at the heart of literally every workflow performed in Mantid. How can a central, legacy operation be replaced, when it has the potential to break any and every workflow?

This will discuss the strangler pattern for legacy system updating, and how to properly make use of it when attempting to refactor old, venerable, yet outdated code dependencies.

The increasing data production rates at accelerator-based light sources necessitate the development and adoption of efficient data reduction and compression techniques [1]. Pre-storage data reduction, as well as various forms of on-the-fly processing systems [2], are becoming increasingly prevalent. Hardware-accelerated computing, particularly heterogeneous GPU–CPU–FPGA systems [2], offers an energy-efficient and sustainable infrastructure for the automated processing of photon science data. Typical applications include spot finding, indexing of diffraction patterns, tomographic reconstruction, small-angle scattering simulations, and azimuthal integration (AZINT).
This contribution focuses primarily on the latter application, AZINT, implemented on FPGAs several years ago [3]. The work explores an efficient end-to-end implementation of all processing stages, including data decompression, on a single FPGA device, with an emphasis on optimizing memory access patterns and bandwidth utilization.
Experimental data are often delivered already compressed by detectors to reduce network bandwidth requirements or stored in compressed form for practical reasons. We present the challenges associated with implementing the bitshuffle-LZ4 decompression method on FPGA hardware and compare it with alternative LZ77-based approaches, particularly Snappy compression. This comparison highlights how the selection of a compression scheme, often guided by performance on general-purpose computing architectures, can significantly impact its efficiency when implemented in dedicated hardware.

[1] N. Soler, V. Favre-Nicolin, LEAPS-INNOV D7.1 - Evaluation report on data rates & volumes and assessment of future needs of LEAPS facilities, doi:10.5281/zenodo.17099463
[2] F. Leonarski et al., J. Synchrotron Radiat., 30, (2023), 227. doi:10.1107/S1600577522010268
[3] Z. Matěj et al., bincount implementation of Azimuthal Integration (AZINT) with FPGAs, gitlab.com/MAXIV-SCISW/compute-fpgas/bincount

SciLog [1] is an electronic log notebook developed and deployed at the Paul Scherrer Institute (PSI). As part of the Open Research Data [2] project, we have further enhanced its integrations with systems within the PSI infrastructure, as well as its interoperability with other FAIR data management tools.

In the context of experiment data management, a logbook is usually linked to a specific proposal. Because datasets in SciCat [3] are also linked to experimental proposals, we have added a widget in SciLog that displays these proposal-linked datasets and allows the user to link additional ones. This integration features seamless Single Sign-On (SSO) from SciLog to SciCat, ensuring the user does not need to supply credentials again.

Furthermore, SciLog is now part of the ELN Consortium and supports exporting logbooks in the ELN File Format [4]. This provides high-level interoperability with other tools supporting the format, such as elabFTW.

We present the lessons learned from SciLog in production and the feedback gathered from the users, details of new features, and the currently planned future developments, which include leveraging the ELN Format to interoperate with OpenBIS) and archiving logbooks directly to SciCat for long-term storage.

References

[1] K. Wakonig, A. Ashton, and C. Minotti, “Scilog: A Flexible Logbook System for Experiment Data Management”, https://doi.org/10.18429/JACoW-ICALEPCS2023-THPDP073

[2] https://ethrat.ch/de/eth-bereich/open-research-data/

[3] C. Minotti et al., “Enhancing Data Management with SciCat: A Comprehensive Overview of a Metadata Catalogue for Research Infrastructures”, https://doi.org/10.18429/JACoW-ICALEPCS2023-THMBCMO02

[4] The ELN File Format: https://github.com/TheELNConsortium/TheELNFileFormat

Powder diffraction is a notoriously ill-posed inverse problem: the observed one-dimensional pattern entangles lattice parameters, space group symmetry, atomic arrangement, microstructural effects, and instrumental contributions, which classical pipelines address sequentially with limited reproducibility. We present CrystalAI, a machine learning framework for end-to-end crystal structure solution from powder diffraction patterns. At its core is a dedicated disentanglement block that learns to separate the physical contributions to the diffraction signal, coupled with crystallographic constraints embedded into the architecture and training. A central design principle is joint training on simulated and real experimental patterns: simulation alone leaves a persistent sim-to-real gap, while experimental data alone is too scarce. We present models for crystal system and space group prediction, together with a generative component proposing atomic positions in the unit cell. To make these models accessible to the crystallographic community, we are integrating them into AIXtal, a Rust-based web application for powder diffraction analysis and refinement.
This integration makes CrystalAI directly relevant to the FAIR data agenda advanced by DAPHNE4NFDI. Such models depend on curated, openly accessible diffraction data and produce reusable artifacts — learned priors, trained models, structured outputs — that themselves require FAIR stewardship.

Effective UI/UX design is essential for scientific graphical user interfaces (GUIs), particularly in environments where complex workflows and time-critical tasks challenge usability. Applying user-centered design principles, accessibility standards, and continuous user feedback can significantly improve the usability and efficiency of experimental physics control systems.

This presentation highlights ongoing efforts to update and improve some of the existing GUIs used within DAPHNE4NFDI (DAta from PHoton and Neutron Experiments for the National Research Data Infrastructure), funded by the German Research Foundation. The work focuses on enhancing usability, accessibility, and visual consistency while addressing the specific needs of scientific users. The process is guided by established UI/UX design principles with particular emphasis on ocular health, inclusiveness, and effective color hierarchy. The presentation will showcase selected redesign processes and visual outcomes, demonstrating how problem-oriented and user-centered approaches can improve interaction workflows and information presentation. Feedback gathered from users will also be discussed to evaluate the impact of these improvements on usability and user experience within experimental physics environments.

SIRIUS, a 4th generation synchrotron light source operated by the Brazilian Synchrotron Light Laboratory (LNLS), is starting its second phase of beamline projects and will support the future integration of Orion, a new BSL-4 national laboratory under development. The increasing number of beamlines and experimental modalities has expanded the scale and complexity of scientific software development, creating challenges for maintainability, collaboration, software reuse, and sustainability.

Over the past two years, coordinated efforts have been conducted to improve the sustainability and scalability of software development across SIRIUS facilities. This work presents three main initiatives: (1) the inventory and reorganization of the institutional GitLab environment, covering approximately 1,600 repositories related to production systems, improving discoverability, consolidating documentation, cleaning deprecated code, and facilitating maintenance and software reuse; (2) collaborative development models that centralize reusable and technically complex infrastructure within technical software groups, while directing contributions from beamline operation teams toward facility-specific layers and familiar technologies, such as low code and Python-based solutions, enabling broader collaboration while maintaining code quality and best practices through tutorials, hands-on workshops, and collaborative code review practices; and (3) the ROSETA initiative, which introduced a structured use-case-driven approach for requirements gathering and user experience analysis, using requirements as a decision-making tool for system architecture and helping define clear boundaries between hardware, control system, orchestration, and graphical interface layers.

This work highlights the impact of these organizational, collaborative, and user-centered efforts on maintainability, software reuse, knowledge sharing, and sustainable software evolution for large-scale scientific facilities.

As Serial Micro- CT moves toward routine, high-throughput imaging of large sample collections, the underlying DAQ software must support flexible orchestration, distributed processing, and seamless integration of online services. We present a decentralized software architecture that addresses these requirements through a stream-oriented design. In this framework, an “experiment” is conceptualized as foundational concept for a measurement, representing an ordered collection of “acquisitions”. An acquisition acts as an asynchronous named data source, while processing functionality is encapsulated in independent “addons” that can attach to and detach from remote streams as needed. These addons consume incoming streams, perform online analysis, and optionally emit new streams to downstream services using ZeroMQ . The architecture thereby separates acquisition from processing and allows online reconstruction, quality assurance, live view, and data-writing tasks to run as decoupled compute nodes serving remote procedure calls. Nodes can further exchange information using event-driven framework, enabling coordination across the decentralized system. In the context of Serial Micro-CT, this approach provides flexibility to scale data processing with experimental demands, while maintaining a clear separation of responsibilities. We also outline ongoing efforts to improve reconstruction throughput on GPUs as part of this distributed workflow. Together, these developments establish a software foundation for scalable and adaptable beamline operation.

With recent improvements to the efficiency of collecting single crystal diffuse scattering at synchrotron x-ray sources, large contiguous scattering volumes of diffraction data comprising 100GB can be collected in 20 minutes with high dynamic range and low backgrounds. The Python package, NXRefine, implements a complete data reduction workflow, from ingesting the data, orienting the single crystals, transforming the data into reciprocal
space coordinates, and generating 3D-ΔPDF maps, i.e., maps of real space interatomic vector probabilities. Data from the Advanced Photon Source are now streamed to Argonne’s Leadership Computing Facility for on-demand processing, enabling diffuse scattering to be tracked as a function of parametric variables such as temperature in a few hours as fast as it is collected. With data volumes of several TB a day, it is imperative to have advanced methods of interrogating the data in real time. We have implemented two complementary machine learning approaches. In the first, billions of voxels collected at multiple temperatures are grouped into a finite set of clusters using the Gaussian Mixture Model in order to identify automatically distinctive temperature dependences resulting, for example, from the growth of superlattice peaks at a structural phase transition. This has been implemented in a Python package called X-TEC. In the second, 3D-ΔPDF maps are modeled by generating all the symmetry modes in the entire space group tree using online crystallographic databases. Convolutional neural networks then identify those subgroups that are compatible with the experimental data, allowing the interatomic displacements to be optimized. I will also discuss how the collection of such large datasets enable different contributions of the diffuse scattering to be separated using Independent Component Analysis.

Supported by the U.S. Department of Energy, Office of Science, Basic Energy Sciences, Materials Sciences and Engineering Division.

Amber fossils offer an unparalleled glimpse into ancient ecosystems, preserving extinct organisms in three dimensions with exceptional detail -from external morphology to internal structures. The AMBCAT project aims to publicise the studies of these rare specimens by centralizing high-resolution digital scans in a globally accessible platform. By democratizing access to these invaluable datasets, AMBCAT fosters interdisciplinary collaboration, and ensures the long-term preservation of paleontological heritage.

To achieve this, the project integrates three key components:

• Data Curation & Registration – Systematic cataloging of specimens and datasets in SciCat, ensuring metadata accuracy and traceability.
• Automated Reconstruction Pipeline – A hybrid workflow combining classical computational methods and machine learning-based techniques for denoising, segmentation, and 3D model generation, enabling high-fidelity reconstructions.
• User-Centric Frontend – An intuitive interface that embeds contextual resources to enhance interpretability and facilitate research.

By bridging cutting-edge technology with paleontological research, AMBCAT aims to advance the study of amber fossils and digital infrastructures for natural history collections.

Metadata standardization at complex scientific facilities, such as synchrotron light sources, is a challenging task due to the diversity of instruments, sample types, experimental techniques, acquisition strategies, and scientific questions supported by these facilities. Current approaches for standardizing metadata storage in NeXus files commonly focus solely on the sample-data acquisition stage and do not treat beamline experiments as a multi-stage procedure. As a result, they often fail to provide a complete view of the metadata from all experiment stages, limiting data and metadata organization, traceability, and explainability.
Assonant is a Python-based package developed at Sirius focused on automating transformations and writing collected metadata to NeXus-compliant files. Its main goal is to be a flexible and extensible tool capable of removing the burden of metadata standardization tasks from beamline staff and control system developers by handling those tasks transparently.
This work presents the status of Assonant and its integration within an automatic metadata writer service developed at Sirius. Such integration has quickly enabled data standardization capabilities in Bluesky-based beamline control systems in a transparent and decoupled way. Although the current implementation focuses on Bluesky-based beamlines, it is designed to be extensible beyond this context.
Additionally, this work covers the current NeXus structure proposed for Sirius beamlines and adopted by Assonant to standardize metadata through a multi-stage approach rather than a sample data acquisition-centered model. The proposed structure reflects the multi-stage nature of beamline experiments by explicitly organizing metadata across experiment stages, improving overall metadata organization, traceability, and explainability.

Scientific research publication abstracts are often highly advanced and primarily accessible to scientific communities. To extend accessibility and broaden the audience beyond this community, an audience-specific summarisation of abstracts called Tailored Abstract Translator (TAT) was implemented using open-source Large Language Models (LLMs).

This implementation makes scientific knowledge accessible to a wider group of users, thereby increasing the value and societal impact of the information. The module is open source and built on top of the InvenioRDM framework, making it suitable for adoption in scientific publication repositories.

The European Spallation Source (ESS) operates its own publication portal (https://publications.ess.eu/
), powered by the open-source InvenioRDM platform. As part of the HALRIC funding initiative, an LLM-based module has been implemented to generate simplified summaries of publication abstracts. This module is pluggable and can be configured and enabled within the existing publication portal. A scheduled daily job has been configured to process publications missing an abstract summary and generate new ones automatically. The LLM models are hosted on an internal Azure instance and consumed by these background jobs.

With the on-premise Kubernetes cluster, the implementation of these jobs is scalable, less complex, and provides improved observability. In the future, it is planned to introduce an administrative validation step, allowing generated summaries to be reviewed and, if necessary, regenerated through refined prompting. The quality of the summaries can further be improved through more targeted prompt engineering.

Two levels of summaries are currently generated: “Simple” and “Very Simple.” The “Very Simple” level provides highly accessible explanations, making it easier for non-specialist audiences to understand the abstract and encouraging further exploration of the subject matter.

Light-source facilities now generate data at rates that overwhelm conventional storage paths — HEPS Phase-1 alone averages around 800 TB per day with peaks of 3.2 TB/s, while typical parallel storage sustains only tens of GB/s. Compression is essential, and the community has developed many specialized algorithms — lossless, hand-crafted lossy, and AI-based lossy — each with different trade-offs across compression ratio, fidelity, and throughput. Yet accessing them in real beamline workflows remains difficult: existing tools lack a unified entry point, cannot switch between lossy and lossless modes at runtime, are tied to a single hardware backend, and offer limited Python and HDF5 integration. Scientists therefore spend significant engineering effort just to evaluate or deploy a single algorithm.

We present PhotonZip, a unified compression framework that directly targets these software-level gaps. PhotonZip exposes a single Python API and HDF5 filter as the entry point, supports runtime switching between lossy and lossless modes, and runs portably across CPU, NVIDIA GPU, and AMD GPU backends. As an initial demonstration, it integrates two of our in-house algorithms: MANS, a multi-byte ANS lossless codec addressing the poor compression of byte-level coders on multi-byte integer detector data; and CAIEC, an end-to-end inference–coding co-acceleration framework addressing the throughput bottleneck of AI-based lossy compression.

PhotonZip's architecture is open: more compression algorithms will be progressively integrated. As the algorithm pool grows, we will develop a need-driven recommendation module that, from user-declared targets (ratio, quality, throughput) and the available hardware, automatically selects the most suitable codec — further lowering the barrier for non-expert scientists to benefit from state-of-the-art compression.

Open source: https://github.com/hpdps-group/PhotonZip

SIRIUS, the 4th generation synchrotron light source operated by the Brazilian Synchrotron Light Laboratory (LNLS/CNPEM), currently hosts multiple beamlines with different experimental techniques and continuously evolving software demands. As beamline activities and data processing requirements grow in complexity, challenges related to interoperability, maintainability, software reuse and collaboration between scientific software groups have become increasingly relevant.

Within this context, SIRIUS beamlines have progressively developed ad hoc solutions for data processing and task automation according to their demands. These solutions, often implemented as custom Python scripts and beamline-specific integrations, successfully addressed local requirements but evolved independently, with limited standardization and varying levels of sophistication.

Current efforts at SIRIUS investigate how workflow technologies can be incorporated into a broader control and data architecture designed to be modular, reusable and interoperable across beamlines. Ongoing discussions explore workflow engines not only as automation tools for processing pipelines, but also as potential building blocks for orchestration layers connecting data acquisition, processing, ingestion, analysis and higher-level decision-making components within the Bluesky ecosystem.

The proposed architecture aims to support workflows spanning multiple stages of the experimental lifecycle, including data pre-processing, transformation into standardized application definitions, data ingestion, post-processing, and analysis tasks. Different workflow technologies are currently being evaluated according to their potential roles within the ecosystem, including Prefect, Airflow and Ewoks. These efforts also seek to support future adaptive and autonomous experimental procedures, ranging from conditional execution and automated region-of-interest selection to AI-assisted analysis and decision-making workflows.

Recent advancements in cloud-based architectures are transforming control systems in large-scale scientific facilities. At J-PARC MLF, the instrument control framework IROHA2 has been widely adopted for neutron experiments across many instruments. Although it has served as a standard framework for over a decade, its tightly coupled, synchronous communication model limits flexibility, making it difficult to adapt to evolving experimental requirements.
To address these challenges, we propose a next-generation instrument control framework as a post-IROHA2 system based on a hybrid cloud architecture, currently in the prototype phase. Emerging use cases such as hybrid cloud-based operation, multi-instrument integration, and data-driven and AI-based technologies require a more scalable and flexible approach. The proposed framework adopts a distributed architecture, where instrument and device control remain on the instrument side, while user-facing components and a Redis-based data store for managing experimental information are deployed in the cloud.
This architecture enables secure remote experiments and facilitates integration with advanced cloud-based AI services. At the same time, physical separation between components introduces additional latency and network constraints, which remain important considerations in system design. In this contribution, we present the hybrid cloud-based framework architecture, including performance evaluation, and outline its current status and future development plans.

The K04 project is an ultra-high throughput beamline project for XChem at Diamond Light Source (DLS) that aims to shift the scientific scope of crystallographic fragment screening by harnessing the increased brilliance of synchrotron light and high levels of automation (Fearon et al, 2025). This work describes some of the recent advancements in the automated data analysis of crystallographic fragment screening experiments taking place at DLS. These include bespoke pipelines for fragment hit identification and ligand fitting, the integration of novel automated model building methods, as well as data dissemination, validation and visualisation tools to aid the modern crystallographer working in the field. Together, this work will both facilitate and accelerate current high-throughput screening efforts such as OpenBind, and the fragment-based drug discovery campaigns of the future.

Fearon et al. (2025), Accelerating Drug Discovery With High-Throughput Crystallographic Fragment Screening and Structural Enablement. Applied Research, 4: e202400192.
Pearce, et al. (2017), A multi-crystal method for extracting obscured crystallographic states from conventionally uninterpretable electron density. Nat Commun 8, 15123.

The OSCARS project represents a unique European collaboration that unites Research Infrastructures across the European Science Clusters (SCs) representing the key domains: environmental research, physics and astronomy, photon and neutron, social, and life sciences. At its core, OSCARS focuses on promoting open science and FAIR data principles and addresses this challenge by three primary objectives: consolidation of existing research data services, engagement with a broad network of stakeholders, and the uptake of open science principles.
The consolidation workpackage focuses on interoperability and introducing novel functionalities by metadata enrichment of existing services. One of the key opportunities of the Photon and Neutron community (PaNOSC) is providing Direct DOI Dataset Access (D3A). This initiative is standardizing the access to data directly with the DOI assigned to it. Currently, several platforms offer data storage with a minted DOI, however without a solution for automatic download (other than proprietary). We have successfully developed a proof-of-concept code based on existing approaches (HTTP content-negotiation; metalink). This involves a mixture of modifying existing production software, like the fsspec python module, and developing new code to build a demonstration of the benefits of this approach. Ultimately, D3A can help advance FAIR data practices and interdisciplinary research.

HEPS is scheduled to complete construction and enter operation in 2026. During operation, HEPS will generate massive, heterogeneous, and highly time-sensitive experimental datasets. Experimental workflows such as tomography imaging, diffraction and scattering analysis, and spectroscopy processing require large-scale data reconstruction, complex software environments, and intensive interactive analysis, posing significant challenges to computational resource scheduling, data processing efficiency, and analysis environment delivery.
To address the diversity of experimental data types, heterogeneous analysis workflows, and dynamically evolving computational demands, this work presents a unified data-driven computing platform for HEPS. The platform is built on a cloud-native architecture integrating virtualization, containerization, and high-performance computing resources, forming a comprehensive scientific computing service framework for online monitoring, offline reconstruction, large-scale batch analysis, and interactive data analysis.
The platform provides unified resource management and elastic scheduling of heterogeneous computing resources, enabling efficient utilization of CPU, GPU, storage, and network resources. Standardized data access mechanisms and encapsulated analysis environments simplify software deployment and improve the reproducibility and automation of scientific workflows. In addition to browser-based access, the platform also provides standardized third-party interfaces and programmable APIs, allowing users and external systems to dynamically request, create, and manage customized data analysis environments and automatically execute analysis tasks.
During the HEPS commissioning and trial operation phase, the platform has been stably deployed and validated across multiple beamlines and associated data processing tasks. It has continuously supported users in accessing computing resources and performing remote experimental data analysis.

At ESRF, tomography data processing is supported by a modular software ecosystem designed to connect acquisition, raw data standardization, reconstruction, visualization, automation, and online data availability. Tomotools, brings together several complementary components used across ESRF tomography beamlines to provide users with reliable and reproducible processing workflows.

The chain starts at acquisition level with Bliss-tomo, which integrates tomography procedures into the ESRF Bliss beamline control system. Acquisition metadata, scan descriptions, motor positions, detector information, and experimental context are then used to provide all the information required to perform any downstream processing. Tools such as nxtomo layer provide a common NeXus-based representation of raw tomography data, making the datasets easier to process and exchange among facilities.
Reconstruction computations is handled by nabu, ESRF’s high-performance tomography reconstruction software, which provides key processing features such as dark/flat-field correction, normalization, center-of-rotation estimation, Paganin phase retrieval, helical reconstruction, stitching, and GPU-accelerated volume reconstruction. Around these core processing tools, tomwer provides graphical interfaces and user-oriented workflows, while ewokstomo enables automated and reproducible execution of tomography processing pipelines. Installed on six ESRF beamlines, ewokstomo connects Bliss acquisition metadata with processing workflows and ESRF data services.

Workflow definitions, parameters, software versions, and input/output data references are saved to provide full data provenance. The final goal is to make reconstructed volumes available to users through the ESRF Data Portal together with the corresponding raw data, acquisition metadata, and processing context. In this way, tomography processing at ESRF becomes traceable, scalable, and increasingly integrated into the beamline data lifecycle.

Light sources facilities depend on HDF5 to manage rapidly growing experimental datasets. At the same time, modern data analysis demand higher I/O performance, greater robustness for long-running acquisitions, improved interoperability with legacy scientific formats, efficient support for irregular data, and stronger data protection. This talk presents recent advances in HDF5 software inspired by real-world use cases from light source facilities and designed to address these challenges while improving HDF5 usability in large-scale scientific environments.
We discuss recent progress toward multi-threaded HDF5, including a new multi-threaded Bypass connector for improving I/O throughput on multi-core systems. We also present HDF5 connectors that expose legacy scientific formats including TIFF through standard HDF5 and APIs without requiring data conversion, enabling existing HDF5 tools to access heterogeneous data while preserving native formats and metadata.
Additional topics include support for sparse and variable-length data for efficient storage of irregular scientific datasets; transparent HDF5 encryption for secure scientific data access; and a complete portable single-writer/multiple-reader (SWMR) implementation, including a recovery tool for restoring files after interrupted writes.
Together, these developments improve HDF5 performance, robustness, and interoperability while demonstrating how a carefully designed software architecture allows mature scientific software to evolve to meet new experimental and computational demands.

Modern beamlines and experimental endstations frequently require rapid integration of heterogeneous devices, ad-hoc analysis pipelines, and temporary control-system extensions. While large facility frameworks provide robust infrastructure, researchers and instrument scientists often still face the practical challenge of connecting new detectors, processing live data streams, or exposing derived quantities to existing control environments under severe time constraints.

CAmagick addresses this gap through a lightweight, modular, command-line driven framework for data acquisition, transformation, visualization, and redistribution. Originally focused on EPICS Channel Access environments, the framework is evolving toward protocol-agnostic interoperability, including Tango, Ophyd, HDF5 and Zarr backends. The project targets “Monday-morning integration” scenarios: rapid deployment of functionality, extending the existing beamline infrastructure without requiring extensive engineering effort.

The framework models acquisition workflows as continuously executing pipelines composed of sources, transformation nodes, sinks, and flow-control structures. Users can construct complex real-time workflows directly from the command line or YAML recipes, such as ROI extraction, numerical processing, live plotting, HDF5/Zarr persistence, and publication of derived quantities as EPICS process variables. The same abstractions are available both interactively and through a Python API, enabling seamless migration to reusable workflows.

A key design goal is minimizing the barrier between experimental idea and operational deployment. Typical use cases include detector
prototyping, protocol bridging, rapid IOC creation, online monitoring.

Continuous Integration and Continuous Deployment are well established modern software development and deployment practices that greatly enhance software quality and development velocity.
They ensure that defects are found much earlier in the software development lifecycle, allow exhaustive testing of software on a wide variety of platforms. This poster presents the experiences gained under DAPHNE Task Area 3 at DESY such as: Using Gitlab REST API for greater automation, Native package managers like RPM & Debian and containerization using Podman instead of Kaniko.

Data derived from photon and neutron experiments are key to many scientific breakthroughs across disciplines ranging from medicine to engineering. Developments in data sources, instrumentation, and detectors lead to rapidly growing data volumes of increased complexity. These developments present both great opportunities and significant challenges to the community. Addressing them requires systematic and sustainable research data management where data acquisition, analysis, and availability are critically important.
The DAPHNE4NFDI consortium (DAta from PHoton and Neutron Experiments for the National Research Data Infrastructure) responds to this need as part of the German National Research Data Infrastructure (NFDI) for harmonized, FAIR compliant data practices and a FAIR ecosystem for the photon and neutron communities. DAPHNE4NFDI develops and maintains community-acknowledged reference databases, integrates open-access repositories and databases, and provides technical tools for metadata collection and storage supporting OneNFDI and EOSC, while offering and maintaining FAIR software. These efforts aim to establish international research data standards, train early-career researchers, and promote sustainable FAIR data practices and tools.
DAPHNE4NFDI aims for close collaboration between large scale facilities, academic institutions, IT specialists, and user communities, integrating FAIR principles throughout the entire data lifecycle - from data acquisition to long term preservation, publication, and reuse.

Effective data management infrastructure is essential to keep research data accessible, efficiently processed, and reusable.
This contribution presents the design and implementation data management solution adopted for European XFEL,
which delivers advanced data services through a four layer storage architecture, each addressing specific data handling challenges.

The Online storage layer serves as a high speed cache, managing extreme data rates of up to 15 GB/s per instrument during experiments.
The High-Performance Storage layer supports both near real time data processing and post experiment analysis.
These layers are connected via a 4.4 km, 1 Tb/s InfiniBand link between the European XFEL experiment hall and the DESY computing centre, enabling fast data transfer.

The Mass Storage layer extends capacity for mid term access and in-depth analysis, while the Tape Archive ensures secure long term preservation with a minimum retention period of 10 years.
Together with connected computing clusters, the infrastructure supports processing of up to 2 PB of data per day, demonstrating its scalability and reliability.

The Endeavour programme [1] at the ISIS Neutron and Muon Source will see both new and upgraded instruments brought online in the coming few years. With the use of new detector technology, projected increased data rates and a desire to collect much more data in event mode adapting the existing acquisition system components was judged impractical.

To address these challenges we are adopting a broker based streaming architecture similar to that planned for the European Spallation Source. This new Linux based neutron acquisition architecture will need to co-exist with our existing Microsoft Windows based systems that support the other parts of experiment control. We will describe our planned architecture, the components we have developed as part of the solution, and our future plans.

[1] https://www.isis.stfc.ac.uk/about/future-of-isis/endeavour/

The Beamline and Experiment Control (BEC) is a new solution for beamline operations developed for the Swiss Light Source upgrade (SLS 2.0) at the Paul Scherrer Institute.

We present a deployment strategy for BEC components and utility services, leveraging on-premise Gitea workflows, runners, and Ansible roles/playbooks. Gitea workflows orchestrate automated deployments, integrating version control with continuous integration and continuous deployment (CI/CD) practices. Gitea runners serve as execution agents, running workflow jobs on a virtual machine that acts as an Ansible controller. Finally, Ansible playbooks and roles streamline configuration management and deployment tasks across development and production environments on a network file system (NFS), enabling scalable and efficient deployment processes.

By combining Gitea Actions CI/CD capabilities with Ansible automation, we expect BEC to achieve a scalable deployment mechanism across all beamlines, facilitating adaptation to evolving requirements and ensuring an optimal user configuration interface within the SLS 2.0 ecosystem.

DonkiOrchestra is a data collection and experiment management framework developed at Elettra Sincrotrone Trieste that takes full advantage of the ZeroMQ distributed messaging system. The intrinsic asynchronicity of its advanced software trigger-driven approach allows for concurrency and map-reduce strategies. The software architecture is extremely tunable and scalable, it can meet the needs of a wide range of scientific applications. The framework is in use at Elettra since 2016 but new challenges are coming with the Elettra 2.0 upgrade. The next fourth-generation X-ray source will significantly enhance the brilliance and coherence of the present facility, experiments will need more speed, better synchronicity and more bandwidth. We are using the dark period to improve and test a new version of DonkiOrchestra. This paper presents the software architecture behind DonkiOrchestra 2.0, its experimental applications, performance results and future perspectives.

The adoption of FAIR data principles in Photon and Neutron (PaN) science is promoted by the DAPHNE4NFDI consortium as part of German National Research Data Infrastructure (NFDI) initiative. In the framework of the X-Ray Reflectivity use case, a FAIR data workflow has been established at beamline P08 at PETRA III, DESY. Dedicated data reduction and analysis packages are provided for the liquid reflectometry LISA end-station, employing machine learning models to achieve near real-time prediction of analysis results.

Data processing workflows are automated using the EWOKS engine developed at ESRF, which ensures automated triggering and reproducibility of processing pipelines. The metadata produced by the instrument and analysis are propagated into the Electronic Lab Notebook SciLog and the Metadata Catalog SciCat. Complementary sample information is captured via International Generic Sample Numbers (IGSN) and linked to research data and metadata records.

This system creates a comprehensive set of information records connected to the raw and analyzed research data. Published PaN experimental results can be made available in a dedicated reference catalog and repository curated by DAPHNE4NFDI. Authors can submit high-quality data to provide them for reuse in the wider PaN community.

X-ray free-electron lasers (XFELs) such as SwissFEL produce extremely intense, ultrashort X-ray pulses that enable experiments probing matter on the femtosecond timescale and Angstrom length scale. Unlike storage-ring light sources, XFEL pulses exhibit significant pulse-to-pulse variations in properties such as intensity, spectrum and arrival time. In addition, many experiments operate in destructive regimes where samples are modified or destroyed after each exposure. These characteristics make single-shot correlated data acquisition essential: detector data, beam diagnostics, instrument configuration and metadata must all be recorded and associated for every individual pulse to enable data analysis and interpretation.
We present an overview of the SwissFEL measurement stack, focusing on how experimental workflows are orchestrated across the facility. The architecture integrates individual libraries and services for experiment control, detector acquisition, metadata collection, online analysis, visualization and storage into a unified system that supports both interactive user experiments and automated operation. Experimental data is recorded and buffered for downstream processing, enabling real-time operations such as data reduction, monitoring and visualization, while also ensuring reliable long-term storage.
Users interact with the system through the slic library, which provides a high-level interface for defining devices and controlling experiments. The library communicates with backend services through the sf-daq API that validates requests and coordinates access to the underlying infrastructure via a message broker. To ensure reproducibility and operational reliability, the software stack is deployed through automated pipelines and continuously monitored to minimize downtime during user operation.

FAIR data management at large-scale research facilities requires more than high-level principles: it depends on practical workflows that connect planning, metadata quality, traceability, data reduction, archiving, access, and reuse. At European XFEL, myMdC is being developed as the production service through which these requirements can progressively be translated into operational data-management practice.

This work presents ongoing efforts to use myMdC to implement the facility's scientific data policy through concrete workflow components, including Data Management Plans (DMPs), metadata validation, dataset reconciliation, archival processes, and publication-oriented data handling. A key policy-driven requirement is that data reduction and archival should be completed within six months after the end of beamtime. Addressing this requirement demands earlier planning, more reliable metadata capture, stronger dataset-to-file reconciliation, and clearer workflow transitions from acquisition to downstream processing and preservation.

Recent developments therefore include richer DMP capture, improved dataset and dataset-file consistency checks, refinements to archival workflows, and extensions to metadata and permission handling needed to support policy-compliant lifecycle management. This contribution emphasises that the work is still at an early and exploratory stage: myMdC is currently being used to prototype and evaluate several possible approaches toward more advanced FAIR workflows, access models, and downstream pipelines. While the final operational model is still emerging, this work shows how policy requirements can begin to be embedded into production services to improve traceability, stewardship, and future reuse of scientific data.

DESY accelerator and beamline GUI applications have evolved over many years and now cover a mix of technologies.
These technologies range from older Windows ActiveX, to .NET, Java-, and Python-based, as well as modern web frameworks like React, with specific cases of using LabView and MATLAB GUI tools.
Some of these solutions depend on proprietary software or heavyweight runtimes such as the JVM or .NET, which can complicate long-term maintenance and deployment.
Recent Rust-based frameworks such as Tauri 2 provide a lightweight approach for combining native desktop applications with modern web technologies. This makes it possible to reuse existing web-based user interfaces and gives access to the large ecosystem of open-source web UI component libraries.

In this contribution, we present our experience migrating legacy DESY GUI applications to a hybrid desktop architecture based on Tauri 2. We discuss the overall application structure, integration with existing control-system infrastructure, packaging and deployment on different platforms, and interoperability with legacy components. We also compare this approach with existing Java-based solutions in terms of runtime footprint, startup behavior, and maintainability.

Single-particle imaging (SPI) experiments at European XFEL generate massive volumes of data that require fast and reliable pre-processing for efficient experiment optimization and further downstream analysis.
In this contribution, we present the in-house developed SPI pipeline, which is available to the users of European XFEL. This is deployed in both online and offline settings, and includes automated hit finding, hit-rate estimation, background characterization from empty frames, and estimation of particle size and scattering center using a spheroid-based model.
The online component, which enables real-time feedback during experiments, is implemented in Karabo as a set of extensions to the existing pipeline correcting data generated by custom-built detectors. The offline component is implemented as a standalone Python library, which is also integrated into the offline correction pipeline, allowing processing results to be automatically stored within facility-generated corrected data files, and therefore making them readily accessible for subsequent analysis stages.
Many processing components of the SPI pipeline are based on established community approaches, while additional algorithmic solutions were developed to enhance robustness and stability of certain processing steps. As the pipeline is deeply integrated into European XFEL’s data workflows and tools, a seamless use of processing results in subsequent data analysis and reduction workflows is granted.
The SPI pipeline is adopted by experimental teams, and has received positive feedback from our users. Future developments include the exploitation of the online SPI pipeline for on-the-fly data reduction, so as to reduce storage pressure and improve experiment efficiency.

The neutron macromolecular crystallography instrument (NMX) at the European Spallation Source (ESS), is now under cold commissioning. As ESS evaluates user interface options for NMX scientific users, MXCuBE-Web — the latest generation of the widely adopted data acquisition software MXCuBE (Macromolecular Xtallography Customized Beamline Environment) — has been selected as the primary UI, marking its first-ever deployment for neutron diffraction.
MXCuBE is a free, open-source project started at ESRF in 2005, built through a growing international collaboration now spanning over 15 institutes including ESRF, MAX IV, HZB, EMBL, DESY, ELETTRA, LNLS, NSRRC, ANSTO, and ESS itself. Its Web incarnation, MXCuBE-Web, runs in any modern browser and is built on a Python/Flask REST backend, SocketIO bidirectional communication, and a React/Redux frontend.
Bringing MXCuBE-Web to NMX offers compelling advantages: a familiar MX user experience for easier experiment control, and a tighter collaboration and contributions back to the scientific and open-source software communities.
These opportunities also come with challenges, such as integrating MXCuBE with NICOS (ESS's standard instrument control layer, that communicates with the EPICS control system), and adapting UI widgets for neutron-specific workflows. Both fronts are progressing and will be discussed in this work, together with current status and lessons learned from this effort.

Automated data analysis at synchrotron beamlines is often limited by the
complexity of managing many processing parameters across large sets of related
measurements. Although the conversion of raw detector data into physically
meaningful representations is largely deterministic, existing approaches
typically embed parameter handling directly in analysis scripts, making
automation, reusability, and reproducibility difficult. We introduce the Spice
concept, a declarative approach for managing analysis parameters in a
scan-aware manner.

Spice separates parameter definitions from the analysis
code and organises their validity along the chronological sequence of
explicitly typized scans. Update boundaries can be automatically assigned
by the machine and refined by the user. The versioned update
mechanism has is built around a hierarchical built-in defaulting mechanism.
This allows parameters to be applied consistently across multiple scans
with drastically reduced user intervention, while still supporting local
adjustments where necessary.

Using X-ray diffraction as an example, we demonstrate how
Spice enables a clear separation between routine, instrument-specific data
preparation (“cooking”) and the subsequent scientific interpretation. We
describe an implementation within the Nx5d framework and discuss how the
approach supports reproducible analysis, automated reprocessing, and
provenance tracking.

The Spice concept provides a general architectural pattern for scalable
and maintainable data processing at synchrotron and other large-scale
experimental facilities, in particular where enabling second-pass
data analysis after FAIRdata/OpenData principles is a priority.

The OSCARS project represents a unique European collaboration that unites Research Infrastructures across the European Science Clusters (SCs) representing the key domains: environmental research, physics and astronomy, photon and neutron, social, and life sciences. At its core, OSCARS focuses on promoting open science and FAIR data principles and addresses this challenge by three primary objectives: consolidation of existing research data services, engagement with a broad network of stakeholders, and the uptake of open science principles. A key initiative addressing these objectives are the Cascading Grants to support small-scale Research Projects focusing on various tasks around the FAIR principles from all five SCs. These Research Projects are facilitated by and aligned with the European Open Science Cloud and prove to be a very successful bottom-up approach for open science adoption.
In this contribution, we present the activities of OSCARS from the focus on the Photon and Neutron Open Science Cluster (PaNOSC), identifying the benefits to our community. We will highlight the activities of the three work packages that engage directly with the SCs, including the establishment of domain-oriented community-based competence centers, enhancing scientific discovery by composability, and the connection to national and international projects in Europe.

Machine learning (ML) tools hold the promise of transforming scientific research by accelerating analysis and enabling new experimental capabilities, including real-time data analysis, informed decision-making during measurements, optimized experimental conditions, and ultimately closed-loop experimental workflows. While the development of functional algorithms represents an important step, many approaches remain at the proof-of-concept stage, and integrating and validating them under real experimental conditions pose substantial challenges. In the context of reflectometry, most prior automation efforts have focused on X-ray reflectometry (XRR), although neutron reflectometry (NR) can indeed benefit to the same extent.

We develop the ML Python package reflectorch, designed for the analysis of XRR and NR data. A key advantage is the incorporation of prior knowledge on the sample during both training and inference [1]. It has already been successfully incorporated into a closed-loop experimental workflow for XRR [2]. Here, we report the first ML-based workflow for real-time NR analysis deployed at the D17 beamline at the Institut Laue-Langevin (ILL) [3]. We integrated the reflectorch package into the data acquisition workflow using the facility's IT infrastructure. The ML-based analysis workflow is periodically triggered, achieving an inference time up to two orders of magnitude shorter than conventional analysis software. This allows the physical parameters of the sample to be tracked with high temporal resolution, supporting continuous monitoring via a graphical user interface and facilitating data-driven adjustments throughout the experiment. Furthermore, a feedback connection to the instrument control has been established, providing the foundation for a closed-loop experiment.

1. V. Munteanu et al. (2024). J. Appl. Cryst. 57, 456-469
2. L. Pithan et al. (2023). J. Synchrotron Rad. 30, 1064-1075
3. A. Rentzsch et al. (2026). J. Appl. Cryst. 59, in print

SIRIUS is the 4th generation synchrotron facility operated by LNLS/CNPEM, with multiple beamlines currently in operation and others under commissioning. To support the growing need for scalable, maintainable, and user-centered experiment control interfaces across these beamlines, sophys-web was developed as a monorepo for building web applications integrated with the SIRIUS Ophyd and Bluesky Utilities (sophys) ecosystem. It provides experiment control UIs across SIRIUS beamlines with reusable components, API clients, and utilities.

Recent developments include two beamline apps: for the SAPUCAIA beamline, dedicated to Small-Angle X-ray Scattering (SAXS), and the QUATI beamline, focused on X-ray Absorption Spectroscopy (XAS) along with new additions, including a sample metadata store, PVWS client for real-time PV updates and Prefect/Tiled integration for data workflow orchestration and visualization.

As the project grows beyond initial development, there is a need for a more scalable development and deployment model, moving away from the initial monorepo architecture, which prioritized rapid prototyping, toward an architecture better suited for open-source development. This includes publishing more general-purpose packages (such as Bluesky HTTP Server and PVWS clients), an app generator for scaffolding new applications and a customizable component library.

The increasingly large volumes of data generated at different synchrotron sources, such as PETRA III and the European XFEL, require a crucial shift from traditional bulk data transfers toward remote and slice-based selective data access. As part of the PHOTONIC (Photon Science - Harmonized Online Tools for Open, Navigable, Interactive Computing Framework) project, this work focuses on developing client-side libraries to integrate a unified API with established scientific software pre-existing at the photon sources. This project represents a framework that would enable researchers to perform real-time data slicing and the corresponding ad-hoc analysis directly within familiar environments such as H5Web and silx-kit [1]. Using the Liquid Interface Scattering Apparatus (LISA) instrument at the beamline P08 (DESY) as a primary use case, we aim to demonstrate how this integration facilitates the rapid evaluation of X-ray reflectivity (XRR) and grazing-incidence X-ray diffraction (GIXD) datasets [2]. The developed client library would include the Tiled data access service in the backend and would be utilized for transparent interoperability between different research facilities [3]. This would enable users to overlay and compare experimental datasets regardless of the underlying storage format. This contribution represents the architectural design of the client-side modules and discusses the implementation of FAIR (Findable, Accessible, Interoperable, and Reusable) data principles through professionalized software standards, including automated testing and standardized metadata handling. These developed tools would eventually lower the barrier for non-expert users and provide a scalable solution for remote scientific collaboration. This work complements the efforts of DAPHNE4NFDI [4] in advancing metadata catalogues and interoperability by enabling practical, fine-grained access to data that has been made findable.

Online analysis refers to data analysis pipelines based on processing of live data streams in near-real-time during an experiment. Such analysis provides rapid feedback, helping in understanding outcomes and optimizing steering of the experiment. At European XFEL, the high data rates generated by MHz detectors make live data processing challenging in terms of both computational intensity as well as network transfer. To tackle these challenges, we must utilize an edge computing approach - moving data processing and reduction upstream closer to the source rather than leaving processing of full data to downstream external monolithic analysis software.

Existing pipelines developed for real-time detector data correction provide an ideal integration point for upstream analysis processing. These correction pipelines run within the Karabo control system and are deployed on dedicated compute nodes which are connected via fast InfiniBand fabric and typically have several high-performance GPUs available.
To exploit excess compute capacity left over after data corrections, we provide a number of programmable integration points within these pipelines. These allow for custom kernels for processing the data as well as for making data reduction decisions on the fly. This opens the door for comprehensive online analysis on the full data streams; the results can be visualized within Karabo or streamed to external analysis and visualization suites.
Karabo integration is an additional benefit of these integration points: Moving analysis parameters and (optionally) visualization inside the control system eases the overhead associated with deployment, setup, and configuration - and allows for future integration with other development projects such as automation and optimization.

We present an overview of the architecture described above along with examples of analysis pipelines already developed and used for various experimental techniques.

It is essential to have effective tools to diagnose emerging issues for the stable operation of control systems in a large experimental facility. This task becomes even more challenging in distributed systems, as the source of problems may appear in different subsystems and at different times.

The control system at MAX IV is based on Tango Controls, and several tools are used for monitoring and troubleshooting. Data from Tango devices is archived in a database and can be explored via ArchViewer, an interactive time-series data analysis tool. The same data is also used in Grafana dashboards, where they can be combined with Prometheus metrics to provide a more comprehensive view of system behavior.
Logs from Sardana and other Tango devices are collected in Graylog. This system provides centralized remote access to activity history and error messages across the entire facility.
The snapshot system enables switching between different operation modes while minimizing human error.
The electronic logbook Elogy is used by the operation support engineers to record information about incoming issues, making it easier to resolve similar problems in the future.

The combination of metrics, logs, snapshots, and operator records provides a useful set of tools that help identify the causes of failures faster and improve understanding of control system behavior, leading to more reliable operation.

Experimental and operational activities at European XFEL depend on timely communication between instrument scientists, users, and support teams. While electronic logbooks have traditionally served as passive records, day-to-day facility operation requires a platform that also supports structured communication, coordinated team access, and automated information flow.

This contribution shows how myLog is evolving into an operational layer for experiment and support workflows at European XFEL, combining structured communication, automated information exchange, proposal-based logbook provisioning, and bot-supported notifications within a single collaborative platform. Following beamtime scheduling and acceptance in myMdC, logbooks can now be created automatically with proposal experiment teams and support staff already assigned, reducing administrative overhead and improving consistency. In parallel, myLog supports the ongoing migration of experimental and operational logbooks from the legacy ELOG service into a unified environment.

Recent developments include the operation of a central bot server, the deployment of bots for notification and information-routing tasks such as "Copy by tags", technical and functional improvements in DAMNIT and Karabo integration, Zulip upgrades with folder support, and the export of logbook streams through a Jupyter-based tool. myLog also now includes an in-house hosted LLM-based support bot that assists support colleagues in operational tasks.

X-ray Photon Correlation Spectroscopy (XPCS) inverts the dynamic behavior of materials at the nanoscale by analyzing the "photon intensity fluctuations" in synchrotron X-ray scattering signals. The multitau algorithm (multi-tau algorithm) calculates lag times at different time levels through a "hierarchical grouping" strategy, which can cover a wide time range while controlling computational complexity. When computing the one-time correlation function and two-time correlation function, the input data are ultra-large-scale datasets (n>10000), leading to low computational efficiency that often requires several hours or even days to complete. In this study, the acceleration of correlation function calculations is realized based on CPU parallel computing and GPU parallel computing, and the methods of data partitioning and computing sequence allocation are described. Experimental results show that CPU parallel computing achieves an average 10-fold speedup, and GPU parallel computing achieves an average 80-fold speedup. Meanwhile, an XPCS data processing platform is presented, which supports interactive data processing.

DREAM is one of the diffractometers in construction at the European Spallation Source (ESS), built by the consortium Forschungszentrum Jülich (Germany) and Laboratoire Léon Brillouin (France).
To ensure smooth operation even for the first-time users, the ESS Data Management and Scientific Computing centre is developing an integrated data pipeline linking all the steps of scientific data processing. To test this pipeline during the construction phase of ESS and of its instruments, we use instrument simulations. Here we report on preliminary implementations of the live and post-beamtime data processing pipeline for powder diffraction data from raw NeXus files to data archiving.

Neutron and x-ray scattering experiments traditionally rely upon
histogrammed data sets, which are analysed using least-squares curve
fitting of multiple probability distribution components to quantify
separately the various scientific contributions of interest. The main
advantage to this approach is the relative ease of deployment due to
its intuitive nature. Despite the great popularity of the method,
there are known drawbacks relative to alternative methods, such as
systematic errors, biases, and instability in some scenarios that are
common in neutron scattering. Improvements over the base methods
include dynamic optimisation of histogram bin width and the
application of modern numerical optimisation methods that can be less
unstable when pushed to the edge of the performance envelope.

In this new study, we demonstrate analysis of neutron scattering data
entirely on an event-by-event basis, without resorting to any kind of
numerical integration, histogramming, or least squares fitting. This
method is demonstrated first using synthetic events for a standard
distribution, to establish the method in a controlled environment;
then in synthetic small angle scattering application, with more
realistic features; and ultimately deployed on event data measured by
the ARCS spectrometer at the Spallation Neutron Source, TN, USA. This
wide range of tests also shows the broad applicability of this method.
The benefits of this approach are revealed: orders of magnitude
greater efficiency (i.e. fewer data points required for the same
parameter accuracy). The efficiency gain can be viewed as either a
significant increase in scientific output, or equivalently reaching
the finest possible time resolution in kinetic studies. Considering
the cost of neutron scattering beam time, it may also result in
significant operational cost savings. The main drawbacks are an
increase in computation time, and perhaps a less intuitive analysis
method for some users.

The High Energy Photon Source (HEPS) is a fourth-generation synchrotron radiation facility scheduled to enter formal operation in June 2026. Enabled by high brightness and advanced detector technologies, HEPS is expected to generate more than 200 PB of experimental data annually across 14 Phase-I beamlines. The wide range of experimental techniques, including imaging, diffraction, scattering, and spectroscopy, produces highly heterogeneous data in throughput, volume, and processing complexity, posing significant challenges for large-scale scientific data processing.
To address these challenges, we developed DAISY (Data Analysis Integrated Software System), a general-purpose scientific data processing framework. DAISY provides high-throughput data I/O, unified access to heterogeneous data sources, and support for elastic and heterogeneous computing. Based on DAISY, multiple domain-specific scientific applications have been developed and deployed on HEPS beamlines, where they support beamline commissioning, beam tuning, and process validation through online and near-real-time data processing. Joint data analysis approaches across multiple experimental methodologies are being actively explored to support increasingly complex scientific use cases, and future work will further incorporate artificial intelligence methods to enhance the efficiency, accuracy, and automation of scientific data processing.
This contribution reports on the design and implementation of the DAISY framework, the application of DAISY-based scientific software across multiple HEPS beamlines, and data processing practices and experiences toward the formal operation of HEPS.

HoToPy is an open-source Python-based toolbox for X-ray holographic and tomographic data reconstruction. It features state-of-the-art phase retrieval algorithms for both the holographic and direct contrast imaging regimes, including nonlinear approaches and extended options for regularization and constraint sets. By combining automatic differentiation with provided fully differentiable forward operators, HoToPy enables straightforward extensions and modifications to the phase retrieval problem. Furthermore, the toolbox includes auxiliary functions for (iterative) tomographic alignment, image processing, and simulation of imaging experiments. Through its modular design and user-friendly interface, HoToPy can be easily integrated into existing pipelines, used as a building block for new applications, and applied in classroom teaching. Implemented using the PyTorch framework, HoToPy supports cross-platform, high-performance computations on both GPU and CPU, and enables seamless integration into machine learning workflows. We demonstrate HoToPy’s capabilities and discuss current developments and challenges in data reconstruction using the nano-tomography instrument ‘GINIX’ at the P10 beamline of the PETRA III storage ring (DESY, Hamburg).

The European XFEL is a megahertz repetition-rate facility producing extremely bright and coherent pulses of femtosecond duration. With its AGIPD, LPD and DSSC detectors specifically built to operate at these repetition rates, the amount of data generated in the context of user experiments can exceed hundreds of gigabits per second, resulting in tens of petabytes stored every year. These rates and volumes pose significant challenges both for facilities and users thereof. In fact, if unaddressed, extraction and interpretation of scientific content is hindered, and investment and operational costs quickly become unsustainable.

These challenges are addressed by the facility on both a policy as well as a technical level. A new Scientific Data Policy has been adopted to clarify the individual responsibilities in data retention and curation. Data reduction becomes a required and early step in the data lifecycle of experiments. Here, the facility aids its users by developing data reduction pipelines, tools and by providing dedicated support.

In this work, we present the adopted data reduction strategy and its integration into the existing facility systems and workflows. These range from end-to-end solutions for established techniques to extendable frameworks in case of unique requirements. Depending on the use case, reduction may happen for streamed data before hitting the disk or after comprehensive analysis.

The massive and diverse experimental data generated by the High Energy Photon Source (HEPS) poses severe challenges to data processing pipelines, rendering traditional fixed serialization and compression strategies inefficient. To address this, we introduce LightPacker, an adaptive decision-making system that automatically applies the optimal serialization and compression algorithm combinations based on specific data characteristics. LightPacker operates through a five-module architecture: a unified Algorithm Library, Feature Analysis for real-time data profiling, Offline Analysis for historical benchmarking, Online Evaluation for rapid algorithm matching, and Decision Execution. By leveraging data features (e.g., sparsity, structure) and sampling techniques, the system minimizes decision overhead while ensuring accuracy. Experimental results demonstrate that LightPacker significantly outperforms fixed strategies in typical HEPS scenarios, effectively reducing network bandwidth occupancy, improving preprocessing efficiency, and minimizing storage costs for large-scale scientific facilities.

At the Spallation Neutron Source (SNS) and High Flux Isotope Reactor (HFIR), users can access automatically reduced data both during experiments (“live reduction”) and soon after data collection completes (“autoreduction”). Users can also monitor experiment parameters and view an instrument-specific report from the automated reduction through a web interface.

This presentation introduces the automated data reduction workflow at SNS and HFIR, including how it is configured by Instrument Scientists, how users can get feedback during ongoing experiments, and the software applications involved. We will review the current state of the automated reduction and propose future enhancements to improve flexibility, e.g. in computing resources, provenance and resilience. Finally, we will briefly discuss the role of autoreduction in automated experiments and new automated workflows.

HDF5 files are common at large-scale facilities. At ESRF, they have become the standard format provided to users. However, accessing and visualizing the data they contain can be challenging, as HDF5 files are binary and organize information in hierarchical groups and datasets.

H5Web is an open-source, web-based viewer developed at ESRF to explore HDF5 files and generate efficient visualizations with WebGL. It supports the NeXus standard, enabling specific views for scientific data. Files can be accessed from remote servers, loaded locally, or even opened directly in the browser for immediate inspection.

At a previous NOBUGS conference, I presented several use cases for H5Web:

• myHDF5: viewing files by uploading them directly in the browser
• vscode-h5web: reading HDF5 files within VS Code
• ESRF Data Portal: allowing users to explore their data without downloading it

This time, I will focus on the ESRF Data Portal integration. I’ll demonstrate how we leverage H5Web’s modular architecture to build custom visualizations for specific techniques, and how we have introduced new features by aligning with complementary standards such as silx and NetCDF.

In pump-probe experiments at the Single Pulse Facility (SPF) at MAX IV, the dynamics of ultrafast phenomena are gauged by gathering 2D diffraction patterns at stochastically random varying delay-times, or ping values, between
the pump and probe pulses. To build statistics and aid in data reduction, data frames are sorted based on their ping value and binned together by summing the corresponding 2D patterns. However, these ping values arrive out of sequence.
Previously employed scripts for sorting and binning required to be run after data acquisition had been completed and could take longer than the experimental run. Additionally, there was no visual aid or guidance for choosing appropriate binning parameters a priori.

To help mitigate these shortcomings, we developed the Ping-Map, a python-based graphical user interface (GUI), communicating over ZMQ with an efficient C++ backend, which is capable of sorting and binning data frames from raw files, even as they are being written to.
Thus, the post-processing can be done during data acquisition, or after, on the order of seconds to minutes. The GUI allows for interactive selection of bin parameters and provides visual constructs that illustrate the progress of the binning, as well as intermediate
bin-averaged data frames and difference maps thereof, for real-time user-interaction.
The modularized backend allows for the binned data to be fed into further downstream post-processing, such as azint (see “Numerically Consistent Azimuthal Integration for 2D pixel detectors for orientated data" – submitted by Stuart Ansell).

[Note to organizers: if the latter is rejected, please remove the last sentence of this abstract.]

MHz X-ray Microscopy is a becoming a major type of user experiments at the European XFEL (EuXFEL) beeing performed at different hard X-ray SASE instruments. Recently, MHz X-ray Microscopy / Phase Contrast imaging setup was fully integrated and implemented at the EuXFEL within the Karabo control system. This includes controls for motors, timing system for triggering and synchronization, fast and slow cameras, analogue to digital converters, fast X-ray detectors, and data acquisition (DAQ) system. All the aforementioned devices and the DAQ system work seamlessly together. Further a more complex setup like the MHz Tomoscopy was integrated. With interest for these imaging methodologies increasing at other free electron lasers and synchrotrons, we have started to perform experiments at other facilities like ESRF and LCLS. However, these facilities lack a fully integrated control and DAQ infrastructure for fast X-ray Microscopy. With this in mind, we have implemented a standalone Karabo control and DAQ infrastructure for MHz X-ray microscopy methodologies. This setup includes all the devices that are integrated in Karabo for X-ray imaging experiments, including the fast cameras, timing system, direct conversion detectors like the Jungfrau 1M, motor controllers including 70+ axis controller for MHz Tomoscopy prototype, and DAQ. Initial tests indicate reliable performance.
This contribution describes the deployment and maintenance of the Karabo control system for this standalone setup and application possibilities at other facilities.

Operando X-ray absorption spectroscopy (XAS) at synchrotron beamlines requires careful monitoring of both spectral data and sample-environment conditions throughout an experiment. During catalysis studies detecting anomalous process states as they develop is critical to ensuring meaningful results. An intelligent, automated analysis pipeline that processes data in near real-time can support researchers by enabling timely intervention rather than post-hoc discovery of problems.

The analysis pipeline presented here forms part of the ROCK-IT [1] project, and leverages a near real-time processing infrastructure to perform automated calibration, normalization, linear combination analysis utilizing the Larch [2] library; alongside continuous monitoring of sample-environment parameters such as temperature profiles and gas composition. This pipeline can issue alerts to beamline scientists and provide direct feedback to the experiment control software to adjust input parameters autonomously. The foundation of this system is Asapo [3], a data streaming framework developed at DESY, and the same infrastructure is designed to scale to experiments with significantly higher data rates at modern and upcoming beamline configurations.

A key use case is the detection of unwanted water formation during operando catalysis studies, an artifact that can arise when specific combinations of temperature and gas flows create conditions for unintended side reactions. The pipeline identifies water formation and triggers corrective actions, either through operator notification or automated feedback to the sample-environment control system, which in this case is Bluesky [4]. This demonstrates how intelligent data pipelines can guarantee sensitive measurements and ensure data of high quality in complex experiments.
References
[1] https://www.rock-it-project.de/
[2] https://github.com/xraypy/xraylarch
[3] https://asapo.pages.desy.de/asapo/
[4] https://blueskyproject.io/

Mature solutions exist for direct control of synchrotron beamline hardware; many advanced use cases now depend on automation and parallelisation. However, these features come with higher complexity and risk: race conditions may cause collisions, wrongly interpreted data, or other errors.

Building on existing tooling which allows spawning Beamline Experiment Control (BEC) clients within isolated scripts, we developed a system for responding to changes in beamline conditions with arbitrary actions executed on a BEC client. The system has access to any information in the BEC message broker as input for its conditions. This architecture allows consistent validation and safety checks.

Our goal is that simple automation tasks should be easy, while complex tasks are possible and assembled from modular combinations of simpler tasks. Envisaged uses include pausing scans when beam is lost and restarting them when it is restored and sending messages to external services (e.g. MS Teams) when chosen signals reach defined values. These can be chosen by beamline staff through configuration changes.

At the other end of the spectrum, actions can include triggering a centring routine when a sample is loaded, or data collection when a sample is centred. Combining all of the above leads to a fully automated experimental workflow, responding dynamically to beam conditions, and notifying users of any errors through direct messages.

Synchrotron X-ray Micro-CT is an invaluable tool for non-destructive 3D morphological visualization of different kinds of small samples in high resolution. To extend this capability to large-scale comparative studies, or the systematic digitization of museum specimens, a radical shift in imaging workflow is required. We present a comprehensive, highly automated pipeline for “serial tomography” that supports routine high-throughput 3D imaging of large sample collections. The approach consolidates robotic sample preparation, self-directing scanning procedures, GPU-accelerated online 3D reconstruction, and AI-assisted quality assurance protocols within a coordinated workflow. Efficient processing is achieved through a decentralized DAQ pipeline, while metadata are centrally managed from sample preparation onward to maintain traceability and interoperability across all stages. The system consists of a UR20 robotic arm equipped with a suction cup gripper and placed on a motorized gantry. The sample tray can facilitate up to 1000 vials of volume 0.2 ml. The robotic arm contains a datamatrix code reader for sample identification. Current sample throughput in a high-quality setting is 15 samples/hour with pixel size 1.3 micron. The developed workflow offers the scientific community access to high-quality 3D data and supports comparative research addressing principal questions across life sciences and material research.

I will describe our system for fully real-time processing of X-ray serial crystallography data streamed directly from the detector, with no intermediate disk storage. Based on the CrystFEL software [1] in combination with the ASAP::O high-performance data framework developed at DESY [2], the system is capable of indexing and integrating more than 1000 frames per second using only a single compute node. It has been deployed at the synchrotron light source PETRA III, where it has been reliably used in a series of user experiments [3]. By finding and removing the largest performance bottlenecks in the software and its environment, we achieved a speed-up of around 50 times since the beginning of the project. Further speedups have since been made by measures such as addressing lock contention and reducing memory allocations.

A prerequisite for real-time processing is that the experimental geometry is accurately known before the experiment. The methods so far employed for determining this geometry have been too slow to run very frequently, or have had bias problems. However, an algorithm in particle physics exists to solve an analogous problem with much higher performance [4]. This algorithm, named Millepede, has been successfully transferred from particle physics to X-ray crystallography [5]. Even when fitting across tens or hundreds of thousands of diffraction patterns, the Millepede method calculates an updated detector model in a negligible amount of time, allowing frequent calibration updates to be a part of our high-speed data processing system.

[1] T. A. White, R. A. Kirian, A. V. Martin, A. Aquila et al. J. Appl. Cryst. 45 (2012) p335.
[2] https://asapo.pages.desy.de/asapo/
[3] T. White, T. Schoof, S. Yakubov, A. Tolstikova, et al. IUCrJ 12 (2025) p97.
[4] V. Blobel, Nuclear Instruments and Methods in Physics Research A 566 (2006) p5.
[5] T. A. White, J. Appl. Cryst. 59 (2026) p594.

At MAX IV we have established a high-performance data acquisition (DAQ) system capable of keeping pace with the high rate detectors enabled by the brightness of the fourth-generation source. Photon counting and charge integrating detectors, together with sCMOS cameras, are unified within a single DAQ framework. Detector data are streamed to a central Kubernetes cluster mounting IBM Storage Scale (GPFS) storage, with Tango providing overall system control. Live feedback from all detectors and cameras is available as standard, and the system is extended with on-the-fly data reduction capabilities through the "Dranspose" framework - a horizontally scalable, distributed data analysis pipeline. We give an overview of the detector suite at MAX IV and describe the DAQ and processing infrastructure, with emphasis on its demonstrated performance for live data streaming and on-the-fly reduction across a range of applications.

Python Accelerator Middle Layer (pyAML) is a joint technology platform for design, commissioning and operation of particle accelerators. It is developed, benchmarked and maintained collaboratively by an open-source community driven by accelerator facilities around the world. pyAML is designed to be fully control system agnostic. External EPICS, Tango and Ophyd-Async backends are already available. DOOCS backend is an ongoing development. One of the biggest challenges is handling the maximum of lab specificities such as complex magnet models or various lattice naming convention. To achieve this aim, pyAML mainly defines various abstract classes and model to provide access to low level accelerator devices. pyAML also use a dynamic configuration model based on pydantic. pyAML offers various tuning and measurement tools such as orbit response matrix measurement and orbit correction or beam based alignment. pyAML is still in a development and specification phase.

Tango-Controls is an object oriented control system used in many synchrotrons, telescopes, laser facilities and more.
Since its creation in 1999 at the ESRF, many institutes joined this free software development effort and are contributing by providing developers, financial support and bright ideas.
The first TANGO 10.x versions have been released recently, providing very interesting new features like the OpenTelemetry support, additional versioning information and alarm events.
PyTango, the Python binding, has switched from Boost-Python to pybind11.
In the last 2 years, several Special Interest Group (SIG) meetings took place to improve the current documentation, to discuss important topics like the Encryption, the TANGO Polling Loop and to prepare the next TANGO Long Term Support version. This last meeting initiated a refactoring work to use the Pointer to IMPLementation (PIMPL) idiom in cppTango, the Tango-controls C++ library, to ease the evolution in future versions.
The Tango-Controls collaboration contract has been renewed for the period 2026-2030.
A new call for tender for Outsourcing of software development for the Tango-Controls community has been published in February 2026, to ease the process of subcontracting tasks for the maintenance and improvements of the Tango-Controls core and tools while ensuring a bright future for the Tango-Controls community.

The recent emergence of agentic workflows in scientific research has highlighted the urgent need for high-quality AI-ready experimental datasets for model training and validation. To be AI-ready, these datasets should be described by rich, machine-readable metadata and provenance, following FAIR data principles. Metadata and provenance perform three functions in agentic workflows: they allow datasets to be auto-discovered, they provide context for interpretation of datasets, and they help establish trust in AI models.

At the Cornell High Energy Synchrotron Source (CHESS), we have developed the FAIR Open-Science Extensible Data Exchange Network (FOXDEN), a suite of lightweight, modular data services that supplements researchersʼ existing experimental workflows, allowing them to easily record metadata, provenance, and other research artifacts in real time. FOXDEN is specifically designed to handle large, unportable datasets and heterogeneous use cases. It helps scientists turn their research artifacts into annotated, AI-ready datasets and publish them with Digital Object Identifiers. FOXDEN is also federated for use at multiple sites and facilities. We describe FOXDENʼs architecture and deployment status at both CHESS and the National High Magnetic Field Laboratory, and we present a blueprint for incorporating it into agentic scientific workflows.

European XFEL is a photon source delivering highly coherent and extremely
short X-ray pulses at megahertz repetition rate. Drawing on lessons learned
during the first six years of operation, in particular the challenge of huge data
volumes, and on insights from participation in FAIR[1] scientific data manage-
ment initiatives, the Scientific Data Policy (SDP)[2] has been revised. In this
work, we present the key features of the SDP, which applies to proposals sub-
mitted from 2025 onward.
Among the newly introduced SDP features are: (I) a data retention and cu-
ration concept based on expanded data category definitions, establishing work-
flows of mandatory data reduction[3] and enabling the provision of open data
collections; (II) the definition of supported data file formats to foster interop-
erability; (III) recommendations for the adoption of FAIR principles along the
data lifecycle, such as proper annotation with metadata; and (IV) data manage-
ment plans (DMPs) as a practical mechanism to accommodate these aspects.
DMPs are part of operational practice since 2025. They follow a pragmatic,
proposal-specific approach focusing on requirements and arrangements around
the different stages of the data lifecycle, starting with resource management
and user support for beamtimes. DMPs cover workflows for both discussion
and documentation of agreements. The data management platform myMdC
integrates administrative DMP information as well as dynamically adaptable
DMP content.

References
1. M. D. Wilkinson et al. (2016). The FAIR Guiding Principles for scientific
data management and stewardship. DOI: 10.1038/sdata.2016.18
2. European X-ray Free-Electron Laser Facility GmbH (2023). Scientific
Data Policy of the European X-Ray Free-Electron Laser Facility GmbH
(Version 2). DOI: 10.22003/XFEL.EU-TR-2025-001
3. E. Sobolev et al. (2024). Data reduction activities at European XFEL:
early results. Front. Phys. 12, DOI: 10.3389/fphy.2024.1331329

China’s High Energy Photon Source (HEPS) is the first national high-energy synchrotron radiation light source and one of the world’s brightest fourth-generation synchrotron radiation facilities. It started to operate and conduct user experiments at the end of 2025.

The 14 beamlines for the phase I of HEPS are projected to produce more than 300PB raw data annually. Efficiently storing, analyzing, and sharing this huge amount of data presents a significant challenge for HEPS.

To ensure the accuracy, availability, and accessibility of the massive data generated at HEPS, we developed a dedicated data management and data service system named DOMAS. It is designed to automate the organization, transfer, storage, distribution and sharing of the experimental data. This paper first introduces the overall profile of HEPS and the progress of its construction. It then outlines the architecture and data flow of DOMAS, followed by an explanation of key technologies and newly implemented functional modules. For instance, we illustrate the automated data tracking process under a tiered storage policy and demonstrate how DOMAS ensures consistent user authorization for data access on both the website and the computing cluster. Furthermore, the effectiveness of the system deployment in the HEPS production environment is presented. Finally, we offer some reflections on the design of data management systems tailored to the needs of advanced light source facilities.

The Shanghai HIgh repetitioN rate XFEL and Extreme light facility (SHINE) will support experiments such as serial crystallography, coherent diffraction imaging, and X-ray spectroscopy. These experiments will generate massive, heterogeneous and time-correlated datasets with scientific and administrative metadata. FAIR data management is therefore essential for facility operation, long-term preservation, data reuse, and future AI-driven scientific discovery.
This presentation describes recent progress in FAIR-oriented data management at SHINE, from policy and metadata standards to system implementation. In collaboration with major synchrotron radiation facilities in China, we contributed to common scientific data policies and metadata standards for large-scale user facilities. Based on these, SHINE has developed facility-specific metadata specifications for XFEL user experiments, covering key entities across the experimental data lifecycle.
An XFEL data management system has been developed based on DOMAS, a scientific data management framework jointly developed by our team and the IHEP, CAS. The system integrates online storage, offline storage, metadata management, and data services. It supports automated migration of experimental data files, metadata extraction and consolidation, controlled data access and data retrieval.
During the joint commissioning with the Spectrometer for Electronic Structure (SES) and Atomic, Molecular, and Optical Science Endstation (AMO) of SHINE, we has identified two practical challenges. The first is consistent permission management for shared storage directories across Linux, Windows, and HPC environments. The second is how to support diverse beamline and endstation DAQ software while maintaining metadata completeness, machine readability, and data traceability. These lessons are important for moving from FAIR data management toward AI-ready datasets.

The OSCARS project represents a unique European collaboration that unites Research Infrastructures across the European Science Clusters (SCs) representing the key domains: environmental research, physics and astronomy, photon and neutron, social, and life sciences. At its core, OSCARS focuses on promoting open science and FAIR data principles and addresses this challenge by three primary objectives: consolidation of existing research data services, engagement with a broad network of stakeholders, and the uptake of open science principles.
The consolidation workpackage focuses on interoperability and introducing novel functionalities by metadata enrichment of existing services. One of the key opportunities that the Photon and Neutron community (PaNOSC) explores is the extension and improvement of the ontology of Photon and Neutron Experimental Techniques (PaNET). As an ontology, PaNET is not merely a flat list of terms: it encodes complex relationships between them. The techniques are characterized by a broad range of specifiers, covering physical probes, physical processes, beam characteristics, purposes and dependencies.
This network of terms enables diverse applications: from discovering experimental data with PaN Finder over finding dedicated training material with PaN Training to guiding users to the most suitable technique for their scientific questions. Ultimately, PaNET can help advance FAIR data practices and interdisciplinary research.

The PSI Data Catalog is Paul Scherrer Institute’s data archiving, sharing, and publishing repository. It is built around SciCat, an open source scientific metadata catalog developed by a collaboration of various international institutes and research facilities. The catalog enables research data management aligned with FAIR principles at PSI large scale facilities, including the Swiss Light Source, Swiss Free Electron Laser, SINQ neutron source, and Swiss Muon Source. This talk will explore the latest updates to SciCat and PSI research data infrastructure.

Recent developments have focused on integrating the PSI Data Catalog with the broader Open Research Data (ORD) landscape through the ORD initiative of the Swiss Federal ETH Domain, targeted at improving and enabling better FAIR and ORD practices among the scientists. Federated login has been implemented in the catalog through the EduGAIN network, improving interoperability with users and services beyond PSI. Connections to other services are also being strengthened, including electronic lab books (OpenBIS, SciLog), complementary repositories (Zenodo, EnviDat, DataSwiss), and processing platforms (Renku, AiiDa). Sharing of metadata is enabled through open formats including JSON-LD, RO-Crate, and the ELN file format, with data transfer facilitated through Object Storage using the S3 protocol.

Much of these developments are facilitated by new features in the latest SciCat versions. After a lengthy migration process, PSI is now using SciCat version 5. Tying SciCat into a diverse ecosystem of interoperable services has been enabled through the highly configurable features of SciCat, including the customizable job system, new UI components, and flexible frontend configuration. By connecting facilities, services, and researchers through open standards and interoperable infrastructure, SciCat is helping advance FAIR and Open Research Data practices across the wider research community.

Scientific facilities are evolving toward automated and data-intensive environments in which experiments continuously generate heterogeneous and large-scale datasets that must be managed under governance requirements. Although metadata catalogues have improved data management, the transient phase between acquisition and long-term cataloguing often remains fragmented, manually handled, script-driven, and beamline-specific. This gap is critical for achieving high-level automatization and especially for autonomous experiments, where decisions must remain reproducible and auditable.
This work presents SIRFlow, a policy-driven framework for data governance automation currently under development for Sirius at CNPEM. SIRFlow operates as an intermediate governance layer between acquisition systems, workflows, and metadata catalogues, coordinating dataset lifecycle states, auditable job and data services events, and policy-aware orchestration.
The framework comprises two main architectural components: a data control plane responsible for policy validation, dataset lifecycle management, and both event and job orchestration; and a data execution plane composed of scalable workers responsible for data movement, processing, streaming, and integration with external systems such as ICAT, high-performance computing (HPC) infrastructures, and AI models. Automation is modeled through auditable job envelopes associated with dataset lifecycle records, enabling reproducibility, safe retries, provenance tracking, and controlled state transitions.
Our key contribution is the explicit distinction between transient and persistent scientific datasets. In the transient phase, datasets may undergo streaming, processing, validation, and curation before promotion to institutional catalogues. Only datasets satisfying governance and quality policies are promoted to long-term systems such as ICAT, reducing storage and catalogue pollution, and enabling robust autonomous experiments.

With the incorporation of more AI in software development, it is important to ensure user applications do not lose focus on the users and take into consideration user needs and workflows. AI can help with rapid development, but issues in consistency and predictable behaviours can arise without user review. There are a variety of ways we can incorporate user experience (UX) practices in the evolving landscape of application development to avoid potential pitfalls. This presentation will touch on ways that AI and UX can work together as well as considerations to keep in mind when coding with AI.

The Advanced Photon Source Data Management System (DM) now provides a web-based user interface. DM is responsible for managing experiment data for facility users by providing capabilities related to storage, access, transfer, metadata, and processing. Previously, users interacted with DM through a command line interface, Python API, or PyQt desktop GUI available locally on workstations at each beamline. The new DM web portal is built on the existing Python API and the Django Globus Portal Framework (DGPF). It introduces role-based access so APS users can only interact with their own experiments while providing access to all experiments and additional features for beamline staff. The web portal includes features available in the GUI to create and edit experiments, edit user access, start and monitor data transfers, view file statistics, and monitor processing jobs. The web portal expands on these features with additional improvements of the ability to view file metadata, start new processing jobs, monitor status of DM services and HPC resources, and archive or restore datasets from tape. Each experiment page links to Globus for more convenient data sharing. Future developments will include a live view of data streams, interactive data visualization, and a monitoring dashboard for local PBS queues.

In the context of operator interfaces (OPIs), EPICS has a mature ecosystem of desktop tools. In contrast, the adoption of web-based solutions remains low: existing approaches typically require desktop applications for OPI design, using the browser only at runtime, or demand JavaScript development from users. WEISS (Web EPICS Interface and Synoptic Studio) addresses this gap by providing a web-native, no-code architecture built on a modern web stack. It offers a complete in-browser environment for designing, configuring, and deploying operator interfaces without requiring programming. In addition to a graphical editor and clearly separated staging and runtime environments, WEISS provides a fast, lightweight deployment model, making selected OPIs instantly available to operators. For version control, direct integration with remote Git repositories enables commits, checkouts, and rollbacks within the same interface, reducing context switching and dependency on external clients. By removing the need for local tooling and reducing development overhead, WEISS lowers the barrier to creating and maintaining EPICS GUIs in modern web environments.

This talk showcases WEISS's current and planned features, typical use cases, and the workflow for deployment of production OPIs using the tool.

The recent rise in capabilities from AI systems has begun to change the way that user interfaces are developed, making it significantly easier for domain scientists to create their own applications, or for developers to rapidly construct applications in previously unfamiliar frameworks.

At the Advanced Light Source, the Photon Science Computing (PSC) group is systematically exploring the use of AI with application development. Several strategies are being employed including hands on hackathon training sessions, publishing agent skills, and conducting internal studies to provide recommendations.

This presentation focuses on one particular case study and its conclusions. A beamline scientist used Agentic AI to create a PyQt desktop application for beamline controls and data acquisition with the Bluesky framework. PSC staff then used Agentic AI to recreate this PyQt app in a modern web stack with React and the Bluesky React library Finch.

Throughout this process, limitations in AI coding were observed including inconsistent styling, re-implementation of existing components, and layout instability. Strategies to mitigate these deficiencies have been made which include modifications to existing code bases, creation of new AI skills, and developing guidelines for using Agentic AI.

This work provides insights and recommendations for how to prepare for the continued advancement and adoption of Agentic AI at user facilities for scientific applications.

At the Australian Synchrotron (ANSTO) we are currently transitioning our user-facing GUIs towards native web interfaces, while retaining our established Qt-based engineering GUIs for staff and engineers. We deliberately separate engineering UIs, built for low-level device interaction, diagnostics, and maintenance, from user-facing web GUIs designed for day-to-day experiment execution. Web-based GUIs are a natural fit for our beamline workflows that extend beyond controlling individual devices, spanning complex acquisition control, metadata capture, and downstream data processing pipelines. Our web applications are custom built to match beamline workflows, while sharing a common set of reusable components from our open-source GUI library. A key driver for the transition is operational efficiency and cyber security considerations: web GUIs can be deployed and updated centrally, enabling faster iteration and consistent behaviour for all users, and the underlying technology is well understood and supported by our cyber security team. Web GUIs also align naturally with remote access, supporting off-site experiments and hybrid user models while maintaining a familiar browser-based experience. This presentation describes our architecture, technology choices, and deployment model, and shares lessons learned from adopting web GUIs as a first-class interface for synchrotron beamlines.

BEC Widgets is a highly modular Qt-based GUI framework developed together with Beamline Experiment Control (BEC) at the Swiss Light Source, Paul Scherrer Institute. The main goal of the platform is to simplify the interaction between scientists and their experimental setup, so they can focus on the science with as little friction as possible.

At the core of the system is the modular BECDockArea, where users can construct and use widgets from a large library covering high-performance data visualization, experiment control, automation and scripting-driven workflows. Widgets can be combined on the fly, even during a running experiment, to create GUIs adapted to the current experimental needs. The same widgets can also be controlled from the BEC command-line scripting interface, allowing users to combine scripted experiment logic with dynamic GUI elements in a running session. The new BEC Widgets property system also allows users to freeze a GUI setup into reusable profiles, which can later be restored for specific use cases.

Recent development focuses on turning this widget framework into a full desktop experience through the BEC Application. Users can still build experimental workspaces with the docking system, but they also get additional application views developed for more advanced workflows. Beamline scientists can integrate new EPICS devices through autogenerated GUI forms for most EPICS based devices, manage experimental accounts, configure message notifications such as automatic Signal or Microsoft Teams messages, and use an integrated development environment for writing macros and scripting BEC and GUI behavior. The editor is aware of the currently running BEC session, including loaded devices and available scans, and can provide context-aware autocompletion.

The first phase of the High Energy Photon Source (HEPS) was completed in 2025. HEPS now offers 14 public beamlines and one optics test beamline, covering a broad range of research areas. HEPS calls for experimental proposals from researchers worldwide. The proposal management process encompasses a comprehensive sequence of stages: submission, multi‑tiered review, beamtime approval, scheduling, on‑site execution, and outcome reporting. To address these needs, we have developed a scientific user service software that enables end‑to‑end management of experimental proposals.

Starting in September 2025, HEPS conducted three rounds of pilot proposal calls via the User Service System, resulting in over 200 approved pilot proposals, more than 4,700 hours of scheduled beamtime, and over 700 user visits. Building on this experience, the user service system was further refined, and the first call for general proposals was launched in March 2026.

Given the rapid advancements in artificial intelligence (AI), we have further designed a human‑computer collaborative platform powered by large language models. This platform seamlessly integrates AI capabilities throughout the entire workflow—from proposal drafting and review to evaluation, experiment planning, and knowledge consolidation. By incorporating intelligent assistance, the system will significantly enhance both the efficiency and quality of proposal processing.

This work outlines the requirements analysis for the user service software, describes the developed framework, elaborates on the core functional modules, and presents specific design strategies for AI‑enhanced implementation.

Delivering reliable beamline software increasingly looks like operating an on-premises “cloud”: many services, close to instruments and data, with rapid iteration and strong cyber security controls. At the Australian Synchrotron (ANSTO), our Scientific Computing team runs essentially all beamline computing services on-site using containers and Kubernetes. Across 10+ Kubernetes clusters we deploy and run shared platform services alongside beamline-specific applications for hardware control, data collection, and data processing, following an “everything is a container” approach. We support this with automated CI/CD and a GitOps-style delivery model, plus on-premises artifact repositories (Docker, Python, and S3-compliant storage) to keep builds and deployments within facility firewalls. Observability is treated as a first-class requirement: centralised logging and metrics (e.g., Prometheus and OpenSearch) and distributed tracing (OpenTelemetry/Jaeger) provide operational insight and auditability. Many cloud concepts were developed for purely virtual services, but our services eventually touch physical beamline hardware - there is only one set, and access to it is inherently restricted. We work around these constraints with digital twins of beamlines, coupled to a traditional dev/stage/prod model, to test and validate changes before they reach beamline operations. Using industry-standard tools reduces bespoke maintenance, improves collaboration, and lets us benefit from mature security ecosystems. Security is built in via authenticated access, Kubernetes RBAC, network policies, vulnerability scanning, patch cycles, and secrets management. This talk presents our architecture and lessons learned operating an on-premises cloud based research infrastructure in a 24/7 user facility.

The Mantid project is almost 20 years old. It started as a C++ GUI for processing muon and time-of-flight neutron scattering data at ISIS. In 2010, ORNL was the first of many to join in the development of the software. Several significant changes have occurred in scientific computing since the first commit to the Mantid repository in 2007. Among them is the rise of Python as a dominant language for scientific computing and the creation of several outstanding libraries. One example is that standard data objects (e.g. numpy arrays and dataframes) have become the lingua franca of scientific Python. This is increasingly relevant as Python becomes a language used for integrating and gluing together various functionalities.

At ORNL, data are measured in event mode, which captures attributes of every detected neutron. As a consequence, the significant power increase at the Spallation Neutron Source (SNS) (currently at 2.0MW) directly correlates with increases in data rates. Meantime, dramatic increases in the number of pixels per instrument correspondingly drive increased compute requirements. All of these factors are prompting a revisit of how Mantid is designed and implemented.

This talk will present the ORNL plan for modernizing Mantid in preparation for the increased power at SNS, event filtering at HFIR, with an eye towards the higher data rate and more pixelated instruments planned at the second target station.

Since its origin in 2007 Mantid [1] has become an essential tool for Neutron and Muon data reduction at ISIS, ORNL, ILL, PSI, MLZ and beyond. The Mantid collaboration have built the data structures, algorithms and scripts needed to process instrument data and a workbench with workflow centred user interfaces to make an expanding range of techniques accessible to scientist and users. Mantid is a free and open-source package, developed and maintained by software engineers across the international facilities in close collaboration with instrument scientists.

We present an update on the work that ISIS have been developing with recent highlights including crystallographic texture analysis and polarised SANS workflows, a new instrument viewer and performance improvements.

[1] https://www.mantidproject.org/

Building on previous work on the ICAT-based ESRF Data Portal (https://icatproject.org, https://data.esrf.fr) presented at earlier NOBUGS editions, this talk explores the next challenge: making metadata richer, more structured, and more findable, interoperable, and reusable. Our goal is to maximize scientific impact by enabling data reuse and reproducibility, capturing structured metadata across the full experiment lifecycle, from sample preparation to publication.
We leverage the ESRF technique ontology (ESRFET, https://w3id.org/PaN/ESRFET) to describe experiments through comprehensive, semantic definitions, ensuring consistent and interoperable representation across beamlines and enhanced interoperability via standards such as NeXus (https://www.nexusformat.org/). Metadata schemas are formalized and validated to ensure completeness and consistency while supporting evolving requirements. In addition, sample metadata plays a key role in linking experiments and enabling cross-experiment traceability.
These enriched metadata enable advanced features in user-facing applications, including technique-specific displays in the Data Portal, as well as integration with domain-specific portals such as the Human Organ Atlas (https://human-organ-atlas.esrf.eu/) and other emerging data collections.
We also report early progress on improving search capabilities over this richer metadata corpus, and reflect on the benefits for data discoverability, reuse and automated harvesting by metadata aggregators. Future work includes extending the approach to additional ontologies and exploring AI-assisted metadata enrichment.

Metadata have become an integral aspect to scientists to work with experimental data due to the more complex machinery used during data acquisition and also the vast amount of data produced at experimental sites.
Publication of scientific results often require the publication of the raw and analysed data. DOIs, digital object identifiers, for collected and analyzed data are therefore often mandatory in journal publications. The FAIR data movement contributes to this efforts and many other initiatives in this direction were made.
At DESY, metadata management in photon science is established using SciCat, a science catalogue for basic cataloguing functionalities and eventual DOI minting. With national initiatives, in particular DAPHNE4NFDI, the SciCat project has grown even more and SciCat at DESY has become a cornerstone for making data FAIR.
This contribution outlines the deployment journey of the data cataloguing and minting service - from development to production. We explain the assisted process for the user to mint a DOI including the case of actual data access requests. We highlight the elements to ensure robustness, stability and functionality in our system and present efforts and commitment to the SciCat collaboration. Scalability and easy maintaince play an important role as we plan to make it available for more DESY beamlines. We also give prospects of how to enhance the usefulness of SciCat at DESY by integration to further site-specific critical systems as library and storage.

Scientific datasets rarely stand alone. Raw data, processed derivatives, and experimental metadata are deeply related, but most data services store them as if they aren't. At the Advanced Light Source, we've been working on closing that gap.
We use Tiled, from the Bluesky project, as our primary catalog and data service. As we built applications on top of it, embedding URLs in metadata fields proved too brittle — links are invisible unless you already know they're there, and per-application link schemas don't support scalable search.
We built splash_links to address this: a data service that stores Entities and Links, forming a queryable persisted graph over our data holdings. Entities hold references to Tiled nodes; Links relate them to each other.
An interesting engineering question turned out to be what to build it on. We'll walk through our evaluation of relational databases, native graph databases, and Triplestore databases — including RDF serialization and SPARQL-based semantic queries — and discuss the trade-offs that informed our final choice.

TIFF is a popular format to store multi-dimensional scientific images in. European Spallation Source ERIC (ESS) will also provide reduced TIFF image stacks to our imaging technique users.

At ESS, we use a data catalogue framework(scicat) to comply with the FAIR data policy. However, the physical properties of reduced data are not always possible to be inherited from parent datasets or recorded as scientific metadata of the catalogue dataset instances. Therefore we needed a standard way of storing the physical properties of the image, e.g. physical coordinates, ideally close to the file.

There are flexible conventions of multi-dimensional image format or community-specific metadata schemas. However, we needed more community-independent and simple, straightforward way of storing scientific metadata. In order to quickly establish the minimum viable product of the image metadata standard, we started the scitiff project.

Scitiff hosts programming-language-independent json metadata schema as well as Python APIs. It piggy-bags on the ImageJ hyper-image-stack format and dict-like plain text ImageJ metadata. Scitiff also inherits the concept of data structure with physical units and dimensions from scipp project.

At the presentation, we will:
- Introduce the concept of the physical units and dimensions in scipp project in relation to the scitiff project.
- Explain how we implemented the scitiff schema that is compatible with scipp data structure.
- Demonstrate the scitiff Python I/O API module.
- Discuss the project boundaries and roadmap.

European XFEL relies on multiple distributed services to support scientific operations, from proposal management through experiment preparation to data acquisition, analysis, and archiving. Keeping workflows coherent across these systems requires more than metadata exchange: it requires shared definitions, validation rules, and governance for how operational metadata is managed. At European XFEL, this role has increasingly been taken on by myMdC.

This contribution presents the evolution of myMdC from a central metadata catalogue into a facility-wide service supporting a shared operational contract across services. Recent developments include the redesign of proposal-team models and role governance, the extension of permissions and service interfaces, the integration of instrument and technique information, the expansion of dataset- and file-level inventory models, and closer alignment with logbook and archiving workflows. A representative example is the iterative refinement of proposal-team roles, where operational experience repeatedly required adjustments to role semantics, responsibilities, and eligibility checks to keep the contract both consistent and practical across services.

The main focus of this contribution is on lessons learned. Our experience shows that central contracts must evolve incrementally rather than through fixed upfront designs; governance and role semantics are as important as technical schemas; metadata alignment depends not only on APIs, but also on operational ownership, and an appropriate degree of flexibility. Effective governance also requires corresponding UI and workflow changes. These lessons may help other large-scale facilities design sustainable and interoperable metadata architectures in production environments.

Scientific data compression has become a critical component in large-scale computing and experimental facilities, where data volumes increasingly outpace storage capacity and I/O bandwidth. This report surveys recent advances and the current state of the field. We review the evolution of error-bounded lossy compression algorithms, including SZ, ZFP, and MGARD, with emphasis on their extended capabilities for feature-preserving and physics-aware reduction. We examine the emerging paradigm of compressed-domain computation, which enables analytic operations to be executed directly on compressed representations without full reconstruction. Hardware acceleration support, including SIMD instructions and offloading to DPUs, is summarized alongside ongoing standardization efforts. The report also addresses the evaluation of scientific fidelity—specifically, methodologies for quantifying compression-induced distortion in downstream analyses. We conclude with a discussion of remaining limitations, including reproducibility concerns, heterogeneous platform portability, and the integration of machine learning techniques into compression pipelines.