NOBUGS 2026

Rising data collection rates from XFELs and synchrotrons necessitate compression techniques to support long-term storage of large datasets, especially for crystallography. To address this, we developed an end-to-end workflow that tests sparsification of diffraction images validated with DIALS processing and iterative merging in the cctbx.xfel suite. At the pixel level, a convolutional segmentation model identifies Bragg peaks. Trained on synthetic diffraction images, the AI model learns patterns in the data that detect weak reflections beneath the background. The resulting reduced images are stored in a sparse coordinate (COO) HDF5 format that adheres to the DXTBX standard, making them DIALS-readable. Of interest is exploring new storage templates that exist in Nexus, while preserving the sparse nature of the data. Using a benchmark dataset, we observe a ~200× compression ratio with negligible loss in CC1/2 (a cross validation metric). The full validation workflow is deployable via JGI Analysis Workflow Service (JAWS) at NERSC-Perlmutter, where Cromwell and Slurm coordinate MPI computing across GPUs and CPUs. Following sparsification, fast binary morphological operations are used to improve compression and merging statistics. Parameters defining these operations (e.g., binary openings/closings) are configured as workflow inputs such that HPC Bayesian optimization can be utilized to minimize loss in crystallographic quality metrics, including CC1/2, anomalous peak height, and completeness. The compression makes long-term storage, transfer, and reanalysis of crystallography datasets more feasible for light source facilities and users, contributing to broader goals in AI-readiness and FAIR compliance.