Speaker
Description
Light-source facilities now generate data at rates that overwhelm conventional storage paths — HEPS Phase-1 alone averages around 800 TB per day with peaks of 3.2 TB/s, while typical parallel storage sustains only tens of GB/s. Compression is essential, and the community has developed many specialized algorithms — lossless, hand-crafted lossy, and AI-based lossy — each with different trade-offs across compression ratio, fidelity, and throughput. Yet accessing them in real beamline workflows remains difficult: existing tools lack a unified entry point, cannot switch between lossy and lossless modes at runtime, are tied to a single hardware backend, and offer limited Python and HDF5 integration. Scientists therefore spend significant engineering effort just to evaluate or deploy a single algorithm.
We present PhotonZip, a unified compression framework that directly targets these software-level gaps. PhotonZip exposes a single Python API and HDF5 filter as the entry point, supports runtime switching between lossy and lossless modes, and runs portably across CPU, NVIDIA GPU, and AMD GPU backends. As an initial demonstration, it integrates two of our in-house algorithms: MANS, a multi-byte ANS lossless codec addressing the poor compression of byte-level coders on multi-byte integer detector data; and CAIEC, an end-to-end inference–coding co-acceleration framework addressing the throughput bottleneck of AI-based lossy compression.
PhotonZip's architecture is open: more compression algorithms will be progressively integrated. As the algorithm pool grows, we will develop a need-driven recommendation module that, from user-declared targets (ratio, quality, throughput) and the available hardware, automatically selects the most suitable codec — further lowering the barrier for non-expert scientists to benefit from state-of-the-art compression.
Open source: https://github.com/hpdps-group/PhotonZip
