Speaker
Description
Light sources facilities depend on HDF5 to manage rapidly growing experimental datasets. At the same time, modern data analysis demand higher I/O performance, greater robustness for long-running acquisitions, improved interoperability with legacy scientific formats, efficient support for irregular data, and stronger data protection. This talk presents recent advances in HDF5 software inspired by real-world use cases from light source facilities and designed to address these challenges while improving HDF5 usability in large-scale scientific environments.
We discuss recent progress toward multi-threaded HDF5, including a new multi-threaded Bypass connector for improving I/O throughput on multi-core systems. We also present HDF5 connectors that expose legacy scientific formats including TIFF through standard HDF5 and APIs without requiring data conversion, enabling existing HDF5 tools to access heterogeneous data while preserving native formats and metadata.
Additional topics include support for sparse and variable-length data for efficient storage of irregular scientific datasets; transparent HDF5 encryption for secure scientific data access; and a complete portable single-writer/multiple-reader (SWMR) implementation, including a recovery tool for restoring files after interrupted writes.
Together, these developments improve HDF5 performance, robustness, and interoperability while demonstrating how a carefully designed software architecture allows mature scientific software to evolve to meet new experimental and computational demands.
