Speaker
Description
The massive and diverse experimental data generated by the High Energy Photon Source (HEPS) poses severe challenges to data processing pipelines, rendering traditional fixed serialization and compression strategies inefficient. To address this, we introduce LightPacker, an adaptive decision-making system that automatically applies the optimal serialization and compression algorithm combinations based on specific data characteristics. LightPacker operates through a five-module architecture: a unified Algorithm Library, Feature Analysis for real-time data profiling, Offline Analysis for historical benchmarking, Online Evaluation for rapid algorithm matching, and Decision Execution. By leveraging data features (e.g., sparsity, structure) and sampling techniques, the system minimizes decision overhead while ensuring accuracy. Experimental results demonstrate that LightPacker significantly outperforms fixed strategies in typical HEPS scenarios, effectively reducing network bandwidth occupancy, improving preprocessing efficiency, and minimizing storage costs for large-scale scientific facilities.
