Speaker
Description
It is essential to have effective tools to diagnose emerging issues for the stable operation of control systems in a large experimental facility. This task becomes even more challenging in distributed systems, as the source of problems may appear in different subsystems and at different times.
The control system at MAX IV is based on Tango Controls, and several tools are used for monitoring and troubleshooting. Data from Tango devices is archived in a database and can be explored via ArchViewer, an interactive time-series data analysis tool. The same data is also used in Grafana dashboards, where they can be combined with Prometheus metrics to provide a more comprehensive view of system behavior.
Logs from Sardana and other Tango devices are collected in Graylog. This system provides centralized remote access to activity history and error messages across the entire facility.
The snapshot system enables switching between different operation modes while minimizing human error.
The electronic logbook Elogy is used by the operation support engineers to record information about incoming issues, making it easier to resolve similar problems in the future.
The combination of metrics, logs, snapshots, and operator records provides a useful set of tools that help identify the causes of failures faster and improve understanding of control system behavior, leading to more reliable operation.
