Journals of Accelerator Conferences Website (JACoW)
JACoW
is a publisher in Geneva, Switzerland that publishes the
proceedings of accelerator conferences held around the world
by an international collaboration of editors.
| URL | https://doi.org/10.18429/JACoW-ICALEPCS2025-WEAG006 |
|---|---|
| Title | Better software observability using Tango Controls with OpenTelemetry - experience at MAX IV |
| Authors |
|
| Abstract | Distributed software systems are complex and the interactions across multiple machines can be difficult to debug and monitor. Log messages are not enough for observability. We need more information about the communication between applications, how each one is executing, and its internal state. In practice, applications can be made more observable using software frameworks such as OpenTelemetry. The Tango Controls framework has built-in support for OpenTelemetry in C++ and Python since version 10.0.0. We are using it operationally at the MAX IV synchrotron. We provide examples of the traces, trends, and other data available when running at scale on a beamline with hundreds of devices. We report on the compute and performance impact for client and server software applications, as well as practical issues. For the backend servers that ingest and query the telemetry data (running Grafana Tempo for traces and Grafana Loki for logs) we report on the compute resources required. |
| Paper | download: WEAG006.pdf |
| Cite | BibTeX, LaTeX, Text/Word, RIS, EndNote |
| Conference | 20th Int. Conf. Accel. Large Exp. Phys. Control Syst. (ICALEPCS'25) |
| Series | |
| Location | Chicago, IL, USA |
| Date | 20-26 Sep 2025 |
| Publisher | JACoW Publishing, Geneva, Switzerland |
| Editorial Board | Kent Wootton (ANL), Jeffery Biss (FNAL), Gustavo Bruno (ANL), Jeff Kirchman (ANL), Kelly Jaje (ANL), Denise Skiadopoulos (ANL), SuYin Grass Wang (ANL) |
| Online ISBN | 978-3-95450-255-4 |
| Online ISSN | 2226-0358 |
| Received | 06 September 2025 |
| Revised | 23 September 2025 |
| Accepted | 28 October 2025 |
| Issued | 25 November 2025 |
| DOI | 10.18429/JACoW-ICALEPCS2025-WEAG006 |
| Pages | 801-806 |
| Copyright | Published by JACoW Publishing under the terms of the Creative Commons Attribution 4.0 license. Any further distribution of this work must maintain attribution to the author(s), the published article's title, publisher, and DOI. |