OpenLLMetry

What it is

OpenLLMetry is a tool that emits standard OpenTelemetry spans from LLM and vector database calls. This allows these traces to be ingested by any OpenTelemetry backend. The goal is to avoid creating a separate observability silo for LLM observability.

By using OpenTelemetry spans, OpenLLMetry enables integration with existing observability stacks. This can simplify the process of monitoring and troubleshooting LLM systems.

Where it fits in a deployment stack

Tools like OpenLLMetry are used to monitor and troubleshoot LLM systems in production environments. They typically sit alongside other observability tools, such as logging and metrics systems. The purpose of these tools is to provide visibility into the performance and behavior of LLM systems.

In a production deployment, LLM systems are often part of a larger pipeline that includes data preprocessing, model training, and inference. Observability tools like OpenLLMetry help engineers understand how these systems are performing and identify potential issues.

By integrating with existing observability stacks, tools like OpenLLMetry can help reduce the complexity of monitoring and troubleshooting LLM systems. This can be especially important in environments where multiple teams are responsible for different parts of the pipeline.

How to evaluate it

To evaluate OpenLLMetry, you need to consider how it will be used in your production environment. This includes understanding the types of data it will be processing, the volume of traffic it will handle, and the existing observability tools it will integrate with. You should also consider the potential failure modes and how they will be handled.

• Feed it a dataset with a mix of simple and complex queries to see how it handles different workloads.
• Inject failures into the system, such as network errors or database timeouts, to see how OpenLLMetry responds.
• Inspect the output of OpenLLMetry to see if it is providing the expected visibility into LLM system performance.
• Measure the latency of the retrieval step under different concurrency levels to understand its performance characteristics.
• Verify that OpenLLMetry is correctly emitting OpenTelemetry spans and that they are being ingested by your existing observability backend.
• Test the scalability of OpenLLMetry by increasing the volume of traffic and monitoring its performance.

What to verify before you commit

Before committing to using OpenLLMetry, you should verify several key aspects of the tool. This includes understanding the licensing terms, data handling practices, and operational costs. You should also consider the potential for lock-in and the ease of migration to alternative tools.

• Check the licence terms to ensure they are compatible with your organisation's requirements.
• Verify how OpenLLMetry handles sensitive data, such as encryption and access controls.
• Estimate the operational costs of running OpenLLMetry, including any potential impact on existing systems.
• Test the failure behavior of OpenLLMetry to ensure it does not introduce single points of failure into your system.
• Evaluate the potential for lock-in by assessing the ease of migrating to alternative tools or vendors.
• Check the compatibility of OpenLLMetry with your existing observability stack and tooling.