llama.cpp

What it is

llama.cpp is an engine for CPU-first inference in plain C/C++ that can run on modest hardware.

It uses GGUF quantisation to shrink models, making them suitable for deployment on edge devices or other hardware with limited resources.

This approach allows for deployment in environments where a GPU cannot be used, although it may result in lower throughput.

Where it fits in a deployment stack

In a typical deployment, tools like llama.cpp are used to enable the efficient execution of machine learning models on a variety of hardware platforms.

They normally sit next to model training pipelines and data ingestion systems, taking over the work of deploying and running trained models in production environments.

These tools are often used in on-premises or edge deployments where direct access to the hardware is required or preferred.

How to evaluate it

To evaluate llama.cpp, you need to consider its performance, compatibility, and overall fit within your existing deployment stack.

• Measure p95 latency of the retrieval step under 50 concurrent sessions.
• Inspect the output of a test dataset to verify accuracy and consistency.
• Feed it a dataset with known edge cases to test robustness.
• Inject failures in the input data stream to test error handling.
• Compare the size of the quantised model with the original model.
• Verify the engine's ability to run on different CPU architectures.

What to verify before you commit

Before committing to the use of llama.cpp, there are several factors that need to be verified, including its licence terms, data handling practices, and operational costs.

• Check the licence terms for any restrictions on commercial use.
• Verify how the engine handles sensitive data and ensure it meets your security requirements.
• Calculate the operational costs of running the engine on your target hardware.
• Test the engine's failure behaviour and verify it meets your reliability requirements.
• Assess the potential for vendor lock-in and verify the engine's compatibility with other tools.
• Check for any dependencies that may add complexity or restrictions to your deployment.