Ollama

What it is

Ollama is a tool that bundles model weights, quantisation, and a local server behind a single command-line interface (CLI). It is designed to run a large language model (LLM) on a laptop. This makes it a straightforward way to get a model up and running for demos, pilots, or development work.

The fact that Ollama includes quantisation and model weights in its bundle suggests it is intended to simplify the process of deploying an LLM. By providing a local server, it also enables self-hosted deployment.

Where it fits in a deployment stack

Deployment tools like Ollama are typically used to manage the lifecycle of a model in production. They sit between the model training pipeline and the application or service that consumes the model's output. In a production system, such tools usually work alongside other components like load balancers, container orchestrators, and monitoring systems.

These tools often take over the work of managing model versions, handling inference requests, and providing a stable interface for applications to interact with the model. They may also be responsible for optimizing model performance, handling errors, and providing metrics for monitoring and debugging.

In general, deployment tools for LLMs need to be able to handle large amounts of data and compute resources efficiently. They must also be able to integrate with a variety of frameworks and libraries, and provide a flexible way to configure and customize the deployment process.

How to evaluate it

To evaluate Ollama, you need to consider how it performs in your specific use case. This involves testing its capabilities with your data and workload. You should look at the quality of the results it produces, as well as its performance characteristics like latency and throughput.

• Measure p95 latency of the retrieval step under 50 concurrent sessions
• Inspect the output of the model for a representative set of input data
• Feed it a large dataset to test its data handling capabilities
• Inject failures into the system to test its error handling and recovery
• Compare the performance of the model with and without quantisation
• Verify that the local server can handle multiple concurrent requests

What to verify before you commit

Before committing to using Ollama, you need to verify several aspects of the tool. Check its licence terms to ensure they are compatible with your project's requirements. You should also review its data handling practices to ensure they meet your security and compliance standards.

• Check the licence terms for any restrictions on usage or distribution
• Verify that the tool handles sensitive data in accordance with your security policies
• Review the operational costs of running the tool in your production environment
• Test the tool's failure behaviour to ensure it meets your availability requirements
• Assess the potential for vendor lock-in and ensure you have an exit strategy
• Confirm that the tool is compatible with your existing infrastructure and workflows