Unstructured

What it is

Unstructured is a tool that extracts data from unstructured sources like PDFs, HTML, and scans, and turns it into structured elements.

It handles the extraction layer, including tables and images, and outputs consistent JSON elements with layout metadata preserved.

This is useful for retriever systems that need clean, structured data to function effectively.

Where it fits in a deployment stack

RAG systems rely on high-quality input data to produce accurate results.

A tool like Unstructured normally sits at the beginning of a data pipeline, where it can extract relevant data from various sources and pass it on to downstream components for further processing.

In a typical RAG deployment, this type of tool would be followed by components that handle retrieval, ranking, and generation.

How to evaluate it

To evaluate Unstructured, you need to consider how it handles different types of input data and how it integrates with your existing RAG system.

• Feed it a dataset with diverse PDF formats and measure the accuracy of the extracted data.
• Inject failures into the input stream, such as corrupted PDFs or missing metadata, and inspect the tool's error handling.
• Measure the throughput of the tool under different concurrency levels.
• Compare the output of Unstructured with manually extracted data to verify its accuracy.
• Inspect the JSON output for consistency and completeness.
• Verify that the tool preserves layout metadata correctly.

What to verify before you commit

Before committing to Unstructured, you need to verify several aspects of the tool, including its licence, data handling, and operational cost.

• Check the licence terms to ensure they align with your organization's requirements.
• Verify how the tool handles sensitive data, such as encryption and access controls.
• Estimate the operational cost of running the tool, including resource usage and maintenance.
• Test the tool's failure behavior, including error handling and recovery.
• Check for potential lock-in, such as dependence on specific formats or vendors.
• Verify the tool's compatibility with your existing infrastructure and components.