AI Retrieval

​

Overview

LLM Retrieval is the process of making your data compatible with LLMs. This is done by embedding your data into a vector space, then indexed into a database. This allows you to run semantic searches on your data, which is the core of LLMs.

​

Use Cases

​

Usage

​

Indexing

We provide APIs to easily push data into our system. We support the following “primitive” data types for ingestion:

• Image URLs (.jpg, .png, .gif, .bmp, .tiff)
• Text (string)
• Embeddings (number[])

This data will then be pushed into our indexing pipeline to generate embeddings and store in a vector db. Checkout the Index endpoint for more details.

​

File Importing

We provide APIs to easily push larger collections of data into our system as well. We support the following file types for ingestion:

• .csv
• .docx
• .pdf
• .pptx
• .txt
• .xlsx

Upon upload, these files run through the following pipeline:

1. The file runs through a series of metadata extractors + augmenters
2. File is converted into a text representation via OCR (if applicable)
3. The text is split into overlapping 500 token chunks
4. These chunks are embedded based on the chosen embeddings model (ada, clip, etc)
5. The embeddings are indexed into a Vector database
6. Vector is indexed into our database

​

Searching

Run semantic search out of the box with our API. We support the following search term types:

• Images (.jpg, .png, .gif, .bmp, .tiff)
• Text (string)
• Embeddings (number[])

Filtered search is also supported. Check out the Search endpoint for more details.

​

Definitions