Data visibility

DataQI uses a highly efficient, demand-driven ingestion pipeline to index and understand your connected data sources. Unlike traditional systems that attempt expensive, full-text extraction of every file immediately, DataQI uses a tiered approach to ensure only relevant content is processed.

Crucially, ingestion is entirely user-scoped. DataQI only has visibility of files that authenticated users have natural access to. If no user has access to a file, it is never ingested. When an early adopter interacts with the system, they trigger the ingestion of the files they have access to. These files are then available to subsequent users who share the same access rights, without needing to be re-ingested.

The demand-driven pipeline

DataQI does not blindly ingest all content. It operates on a three-tier model:

1. Structural Tier: Captures and embeds the file path and metadata instantly, making the file structurally visible.
2. Keyword Tier: Extracts and embeds shallow keywords.
3. Content Tier: Extracts, chunks and embeds the full text of the document.

Ingestion into the deeper tiers is triggered dynamically by user activity. When DataQI locates a file that matches the user’s intent, keywords and content from the most relevant files in the search (as well as a selection of semantically similar files) are triggered in the background.

Duplication

DataQI decouples the file location from its content identity using unique file hashes. If the exact same file exists in multiple locations (and is accessible by multiple users), its content is only extracted and embedded once.

What does supported mean?

Because DataQI uses a tiered ingestion strategy, the definition of “supported” depends on the tier being discussed.

Level of support	Description
Fully supported	All relevant content in the file can be processed by the given tier (e.g., all metadata for the Structural Tier, or all plain text for the Content Tier).
Partially supported	Only some of the content in the file can be processed. The file contains a mixture of understood and not understood content types (e.g., a PDF containing both readable text and unreadable images).
Not supported	None of the content in the file can be processed by the given tier. For example, a .bin file is fully supported at the Structural Tier, but not supported at the Keyword or Content Tiers.

Supported file types

File support in DataQI is tier-dependent: the supported file types vary between the Structural, Keyword and Content Tiers.

Structural Tier

At the Structural Tier, every file type is fully supported. DataQI can see the existence, metadata, and location of any file the user has access to in a connected file system, regardless of its format.

Keyword Tier

The Keyword Tier supports a broad range of text-containing files for shallow keyword extraction. Any file type not included in this table can be considered not supported for Keyword Tier ingestion.

File type	Support level	Supported content	Unsupported content
.txt, .md, .html, .htm, .xml, .json, .pdf, .docx, .pptx	Fully supported	Plain text	Everything else (images, audio, advanced formatting, interactive elements, etc.)

Content Tier

The Content Tier performs deep, full-text extraction. The below table outlines the explicit file types and content types supported at the Content Tier. Any file type not included in this table can be considered not supported for Content Tier ingestion.

File type	Support level	Supported content	Unsupported content
.txt, .md, .html, .htm	Fully supported	Plain text, markdown, structural tags	Simulated tabular data (e.g. using spaces, tabs, or pipes to create the appearance of a tabular display) will be treated as plain text.
.pdf	Partially supported	Native text, headings, lists, tables	Images, audio, hyperlinks, or scanned document pages (OCR is unsupported).
.docx	Partially supported	Plain text, headings, lists, tables	Embedded images or audio, advanced formatting, comments, tracked changes, styles, formulae, or interactive elements.
.pptx	Partially supported	Slide text, text inside shapes	Embedded images or audio, advanced formatting, comments, tracked changes, speaker notes, styles, formulae, or interactive elements.

For the purposes of the table above, the term “images” includes raster or vector images of any kind, including but not limited to:

• Photographs & pictures
• Image sequences and videos
• Graphs and charts
• Barcodes/QR codes
• Text contained within a scanned image

Supported content types

Various types of supported content are handled in different ways by DataQI.

Content type	Description
Text	Raw text is the most common type of content that Large Language Models train against (embed). DataQI uses text to augment the LLM’s training to provide accurate, grounded answers.
Text formatting	Simple formatting (e.g. headers, paragraphs, lists) provides structural context to the text data that DataQI understands and preserves alongside the embedded data.