Skip to content
DataQI Documentation

Data ingestion

DataQI uses a highly efficient Demand-Driven approach to process your data. Rather than attempting to aggressively process and deeply analyse the meaning of every single file across your enterprise on day one—which uses too much computing power, is slow, and risks cluttering the system with irrelevant or inaccessible files—DataQI prioritises processing based on what you are actually searching for.

The 3-tiered ingestion model

Section titled “The 3-tiered ingestion model”

DataQI processes files dynamically through up to three levels of processing:

  1. Tier 1 (Structure): When a file is discovered, DataQI instantly captures its basic structure (filename, path, and size). This ensures the file is immediately visible to the platform’s routing engine, allowing it to be discovered by search even while deeper processing runs in the background.
  2. Tier 2 (Keyword): To avoid blindly processing all files, this tier is initially triggered when your searches match a file’s basic structure. Once triggered, the system quickly scans the file to generate semantic keywords and summaries, and then automatically continues in the background to scan related neighbour files.
  3. Tier 3 (Content): Deep reading of the full text and vectorisation. This intensive processing is purely reactive—it is triggered only when your search intent or keyword matches indicate that the document is highly relevant to your query.
graph TD
    A[New File Discovered] -->|Background| B[Tier 1: Structure]
    B -->|Triggered by Search & Background| C[Tier 2: Keyword]
    C -->|Triggered by Search| D[Tier 3: Content]

By treating natural language queries as the catalyst, DataQI continuously guides its own ingestion pipeline, focusing deep processing power precisely where they provide the most value.