System overview

This section provides a high-level overview of the DataQI system architecture, designed specifically for system administrators and platform engineers. Understanding these components will assist you in allocating resources, managing network security, and troubleshooting the platform.

DataQI is deployed as a suite of interconnected application and infrastructure services within a Kubernetes cluster. It is designed to be highly scalable and can be deployed in completely air-gapped environments, provided that the required AI models are accessible via a gateway or local hardware (such as local GPUs).

Component architecture

The following diagram illustrates the high-level relationship between the application services, infrastructure data stores, and external integrations.

graph TD
    User([User browser]) --> Gateway(Gateway/Ingress)
    
    subgraph K8sCluster ["K8s cluster"]
        subgraph DataQINamespace ["K8s namespace"]
            
            subgraph AppPods ["Application workloads"]
                subgraph User interface
                    SPA("spa")
                end
                subgraph Orchestration
                    API("api")
                    TelemetryIngress("telemetry-ingress")
                end
                subgraph Workers
                    Extractor("extractor")
                    Transformer("transformer")
                end
                subgraph Security
                    Keycloak("keycloak")
                end
            end
                        
            Gateway --> SPA
            Gateway --> API
            Gateway -- "/telemetry" --> TelemetryIngress
            Gateway --> Keycloak
            API <--> Extractor
            API <--> Transformer

            subgraph Observability
                OtelCollector("otelcollector")
            end
            TelemetryIngress --> OtelCollector
            
            subgraph InfraPods ["Infrastructure workloads"]
                subgraph Databases
                    PostgresKeycloak[("pg-keycloak")]
                    PostgresAPI[("pg-api")]
                    PostgresAudit[("pg-audit")]
                end
                subgraph Caching
                    Valkey[("valkey")]
                end 
                subgraph Messaging
                    NATS{"nats"}
                end
            end
        end
    
        API -- Local GPU hosting --> GPULLM[[LLM]]
    end
    
    API -- Remote/SaaS models --> ExternalAI[[LLM]]
    
    SPA -.-> API
    SPA -. "Authenticated OTLP/HTTP" .-> TelemetryIngress
    
    style GPULLM stroke-dasharray: 5 5
    style ExternalAI stroke-dasharray: 5 5

Application services

• spa: The front-end user interface. This is a lightweight service that serves the static assets to the user’s browser.
• api: The primary backend service. It manages business logic, orchestration of AI requests and user permissions.
• telemetry-ingress: A stateless .NET edge adapter that accepts browser logs, traces, and metrics only from authenticated SPA users. It shares the API hostname under /telemetry, applies request limits, and forwards accepted OTLP traffic to a private collector receiver without forwarding the user’s token.
• keycloak: Manages identity and account federation, deployed and managed by the Keycloak Operator. The Operator handles deployment lifecycle, rolling upgrades, and high-availability clustering. DataQI supports flexible authentication, allowing for local accounts managed directly within Keycloak, or integration with your organisation’s existing Single Sign-On (SSO) provider (which is our recommended approach).
• extractor: Stateless background workers responsible for document content extraction.
• transformer: Stateless background workers responsible for local embedding model execution.

Infrastructure services

• pg-api: The primary relational database for the API’s structured application data persistence, configuration, and state.
• pg-audit: The relational database dedicated to storing cross-service audit events.
• pg-keycloak: The relational database dedicated to Keycloak’s identity data and configuration.
• valkey: A distributed cache used for transient state and as a backplane for real-time web socket communications.
• nats: The message broker that ensures guaranteed message delivery between the core API and the background workers.
• shared-filesystem: An RWX Persistent Volume Claim (PVC) used as a short-lived, cross-pod file cache for transfers between services such as the API and extraction workers. It does not require backup.
• persistent-filesystem: A separate RWX PVC for durable binary files that remain until explicitly deleted and should be included in platform backup policies.
• otelcollector: A telemetry aggregation service. It collects traces, metrics, and logs from DataQI components, sanitises browser telemetry on a separate private receiver, and can then send the signals to your organisation’s internal monitoring tools or an external observability platform.

Deployment considerations

DataQI is built to scale out based on load. Components such as the API and extraction workers can be horizontally scaled across multiple nodes to handle increased demand.

Network egress requirements

If you are deploying into an environment with strict network egress controls, the cluster must be permitted to reach:

• OIDC provider: Outbound HTTPS access to your organisation’s identity provider (e.g., Entra ID, Okta) if integrating with an external Single Sign-On (SSO) solution.
• AI provider endpoints: Outbound HTTPS access to remote LLM APIs (such as Azure OpenAI or Gemini), unless you are relying exclusively on locally hosted GPU models.
• Other external connections: DataQI may be configured to access other external systems such as SMB file shares or business systems, multiple workloads may need access to these based on your specific configuration.

Data persistence

When provisioning storage and planning disaster recovery, it is important to distinguish between stateful components that require backing up and ephemeral components that do not:

• Persistent data (Requires backup):
  • pg-api (Core application data and configuration)
  • pg-audit (Audit logs and event history)
  • pg-keycloak (Identity configuration and local user accounts)
  • persistent-filesystem (Durable binary files)
• Persistent data (No backup required):
  • nats (Used for message queues between services. State is persisted to disk to survive pod restarts, but long-term backups are unnecessary as messages are transient)
  • shared-filesystem (Cross-pod cache files used during processing)
• Ephemeral data:
  • valkey (Used strictly as a transient cache and SignalR backplane, running in-memory)