Skip to main content
TEC-04 · Technology & Software

Continuous Blob Knowledge Ingestion and Retrieval

Outcome​

Continuously detect new or changed Azure Blob Storage objects, parse and chunk them, generate embeddings, persist chunks and vectors in Azure Cosmos DB, and let platform engineers compare vector, full-text, hybrid and semantic-reranked retrieval over the same corpus while honoring source ACLs, versions and ingestion status.

Portfolio attributeValue
Primary personaAI platform engineer and data engineer
Primary aggregateSourceDocumentIngestion
Cosmos DB containers1
Grounding dataBlob documents, object metadata, ACLs, versions, and ingestion status
Development estimateUSD 37-360 monthly dev estimate
Runtime optionsPython FastAPI and C# ASP.NET Core

Productivity hypothesis​

These measures reflect the workflow, evidence, decision boundary, and industry risk for this use case.

Impact typeMeasureConservative internal assumptionCustomer measurement
ProductivityBlob-change-to-searchable-content time20-35% reduction(baseline median blob-change-to-searchable minutes - assisted median minutes) / baseline median minutes * 100
Quality or riskContent searchable within freshness and relevance SLA10-20% relative improvement(assisted eligible chunks searchable within SLA - baseline rate) / baseline rate * 100

Important: Conservative internal pilot assumptions only; not public benchmarks, commitments, or expected customer results. Validate against a comparable baseline or matched control cohort with mandatory human review.

Rationale: The engineering workflow benefits from faster ingestion-to-search and better retrieval freshness, but it is a platform capability rather than a direct business outcome.

Pilot guardrail: Preserve ACLs, versioning, deletion propagation, freshness, and protection from unauthorized retrieval.

Customer validation plan​

  1. Baseline the current workflow with representative users and tasks.
  2. Run the accelerator on the same task mix with human review retained.
  3. Measure blob-change-to-searchable-content time and content searchable within freshness and relevance sla with the documented formulas.
  4. Replace internal assumptions with customer-observed results before a business case.

Why Azure Cosmos DB​

This AI solution needs an operational data plane, not only a vector index:

  • Hybrid vector and full-text retrieval combines semantic recall with exact identifiers, codes, names, and domain language.
  • Hierarchical partition keys and metadata predicates keep tenant and domain boundaries in the normal query path.
  • Operational JSON, source chunks, workflow and agent state, feedback, and audit records remain together for low-latency reads and governed writes.
  • Change feed reacts to source and operational updates without polling.
  • Autoscale throughput, configurable consistency, and multi-region distribution support variable demand and enterprise availability.

Domain workflow​

Domain workflow

StageActorInput → outputApproval
1. Blob change detectionEvent Grid blob-created and blob-deleted subscriptionblobChangeEvent → SourceDocumentPolicy controlled
2. Parse, chunk and embedIngestion function with Document Intelligence and Azure OpenAI embeddingsSourceDocument → ContentChunkPolicy controlled
3. Version cutover and stale chunk retirementChange-feed ingestion coordinatorIngestionRun → currentVersionChunkSetPolicy controlled
4. Retrieval mode comparisonAI platform engineerRetrievalExperiment → RetrievalResultSetPolicy controlled
5. Engineer promotion of a retrieval configurationAI platform engineer or data engineerRetrievalResultSet → promotedRetrievalConfigHuman required

Technical architecture​

Technical architecture

Cosmos DB data model​

ContainerHierarchical partition keyDocumentsSearch paths
sourceDocumentIngestions/tenantId, /sourceDocumentIdSourceDocument, IngestionRun, ContentChunk, RetrievalExperiment, RetrievalResultSet, GroundedAnswer/contentVectorEmbedding, /queryVectorEmbedding, /contentVector, /queryVector

Query contract​

QueryContainerScopePurpose
PureVectorRetrievalsourceDocumentIngestionscross-partition-boundedRank current-version chunks by vector distance alone as the semantic baseline in the comparison. Intentional cross-partition corpus retrieval: the ingested corpus spans sourceDocumentId partitions by design, so the scan is bounded by the tenantId prefix, the current-version flag, the caller's ACL principals and a topK limit.
FullTextBm25RetrievalsourceDocumentIngestionscross-partition-boundedRank the same chunk set by lexical BM25 scoring to expose where keyword matching beats embeddings. Intentional cross-partition corpus retrieval: bounded by the tenantId prefix, the current-version flag, the caller's ACL principals and a topK limit over the full-text index.
HybridRrfRetrievalsourceDocumentIngestionscross-partition-boundedFuse vector and full-text rankings with reciprocal rank fusion for the hybrid arm of the experiment. Intentional cross-partition corpus retrieval: bounded by the tenantId prefix, the current-version flag, the caller's ACL principals and a topK limit fused by reciprocal rank fusion.
IngestionBacklogAndFailuressourceDocumentIngestionscross-partition-boundedShow blobs whose newest version is not yet searchable so staleness can be reported honestly in answers. Intentional cross-partition control-plane query: bounded by the tenantId prefix, the non-terminal ingestion states and the freshness SLA cutoff.

Python and C# services route each document type to its declared container and hierarchical partition key. ETag-based writes protect mutable aggregates, while append-only records and bounded cross-partition queries preserve scale.

Azure deployment​

Azure deployment

Clone the source repository, authenticate with Azure, and run:

Set-Location accelerators/technology-software/continuous-blob-knowledge-ingestion-and-retrieval
az login
./Deploy.ps1 -Runtime python -Location eastus2

Change python to csharp to deploy the .NET implementation.

Explore the implementation​

Return to all accelerators