Continuous Blob Knowledge Ingestion and Retrieval
Outcome
Continuously detect new or changed Azure Blob Storage objects, parse and chunk them, generate embeddings, persist chunks and vectors in Azure Cosmos DB, and let platform engineers compare vector, full-text, hybrid and semantic-reranked retrieval over the same corpus while honoring source ACLs, versions and ingestion status.
| Portfolio attribute | Value |
|---|---|
| Primary persona | AI platform engineer and data engineer |
| Primary aggregate | SourceDocumentIngestion |
| Cosmos DB containers | 1 |
| Grounding data | Blob documents, object metadata, ACLs, versions, and ingestion status |
| Development estimate | USD 37-360 monthly dev estimate |
| Runtime options | Python FastAPI and C# ASP.NET Core |
Productivity hypothesis
These measures reflect the workflow, evidence, decision boundary, and industry risk for this use case.
| Impact type | Measure | Conservative internal assumption | Customer measurement |
|---|---|---|---|
| Productivity | Blob-change-to-searchable-content time | 20-35% reduction | (baseline median blob-change-to-searchable minutes - assisted median minutes) / baseline median minutes * 100 |
| Quality or risk | Content searchable within freshness and relevance SLA | 10-20% relative improvement | (assisted eligible chunks searchable within SLA - baseline rate) / baseline rate * 100 |
Important: Conservative internal pilot assumptions only; not public benchmarks, commitments, or expected customer results. Validate against a comparable baseline or matched control cohort with mandatory human review.
Rationale: The engineering workflow benefits from faster ingestion-to-search and better retrieval freshness, but it is a platform capability rather than a direct business outcome.
Pilot guardrail: Preserve ACLs, versioning, deletion propagation, freshness, and protection from unauthorized retrieval.
Customer validation plan
- Baseline the current workflow with representative users and tasks.
- Run the accelerator on the same task mix with human review retained.
- Measure blob-change-to-searchable-content time and content searchable within freshness and relevance sla with the documented formulas.
- Replace internal assumptions with customer-observed results before a business case.
Why Azure Cosmos DB
This AI solution needs an operational data plane, not only a vector index:
- Hybrid vector and full-text retrieval combines semantic recall with exact identifiers, codes, names, and domain language.
- Hierarchical partition keys and metadata predicates keep tenant and domain boundaries in the normal query path.
- Operational JSON, source chunks, workflow and agent state, feedback, and audit records remain together for low-latency reads and governed writes.
- Change feed reacts to source and operational updates without polling.
- Autoscale throughput, configurable consistency, and multi-region distribution support variable demand and enterprise availability.
Domain workflow

| Stage | Actor | Input → output | Approval |
|---|---|---|---|
| 1. Blob change detection | Event Grid blob-created and blob-deleted subscription | blobChangeEvent → SourceDocument | Policy controlled |
| 2. Parse, chunk and embed | Ingestion function with Document Intelligence and Azure OpenAI embeddings | SourceDocument → ContentChunk | Policy controlled |
| 3. Version cutover and stale chunk retirement | Change-feed ingestion coordinator | IngestionRun → currentVersionChunkSet | Policy controlled |
| 4. Retrieval mode comparison | AI platform engineer | RetrievalExperiment → RetrievalResultSet | Policy controlled |
| 5. Engineer promotion of a retrieval configuration | AI platform engineer or data engineer | RetrievalResultSet → promotedRetrievalConfig | Human required |
Technical architecture

Cosmos DB data model
| Container | Hierarchical partition key | Documents | Search paths |
|---|---|---|---|
sourceDocumentIngestions | /tenantId, /sourceDocumentId | SourceDocument, IngestionRun, ContentChunk, RetrievalExperiment, RetrievalResultSet, GroundedAnswer | /contentVectorEmbedding, /queryVectorEmbedding, /contentVector, /queryVector |
Query contract
| Query | Container | Scope | Purpose |
|---|---|---|---|
| PureVectorRetrieval | sourceDocumentIngestions | cross-partition-bounded | Rank current-version chunks by vector distance alone as the semantic baseline in the comparison. Intentional cross-partition corpus retrieval: the ingested corpus spans sourceDocumentId partitions by design, so the scan is bounded by the tenantId prefix, the current-version flag, the caller's ACL principals and a topK limit. |
| FullTextBm25Retrieval | sourceDocumentIngestions | cross-partition-bounded | Rank the same chunk set by lexical BM25 scoring to expose where keyword matching beats embeddings. Intentional cross-partition corpus retrieval: bounded by the tenantId prefix, the current-version flag, the caller's ACL principals and a topK limit over the full-text index. |
| HybridRrfRetrieval | sourceDocumentIngestions | cross-partition-bounded | Fuse vector and full-text rankings with reciprocal rank fusion for the hybrid arm of the experiment. Intentional cross-partition corpus retrieval: bounded by the tenantId prefix, the current-version flag, the caller's ACL principals and a topK limit fused by reciprocal rank fusion. |
| IngestionBacklogAndFailures | sourceDocumentIngestions | cross-partition-bounded | Show blobs whose newest version is not yet searchable so staleness can be reported honestly in answers. Intentional cross-partition control-plane query: bounded by the tenantId prefix, the non-terminal ingestion states and the freshness SLA cutoff. |
Python and C# services route each document type to its declared container and hierarchical partition key. ETag-based writes protect mutable aggregates, while append-only records and bounded cross-partition queries preserve scale.
Azure deployment

Clone the source repository, authenticate with Azure, and run:
Set-Location accelerators/technology-software/continuous-blob-knowledge-ingestion-and-retrieval
az login
./Deploy.ps1 -Runtime python -Location eastus2
Change python to csharp to deploy the .NET implementation.
Explore the implementation
- Launch signed-in demo
- Implementation article
- Domain data model
- Workflow and API contract
- Python application
- C# application
- Domain container Bicep
- Platform Bicep
- Cost and configuration
- Source repository