Verified against Microsoft Learn documentation for Foundry IQ and Azure AI Search agentic retrieval as of July 2026. Foundry IQ is evolving quickly, and several capabilities are preview-gated by API version, so treat specific limits and availability as current-at-writing and confirm against the linked Microsoft pages before relying on them.
What Problem It Solves
Generative models are trained on a large but finite corpus with a fixed cutoff date and no access to private data. To answer questions about information the model was never trained on, that information has to be supplied at request time. Retrieval-augmented generation (RAG) does this: an application retrieves relevant material from an external source and adds it to the prompt before the model generates. Retrieval quality sets the ceiling on output quality. Weak retrieved data produces weak answers regardless of how capable the model is.
Classic RAG on Azure AI Search uses a single-query pipeline against a single index, where each index represents a single source. Real enterprise questions rarely map cleanly to one source, and stitching several sources together has traditionally been the application developer's problem. Foundry IQ exists to move that orchestration out of the application and into a managed knowledge layer.
Foundation: How Azure AI Search Retrieves
Foundry IQ is built on the agentic retrieval capability of Azure AI Search, so the underlying retrieval mechanics carry forward.
Search across indexed content combines two methods:
- Lexical (keyword) search works well for high-discrimination terms such as a product SKU or a proper name.
- Vector (semantic) search handles natural language, where many words share a meaning, one word carries several meanings, and idioms cannot be read literally. Content is chunked, run through an embedding model to produce high-dimensional vectors that represent meaning, and stored in a vector index. Queries are embedded the same way, and the closest vectors are the closest in meaning.
Hybrid search runs both and fuses the result sets, then a semantic reranker reorders and scores candidates against the query and keeps the most relevant. This is strong, but a classic query still targets one index tied to one source. That single-source limit is what the agentic layer removes.
What Foundry IQ Adds
Foundry IQ provides a managed, reusable knowledge layer rather than a per-source integration. It replaces the single-query pipeline with an agentic, multi-query one: a single request can plan across multiple sources, decompose the question into subqueries, run them in parallel, rerank each, and merge the results.
Knowledge Sources and Knowledge Bases
Two concepts define the model:
- A knowledge source is one input the engine can search (an Azure AI Search index, a blob container, a SharePoint site, OneLake, the web, and so on).
- A knowledge base is a reusable, topic-centric collection of knowledge sources plus an optional LLM and the parameters that govern retrieval. It is the single entity an app or agent queries.
An application points at one knowledge base and receives grounding data across every source it contains. This removes the per-agent plumbing that classic RAG required: multiple separate queries stitched together in application code. The same knowledge base can ground many different agents and apps.
Supported Knowledge Source Types
Indexed sources are ingested into Azure AI Search, where Foundry IQ manages the full pipeline automatically (ingest, chunk, vectorize, enrich):
- Existing Azure AI Search indexes
- Azure Blob Storage
- A specific SharePoint site (indexed)
- Fabric OneLake, including Fabric IQ, whose ontology maps raw OneLake data to real enterprise entities and relationships
Remote sources are queried in place and bypass local indexing:
- Remote SharePoint via M365, which uses M365's own semantic index scoped to the requesting user (requires a Microsoft 365 Copilot license) and respects Microsoft Purview sensitivity labels and access control lists during retrieval
- Web, powered by the Bing Grounding API
- MCP as a knowledge source, in private preview (see the MCP note below, which distinguishes this from how a knowledge base is exposed to agents)
The indexed versus remote distinction is invisible at the query API but matters operationally. Indexed sources mean you own indexing, storage, and freshness, and get automatic synchronization. Remote sources offload that but add dependencies such as the Copilot license, the Bing Grounding API, or preview availability.
Configuring a Knowledge Base
Setup requires an existing Azure AI Search resource, then one or more knowledge bases created against it. Three configuration decisions define behavior and cost.
1. Retrieval Reasoning Effort
A knowledge base can reference an LLM (from Azure OpenAI in Foundry Models) for planning and synthesis and sets a retrieval reasoning effort that controls how much LLM processing each query gets. The effort defaults to low.
- Minimal — No LLM query planning. Runs direct keyword and vector searches across every source in the knowledge base and returns the best-matching passages. Predictable and cheapest. Per-source overrides are ignored since all sources are always searched. Recommended for migrating from the classic Search API or when you want to manage query planning yourself.
- Low (default) — Adds LLM query planning and source selection. The engine reads source names, index descriptions, and retrieval instructions to select sources and generate focused subqueries.
- Medium — Everything in low, plus reflective search. A semantic classifier evaluates the first pass, and if results are insufficiently relevant it runs a follow-up iteration with a revised query plan that can refine terms or add sources such as the web. Region-restricted, with higher resource limits than low and minimal.
Higher effort costs more tokens, adds latency, and increases cost, in exchange for higher relevance. Microsoft's own benchmarking reports the largest jump comes from minimal to low (query planning), with medium's reflective pass adding further gains. The correct level depends on question complexity, not on defaulting to the highest.
2. Output Mode
Two modes control what the retrieve call returns:
- Extractive data returns raw, verbatim content for an agent to reason over. This is Microsoft's recommended mode for most Foundry IQ scenarios, and the right choice when your own agent does the reasoning. Synthesizing on top of it would be redundant work.
- Answer synthesis uses the LLM to generate a complete natural-language answer plus references. Suited to standalone apps where retrieval output goes straight to the user with no agent in between. It runs slower.
Adding any web source forces answer synthesis. If your architecture assumed extractive chunks, introducing a web source changes that behavior.
3. Descriptions, Instructions, and Overrides
At low and medium effort, source selection depends on what you write. Source selection uses the knowledge source name, the description on an indexed source, and the natural-language retrieval instructions defined on the knowledge base or passed on the retrieve request. Weak descriptions produce bad routing. Retrieval instructions also guide prioritization, for example: prefer the internal source for technical questions and fall back to web only when no answer is found. Setting alwaysQuery on a source forces it into every query regardless of effort. This is prompt engineering relocated into knowledge base configuration, and it is easy to under-invest in.
The Agentic Flow
At low and medium effort, a request runs as: plan and select sources, decompose into subqueries, run the subqueries in parallel (keyword, vector, or hybrid), semantically rerank each, retain references for citation, and merge into a unified response. At medium, a reflective pass can trigger a second iteration. A single complex request can produce several planning iterations and dozens of retrieval calls before returning an answer with its supporting references. Merged content is always returned; source references and the execution activity log are optional and useful for debugging query plans.
Limits, Availability, and Cost
Because these values change and are gated by API version, confirm against the Azure AI Search service limits and Foundry IQ FAQ pages before relying on them.
Capacity. Limits vary by pricing tier, API version, and reasoning effort. In the 2026-05-01-preview, most paid tiers allow up to 10 knowledge sources per knowledge base, and all effort levels share the same source limit. Basic services created before April 3, 2024 are capped at 5 knowledge bases and 5 sources. A free tier exists with limited capability but managed-identity role-based access requires Basic tier or higher.
Per-query caps are separate from total caps. These bound one retrieval call, not the whole knowledge base. Medium effort supports up to 5 knowledge sources and 5 subqueries per query with a 10,000-token budget. Do not confuse this with the total sources a knowledge base can hold.
Availability. Agentic retrieval is generally available in the 2026-04-01 REST API for minimal, extractive retrieval. The full feature set, including using an LLM with non-web sources and the preview knowledge source types, requires the 2026-05-01-preview. The Azure portal and Microsoft Foundry portal expose the preview features. An LLM is required when a web source is present, optional for other sources in the preview API, and unsupported in the GA API.
Billing splits across two services. Azure AI Search bills for retrieval tokens consumed during subquery execution and semantic ranking, with a free monthly allowance and pay-as-you-go after. Azure OpenAI bills separately, always pay-as-you-go, for the input and output tokens used in query planning and answer synthesis, based on the model you assign. Effort level and output mode are therefore direct cost levers on both bills.
A Note on MCP
MCP appears in two distinct roles that are easy to conflate:
- As transport to the agent. A knowledge base is exposed to an agent through an MCP tool (
knowledge_base_retrieve). This is the standard way Foundry Agent Service connects to a knowledge base and is generally available. - As a knowledge source. MCP as a source type inside a knowledge base is in private preview. This is separate from the transport role and may not appear in your portal.
Where Foundry IQ Sits Among the IQ Products
Microsoft positions three IQ workloads that target different domains and are meant to work together:
- Foundry IQ: organizational knowledge for agents
- Fabric IQ: business and analytics data, via an ontology over OneLake
- Work IQ: Microsoft 365 work signals and user context
The framing sometimes used is a data-to-knowledge progression, where raw data becomes information through processing and information becomes knowledge through context and interpretation that can be reasoned over. The IQ products supply the knowledge layer; the agents built on top supply the reasoning. Read "answer any question" as marketing rather than a guarantee, since retrieval quality still bounds every result.
Practical Takeaways
- For multi-source grounding on the Microsoft stack, Foundry IQ removes real orchestration code and is worth adopting.
- Reasoning effort is the setting you will revisit most, because it governs the relevance-versus-cost-and-latency trade-off across two separate bills. Start at the default low and move to medium only where reflection measurably helps.
- Use extractive data when your own agent reasons over the content; reserve answer synthesis for standalone apps. Remember that a web source forces synthesis.
- Source descriptions and retrieval instructions are load-bearing at low and medium effort. Budget real time for them.
- For governance-sensitive data, confirm the Copilot license requirement and Purview label handling on remote SharePoint, and verify permission-aware retrieval end to end.
- Treat every limit and availability statement here as version-gated and check the live Microsoft pages before committing an architecture.
No comments yet