RAG for agents: where a plain agent falls short and what Foundry IQ changes

This week (October 12–14, 2026) I’m sitting in the AI-103 course, Developing AI Apps and Agents on Azure. One slide compared a plain agent with a RAG agent: five limitations on the left, five fixes on the right. The slide is correct. As service owner for M365 collaboration and Copilot at a service provider with customers in regulated sectors (KRITIS, NIS2), I read it with SharePoint, permissions and governance in mind, and from there several of the fixes look like new work for the people who run the data.

TL;DR
RAG makes an agent only as current, correct and safe as the content and permissions behind it. Oversharing and stale documents in SharePoint turn directly into bad answers.
Foundry IQ gives you one shared knowledge base for many agents, built on Azure AI Search agentic retrieval. That fixes the “every team builds its own pipeline” problem.
As of October 2026, the parts you’d want for SharePoint in a regulated tenant (both SharePoint knowledge sources, LLM query planning, permission enforcement) are still in preview.

RAG in a few sentences

A language model only knows its training data. Retrieval augmented generation (RAG) adds a step before the answer: the agent searches connected sources for passages that fit the question, puts them into the prompt and lets the model answer on that basis, ideally with a reference to where each statement came from. The model stays the same; the material it works with changes.

Five limits of a plain agent

Limit of a plain agent	What happens without RAG	What RAG changes
Knowledge cutoff	Answers reflect the state of the training data, so policies and products can be months or years out of date.	Current documents are pulled from connected sources at question time.
No access to private data	The agent can only offer general knowledge, not your process, contract or system.	Internal knowledge bases become searchable for the agent.
Missing organizational context	Advice ignores your approval chains, workflows and security policies.	Answers are based on the policies and procedures that were retrieved.
Made-up answers	Plausible but wrong statements become a compliance risk.	Answers point to the source they’re based on, so people can check them.
Scaling	Every team builds and runs its own RAG pipeline with its own index, chunking and permission logic.	One shared knowledge platform serves many agents, for example Foundry IQ.

Where the slide gets too optimistic

“Always current” means “as current as the index”

Foundry IQ distinguishes two kinds of knowledge sources. Indexed sources copy content into an Azure AI Search index: chunking, embeddings, metadata, optionally permissions. They’re as fresh as the last indexer run, and you set that schedule yourself. Remote sources store nothing and query the source system on demand at retrieval time, so the content is current by design.

Permissions follow the same logic, and that’s the part I’d watch more closely than content. For indexed SharePoint with ACL ingestion (preview), Microsoft documents that item-level permission changes are picked up on the next indexer run. Changes on a parent scope (site, library, folder) are not; those need an explicit resync. Until then, the index serves stale permissions. If your offboarding or need-to-know process relies on removing someone from a site, that gap belongs in your risk assessment.

A citation doesn’t make an answer correct

A source reference proves that a passage was retrieved. It doesn’t prove the passage was the right one, still valid, or reproduced faithfully. RAG reduces hallucinations; it doesn’t eliminate them. The model can still combine two correct passages into a wrong conclusion, or quote the 2023 version of a policy because it ranked higher than the 2026 one. Citations make errors easier to find, but someone still has to look.

RAG turns SharePoint hygiene into agent quality

With RAG, the agent’s answers are a direct function of what’s in the sources. Everything that already bothers you in SharePoint ends up in the answers:

Oversharing: content shared too broadly is found and quoted, as permission enforcement works with the permissions that exist, not the ones you intended.
Stale documents: an outdated policy that nobody archived competes with the current one.
Duplicates: three copies of the same process description in three versions give the retrieval engine three options to choose from.

Permissions and sensitivity labels become quality controls. Scoping a knowledge source to the right sites, removing “Everyone except external users” shares and archiving outdated content directly improve the answers.

An agent only knows the approval chain that’s written down

RAG can only ground answers in approval chains and security policies that are documented somewhere the agent can reach. In many organizations, the real approval chain lives in people’s heads or in a Teams chat from 2022. In practice, a RAG project often starts as a documentation project.

Foundry IQ in short

Foundry IQ is the managed knowledge layer in Microsoft Foundry (formerly Azure AI Foundry). It consists of three building blocks, all running on Azure AI Search:

Knowledge base: the top-level object. It groups one or more knowledge sources and controls retrieval behavior. Multiple agents can share one knowledge base.
Knowledge sources: connections to indexed or remote content. As of October 2026, Microsoft Learn lists, among others, search index, Azure Blob, OneLake, Azure SQL, file upload, indexed and remote SharePoint, Fabric data agents and ontologies, MCP servers, Work IQ and Bing web grounding.
Agentic retrieval: the retrieval engine. With an optional LLM, it breaks a question into subqueries (query planning), picks the relevant sources, runs searches in parallel, reranks the results and returns extracted content with references. The retrieval reasoning effort controls how much LLM work goes into this: minimal uses no LLM (up to 10 sources), low allows three sources and three subqueries, medium five of each plus an iterative pass.

Agents connect via the knowledge base’s MCP endpoint: in Foundry Agent Service (as an MCP tool), with Microsoft Agent Framework, or in any custom app via REST and SDKs. Copilot Studio reaches a knowledge base indirectly: you connect a Foundry agent that uses the knowledge base to your Copilot Studio agent (preview).

Foundry IQ is one of three “IQ” workloads. Work IQ provides collaboration context from Microsoft 365 (documents, meetings, chats). Fabric IQ models business data in Microsoft Fabric. Foundry IQ connects enterprise knowledge across Azure, SharePoint, OneLake and the web. The three are independent but combinable; Work IQ is even available as a Foundry IQ knowledge source.

SharePoint as a knowledge source

For SharePoint, you have two options, both in preview as of October 2026:

Indexed SharePoint generates an indexer pipeline and copies the content into Azure AI Search. Permissions only apply if you configure ACL ingestion and pass the user token at query time. The documented limitations are worth reading first: sharing links scoped to “Anyone” or “People in your organization” aren’t supported, Entra groups nested inside SharePoint groups aren’t expanded, and the SharePoint indexer page states that tenants with Conditional Access enabled aren’t supported. For most regulated tenants I know, that last point alone is a show-stopper.

Remote SharePoint uses the Copilot Retrieval API and queries SharePoint live on behalf of the user. Nothing is copied; the Retrieval API enforces SharePoint permissions and Microsoft Purview sensitivity labels itself. Microsoft recommends this path for custom RAG apps on SharePoint. The trade-offs: every user needs a Microsoft 365 Copilot license or Retrieval API pay-as-you-go (preview, requires at least one Copilot license in the tenant). There’s a limit of 200 requests per user per hour and a maximum of 25 results per query, OneDrive content isn’t included, and only text is retrieved.

If your users already have Copilot licenses, I’d make remote SharePoint the default: no second copy of your data, no second permission model to keep in sync.

What this means in regulated environments

Permission-aware retrieval isn’t automatic. Microsoft’s docs are explicit: without the user’s token in the request, permission-enabled sources return results unfiltered. If an agent calls the knowledge base with its own managed identity, every user gets the same result set. Build and test the token passthrough before the first pilot, not after.
Traceability is a real benefit. References with URLs and sensitivity label information show which documents an answer was based on. For audit questions, that beats “the model said so”, provided your logging captures it.
Check the preview status line by line. With the stable API version 2026-04-01, only search index, Azure Blob, OneLake and web knowledge sources are generally available, with minimal reasoning effort. Both SharePoint sources, LLM query planning, answer synthesis and the permission features are preview, without an SLA and not recommended for production. The Foundry portal and the Azure portal expose all agentic retrieval features as preview only. For a KRITIS customer, that’s a pilot, not a production service.

In my post on Microsoft’s four-stage model for responsible generative AI, grounding is the third of four mitigation layers, alongside the system message. Foundry IQ is a concrete way to build that layer. It doesn’t remove the need for the other three.

Takeaways

Clean up before you connect. Oversharing, stale documents and duplicates in SharePoint become agent errors. A site cleanup is the cheapest quality improvement you’ll get.
Prefer remote SharePoint if users have Copilot licenses. No second copy, permissions and labels enforced at the source, no ACL sync to monitor.
Test the user token path explicitly. Without it, permission-enabled sources return unfiltered results, and every user sees the same content.
Treat Foundry IQ on SharePoint as a pilot for now. Plan production use once the SharePoint knowledge sources and permission features reach general availability.