A support lead asks the company assistant a simple question: what is our refund policy for annual plans?
The assistant finds a lot. The current policy page. The version it replaced two years ago. A Slack thread where three people debated an exception for a large customer. A proposal to change the policy that was written, discussed, and never adopted. A support macro that paraphrases an older version of the page.
Every one of those sources is real, and every one mentions refunds on annual plans. The answer that comes back blends them. It is fluent and confident, and it describes a policy the company does not actually have.
Nothing was missing. That was the problem.
The Instinct Is to Give It Everything
When an AI answer is wrong, the natural diagnosis is that it did not know enough. So the fix feels obvious: connect more systems, index more folders, raise the limit on how much text goes into each answer. If the answer lives somewhere in the company, surely including more of the company makes it more likely to be found.
Sometimes that is right. An assistant that cannot see your ticketing system will not answer questions about tickets. Coverage matters.
But coverage and context are different things. Coverage is what the system can reach. Context is what actually goes into a specific answer. Treating them as the same thing is how companies end up with an assistant that can see everything and still answers badly.
Why More Can Be Worse
A model answering a question works from whatever text it is handed for that question. It does not know which of those passages is the current policy and which is a draft someone abandoned. It sees words that are relevant to the question, and it tries to produce an answer consistent with all of them.
Add a passage that is related but wrong, and the model has to decide what to do with it. Sometimes it ignores it. Sometimes it averages it in. Sometimes it picks it, because it was phrased more clearly than the right one.
Notice what did not go wrong in the refund example. The current policy page was retrieved. It was sitting right there in the input. The answer still came out wrong, because the passages around it pulled in other directions. That is a different failure from finding the wrong source, and it is the one that gets worse as you add more.
There is research on this too. A 2023 study from researchers at Stanford and collaborating institutions, Lost in the Middle, found that language models used relevant information less reliably when it sat in the middle of a long input than when it appeared near the beginning or the end. Models have improved since, but the practical lesson has held up: a longer input is not automatically a better one. The right passage can be present and still lose to the passages around it.
Inside a company, the passages around it are rarely random noise. They are near misses. The old version of the policy. The draft. The exception that applied to one customer. The Slack message that was right at the time. Those are the hardest kind of noise, because they look exactly like the answer.
The Right Source, Outvoted
Most AI systems decide what to include by looking for text that resembles the question. That is a reasonable way to find candidates, and it tends to find the right source. The trouble is everything it finds alongside it.
A proposal to change the refund policy is very similar to the refund policy. A two-year-old version of the pricing page is almost identical to the current one. A thread where someone asked whether an exception was possible will look a lot like a thread where one was approved. All of them get included, and once they are in the input, they compete with the passage that is actually correct.
In Your AI Found the Right Document. It Still Gave You the Wrong Answer. the problem was knowing which source the company stands behind. This is the next problem down. Even when the right source is in hand, five of its cousins came along for the ride, and the answer has to survive them.
This is also why "just connect everything" gets riskier as a company grows. More systems mean more drafts, more versions, more threads, and more near-duplicates. The pile of plausible but wrong material grows faster than the pile of current, authoritative material.
Focus Is a Design Decision
The alternative is not to connect less. It is to be deliberate about which sources should inform which kinds of work.
A support team answering customers should draw on the current knowledge base, published policies, and recent resolved tickets. It probably should not draw on the product team's brainstorming docs, even though those docs mention the same features. An engineering team debugging an incident needs the runbooks, the code, and the recent deploy history, not the sales deck that describes the same system in marketing terms.
None of that is about hiding information. The brainstorming doc is not secret. It is just not what a support answer should be built from.
This is where most of the leverage is. A smaller, well chosen set of sources beats a large undifferentiated one, because every source in the smaller set earned its place. The question to ask about any AI deployment is not "how much can it see?" It is "for this team and this kind of question, what should it be looking at?"
How Modly Handles It
Modly connects the systems a company already uses, such as Slack, Jira, Confluence, GitHub, Salesforce, and Google Drive. Connecting a system is the coverage step. What happens next is what keeps coverage from turning into overload.
Context Profiles decide what a team's answers draw from. Sources are bundled into role-specific profiles, so a support profile can point at the knowledge base, policies, and ticket history, while an engineering profile points at code, runbooks, and technical docs. The same question asked in different profiles draws on different material, because the work is different. We covered the idea in more depth in Context Profiles: Why AI Shouldn't Treat Everyone the Same.
Permissions narrow it further, on every query. Within a profile, each person only gets answers built from sources they are allowed to access. Focus and access control are different jobs, and Modly does both.
Every answer cites what it used. When the refund answer comes back, the sources behind it are listed inline. If an abandoned proposal made it into the answer, the citation shows it, and the person asking can see immediately that something is off. Without citations, context overload is invisible. With them, it is one click away from being caught.
Modly says "I don't know" when the sources do not support an answer. A focused context sometimes means the answer is not there. Saying so is more useful than stretching a near-miss into something that sounds complete.
None of this makes stale documents disappear. If an old policy page is still sitting in a folder a profile includes, it can still show up. The difference is that a team can choose what its profile includes, and can see from the citations when something should not be there.
What Changes
Go back to the support lead and the refund question.
With a support profile built around the current policy pages, the knowledge base, and recent resolved tickets, the abandoned proposal and the product team's discussion threads are not part of what the answer is built from. The current policy has far less to compete with, and the answer cites what it used, so the support lead can check it in seconds.
That is not a guarantee. If the old version of the page still sits in a folder the profile includes, it can still make it into an answer. The difference is that the citation makes it visible, and the fix is a decision about the profile, not a mystery about the model.
That is the shift. Coverage is what your AI can reach. Context is what it should use for this question. Most teams have spent their effort on the first. The second is where answers get better: give each team the right sources, and make it obvious which ones were used. It sounds like less. In practice, it is how an assistant goes from impressive in a demo to reliable in a real company.
Post three was about an AI seeing more than a person should. This one is about an AI using more than an answer should.
If you want to see how Context Profiles and cited answers work on your own systems, we run a 30-minute walkthrough on real data: Book a demo.
Related reading:
