Less Is More: Why Retrieving Fewer Documents Can Improve AI Answers

There is a poorly understood principle: when it comes to retrieval-augmented generation (RAG), sometimes less is more. Starting from RAG implementation in AI assistants, organizations have raced to feed these systems with documents. It turns out that an abundance of documents may actually harm performance for many organizations.

The Information Overload Problem

Modern AI systems such as large language models (LLMs) have the amazing ability to understand and respond to speech or text. However, these systems suffer from retrieval contexts that are too rich; they are deeply overwhelmed, like a human trying to write an essay while reading countless reference books.

---

---

With regards to this phenomenon, Dr. Maya Patel, TechFusion AI labs Chief Scientist, stated, “This is what we call a digital information overload.”

“An AI program suffers with identifying the most pertinent details when flooded with documents that have lateral relevance to the question at hand. Everything is reduced to background sounds.”

Dr. Patel elaborated her perspective on information overload through her experiences with RAG systems in automation. The implementation of AI and automation in changing facets of customer service, finance, and even healthcare leads to an unshakeable conclusion: the quality of information retrieval overshadows the quantity.

Quality Over Quantity: The Research Findings

Stanford’s AI Lab has put a number to that effect. In research focused on RAG systems with differing retrieval thresholds, it was noted that precision drops sharply when more than five to seven documents are retrieved for a single query.

“We tracked the relevance of answers as well as their factual accuracy,” remarks lead researcher Wei Zhang.“ Even when the set of pulled 10 – 15 documents was much less relevant, the systems pulling 3 – 5 highly relevant documents consistently outperformed. It was bafflingly more accurate, after all.”

The key metrics from the study are telling:

• Systems retrieving 3-5 relevant documents achieved 92% factual accuracy
• Systems retrieving 10+ documents saw accuracy drop to 76%
• User satisfaction scores were 34% higher with fewer, more relevant retrievals
• Response generation time decreased by 40% with smaller context windows

---

---

The Cognitive Load Theory of AI

What explains this seemingly paradoxical effect? Experts point to parallels with human cognition.

“LLMs, despite their different architecture, face similar attention and working memory constraints as humans,” explains cognitive scientist Dr. James Moreno. “When presented with too much information, they exhibit something akin to cognitive load issues… struggling to maintain focus on the most relevant details while processing excessive context.”

This insight has led to the development of what some researchers call “AI cognitive load theory,” which suggests optimal information thresholds for different types of AI tasks.

Implementing the Less-is-More Approach

Forward-thinking organizations are already adapting their RAG implementations based on these findings. Key strategies include:

1. Semantic Chunking

Rather than arbitrarily splitting documents by token count or paragraph breaks, advanced systems now use semantic chunking—dividing information based on conceptual boundaries and topic coherence.

“Think of it as creating logical thought units rather than arbitrary text blocks,” explains Sanjay Mehta, CTO of DataSense AI. “This ensures that retrievals contain complete ideas rather than fragmented information.”

2. Two-Stage Retrieval

Many cutting-edge systems now implement a two-stage retrieval process: first casting a wide net to identify potentially relevant documents, then applying a more discriminating reranking algorithm to select only the most pertinent information.

“The initial retrieval might surface 20-30 documents,” notes RAG specialist Olivia Chen. “But the reranker will ruthlessly prune this down to the 3-5 most information-dense and relevant passages before sending them to the LLM.”

---

---

3. Confidence-Based Retrieval

Perhaps most intriguing is the trend toward confidence-based retrieval, where systems dynamically adjust how much information they pull based on their confidence in answering a query.

“For straightforward questions where the AI has high confidence, it might retrieve just one or two documents for verification,” explains Dr. Patel. “For complex queries requiring nuanced understanding, it might retrieve more—but still carefully filtered for relevance.”

Real-World Impact

The benefits of this approach extend beyond mere academic interest. Companies implementing these “less is more” RAG strategies report significant improvements:

• Financial services firm Meridian Capital reduced hallucinations in their advisor AI by 78% after implementing semantic chunking and stricter retrieval thresholds
• Healthcare provider NextGen Medical saw diagnosis support accuracy improve by 23% when they switched to quality-focused retrieval
• Customer service platform ResponseIQ decreased resolution times by 35% while improving satisfaction scores

The Future of Information Retrieval

With the passage of time, the researchers in this field have started to look into new elegant solutions to the retrieval problem at hand.

“A few documents can be put aside but my system is able to choose exactly when to get them,” says Zhang, who has a computer science background. “Some queries benefit from upfront retrieval, while others are better served by targeted retrieval after the model has begun formulating a response.” Zhang further contextualizes other scenarios where my approach accelerates productivity.

“The multi-hop retrieval approach is able to follow a user’s lead as the user tries to fill gaps in their knowledge with information available instead of following a script like approach.”

The Human Parallel

Perhaps the most compelling aspect of this research is how it mirrors our own experience with information processing. In an age of information abundance, humans too struggle with distinguishing signal from noise.

“There’s something profound in the discovery that AI systems, like humans, can become less effective when overloaded with information,” reflects Dr. Moreno. “It suggests that effective thinking—whether biological or artificial—isn’t just about having access to information, but about having the right information at the right time.”

For businesses and AI developers alike, the lesson is clear: when it comes to retrieval-augmented generation, the path to better AI doesn’t lie in retrieving more, but in retrieving smarter. In the quest for AI that provides clear, factual, and helpful responses, less truly is more.

---

---

---