Building a Closed-Domain Agentic AI Knowledge Assistant with Hybrid RAG

I recently built an "Agentic AI Knowledge Assistant" that combines "Retrieval-Augmented Generation (RAG), hybrid search, and an LLM agent" to answer questions strictly from a predefined knowledge base.

One of the main goals of this project was to address a common problem with LLM applications: the model should not answer questions using its general pretrained knowledge when the required information is not available in the knowledge base.

How it works

The system follows a retrieval-first approach:
User Question → Hybrid Retrieval → Relevant Context → AI Agent → Final Answer:
The knowledge base is divided into smaller chunks using "LangChain's RecursiveCharacterTextSplitter". Each chunk is converted into embeddings using:
"sentence-transformers/all-MiniLM-L6-v2"
For retrieval, I implemented two approaches:

• FAISS for semantic vector search
• BM25 for keyword-based search These are combined using a weighted hybrid scoring mechanism: "Hybrid Score = 0.7 × Vector Score + 0.3 × BM25 Score" This allows the system to benefit from both semantic similarity and exact keyword matching.

The workflow looks like this:
User Query
↓
Hybrid Search
↓

FAISS || BM25
Vector || Keyword
Search || Search
↓
Relevant Knowledge Base Chunks
↓
Retrieved Context
↓
Qwen Language Model
↓
Final Answer

Agentic AI Layer

The retrieval system is exposed to the agent through a custom:
knowledge_base_search() tool.
The project uses smolagents CodeAgent along with the:
"Qwen/Qwen2.5-72B-Instruct" model for response generation.
The agent retrieves relevant information from the knowledge base before generating a response.

Closed-Domain Knowledge Restriction

One of the most important features of this project is the strict knowledge-base-only approach.
The assistant is instructed not to use:

• Wikipedia
• Internet searches
• External websites
• External APIs
• External documents
• General pretrained knowledge
• Guessing or assumptions

If the knowledge base does not contain sufficient information, the intended response is:

"The knowledge base does not contain enough content to answer this question."
This makes the system more suitable for applications where responses need to remain within a controlled information domain.

Technologies Used

• Python
• LangChain
• Sentence Transformers
• FAISS
• BM25
• Hugging Face Transformers
• smolagents
• Qwen 2.5 72B Instruct

What I Learned

Building this project helped me understand how different components of an AI application work together rather than treating an LLM as a standalone system.

In particular, I gained practical experience with:

• Document chunking
• Text embeddings
• Vector databases/search
• BM25 retrieval
• Hybrid search
• Retrieval-Augmented Generation
• AI agents and tools
• LLM integration
• Closed-domain AI systems
• Controlling unsupported LLM responses

Future Improvements

Some improvements I would like to implement next include:

• Adding PDF and document ingestion
• Persistent vector storage
• A web-based interface
• Better retrieval evaluation
• A program-level relevance gate before sending queries to the LLM
• Conversation history while maintaining the closed-domain restriction

This project was a great hands-on experience in understanding how RAG + Hybrid Search + Agentic AI + LLMs can be combined to build a more controlled AI assistant.