Building an Agentic RAG AI Agent with FAISS, BM25 and Qwen

I recently completed a project as part of my AI Engineering learning journey: an Agentic RAG AI Agent that answers user questions based on a provided knowledge base.

The main goal of this project was to build an AI assistant that does not simply rely on its general knowledge. Instead, it first searches a knowledge base, retrieves relevant information, and then generates an answer using that retrieved context.

What is the idea behind the project?

Large Language Models can sometimes answer questions using their pre-trained knowledge, even when that information is not available in the documents provided to the system. To address this, I built a Retrieval-Augmented Generation (RAG) pipeline.
The workflow looks like this:

User Query
↓
Hybrid Search
↓

FAISS || BM25
Vector || Keyword
Search || Search
↓
Relevant Knowledge Base Chunks
↓
Retrieved Context
↓
Qwen Language Model
↓
Final Answer

Technologies Used

• Python – Main programming language
• Sentence Transformers – Generating text embeddings
• FAISS – Semantic vector similarity search
• BM25 – Keyword-based search
• Hybrid Search – Combining vector and keyword search
• Qwen2.5-72B-Instruct – Generating the final response
• Hugging Face Transformers – Loading and using the language model
• Visual Studio Code – Development environment

How the system works

1. Knowledge Base

The system starts with a predefined knowledge base containing information about specific topics.
The goal is to ensure that the AI assistant answers questions only from this available information.

1. Document Chunking

Large documents are divided into smaller chunks.
This helps the retrieval system find the specific section of a document that is most relevant to the user's question.

1. Embedding Generation

Each text chunk is converted into a numerical representation called an embedding using a Sentence Transformer model.
These embeddings allow the system to perform semantic search based on meaning rather than only exact keywords.

1. FAISS Vector Search

When a user asks a question, the question is also converted into an embedding.
FAISS compares the query embedding with the document embeddings and retrieves the most semantically similar chunks.

1. BM25 Keyword Search

In addition to semantic search, I also implemented BM25 for keyword-based retrieval.
This is useful for finding exact terms, technical concepts, and important keywords.

1. Hybrid Search

The results from FAISS and BM25 are combined to improve retrieval quality.
This gives the system the advantages of both semantic understanding and exact keyword matching.
Hybrid Search = Vector Search + Keyword Search

1. Answer Generation with Qwen

The most relevant chunks are combined into a context.

The Qwen language model receives:

• The retrieved knowledge base context
• The user's question
• Strict instructions to answer only from the provided context The model then generates the final answer.

Example Interaction: Question Inside the Knowledge Base

You: What is RAG?

Searching knowledge base...

Agent: RAG stands for Retrieval-Augmented Generation.
It is an AI approach that combines information retrieval
with language generation. The system retrieves relevant
information from a knowledge base and provides it as context
to a language model to generate an answer.

In this case, the system finds relevant information in the knowledge base and uses it to generate the answer.

Example Interaction: Question Outside the Knowledge Base

You: Who is the Prime Minister of India?

Searching knowledge base...

Agent: Sorry, I cannot answer this question because
the required information is not available in the
provided knowledge base.

One of the important features of this project is attempting to prevent the AI from answering questions that are outside the provided knowledge base.

The model is instructed to:

• Answer only from the retrieved context
• Avoid using unsupported general knowledge
• Avoid guessing or creating information
• Reject questions when sufficient information is not available

Challenges I Faced

During development, I initially experimented with a CodeAgent approach. However, the language model sometimes generated unsupported tool calls and did not consistently follow the code format required by the agent framework.

For example, instead of using the available knowledge base search tool, the model attempted to call a tool that did not exist.

I simplified the architecture by using a direct RAG pipeline:
User Query
↓
Retrieve Relevant Context
↓
FAISS + BM25 Hybrid Search
↓
Provide Context to Qwen
↓
Generate Final Answer

This approach was simpler and more reliable for my project. I also experimented with different Qwen model sizes. I moved to Qwen2.5-72B-Instruct to reduce response time and make the project more practical for local execution.

What I Learned

Through this project, I gained hands-on experience with:

• Retrieval-Augmented Generation
• Text chunking
• Text embeddings
• Vector databases and similarity search
• FAISS
• BM25
• Hybrid search
• Prompt engineering
• Knowledge-grounded AI systems
• Large Language Model integration
• Building an AI project in Python and VS Code

Future Improvements

Some features I would like to add in the future include:

• PDF and document upload support
• A web interface using Gradio or Streamlit
• Source citations with every answer
• Better relevance threshold checking
• Persistent vector storage
• Conversation history
• Support for multiple knowledge bases
• RAG evaluation metrics

This project was a great hands-on learning experience for understanding how modern AI assistants can be grounded in external knowledge.

I'm continuing to learn and build more AI Engineering projects. Feedback and suggestions are welcome!