Building a RAG-Based Academic Assistant
A RAG (Retrieval-Augmented Generation) academic assistant helps students and researchers find information from course materials, papers, and textbooks. This post documents the end-to-end architecture, from document ingestion to answer generation, along with evaluation findings.
Architecture
The system follows a classic RAG pipeline: document ingestion → chunking → embedding → storage → retrieval → generation. Weaviate serves as the vector database, Ollama provides local embeddings and LLM inference, and a Flask API orchestrates the flow.
Ingestion & Chunking
PDF academic papers are processed using PyMuPDF. Text is extracted and split into chunks of 512 tokens with 50-token overlap using a RecursiveCharacterTextSplitter. Overlap ensures that cross-chunk references (e.g., "as discussed in Section 3") remain coherent.
Embeddings & Storage
Ollama's nomic-embed-text model generates 768-dimensional vectors for each chunk. Vectors are stored in Weaviate with the class name AcademicPaper. Each object includes metadata: paper title, author, publication date, and section heading.
Retrieval
User queries are embedded using the same Ollama model, and the nearest-neighbor search is performed in Weaviate using vector similarity (default: cosine). The top-k results (k=5) are passed to the LLM as context. A filter excludes chunks with low similarity scores (< 0.65).
Generation
Ollama's llama3.2 model generates answers conditioned on the retrieved context. A prompt template structures the flow:
Context: {retrieved_chunks} Question: {user_query} Answer: Provide a concise, accurate answer based solely on the context above.Evaluation
Sample questions and answers were tested against a set of 10 biology and history papers. Retrieval precision@5 was 72% for biology papers and 61% for history papers. Answer-quality evaluation used a simple correctness scale: 1 = incorrect, 2 = partially correct, 3 = fully correct. Average score: 2.4/3.0.
Retrieval Failures
- Technical terms split across chunk boundaries resulted in missed matches.
- Synonym usage in queries (e.g., "heart attack" vs "cardiac event") failed to match stored text.
- Tables and figures in PDFs were not extracted, losing structured data.
Answer-Quality Evaluation
Human review of 50 generated answers showed that 62% were factually consistent with the source, 28% contained hallucinated details, and 10% were irrelevant. A more rigorous evaluation would use an LLM-as-a-judge metric with a curated rubric.
