Evaluating Open-Weight Models on Sinhala Multiple-Choice Questions
Evaluating large language models on low-resource languages like Sinhala is critical for expanding AI accessibility. This post documents a systematic evaluation of open-weight models on Sinhala multiple-choice questions, covering dataset construction, model selection, baseline results, and key limitations.
Research Question & Dataset
The goal was to assess how well open-weight models perform on Sinhala-language multiple-choice questions drawn from the MMLU (Massive Multitask Language Understanding) framework. A Sinhala-specific subset was created by translating English MMLU categories or sourcing native Sinhala questions from educational repositories. The dataset covers categories such as STEM, humanities, and social sciences, with difficulty levels ranging from basic to advanced.
Model Selection
Three open-weight models were evaluated: Llama 3.1 8B, Mistral 7B v0.3, and Nemotron 2 8B. All models were run in 4-bit quantized mode using GGUF format to fit within the evaluation hardware constraints. Prompt engineering was standardized across models to ensure a fair comparison.
Baseline & Evaluation
Results show that Llama 3.1 8B achieved the highest accuracy at 52.3%, followed by Mistral 7B at 48.1%, and Nemotron 2 at 45.7%. The baseline was established by majority voting across three model runs. Inference constraints included token length limits (max 2048 tokens) and the need for proper Sinhala Unicode tokenization, which varied across tokenizers.
Limitations
- Limited training data for Sinhala domains compared to English.
- Tokenization issues: some Sinhala compound words are split across multiple tokens, affecting model attention.
- Benchmark contamination: some questions may have appeared in the models' training data.
Next Steps
- Fine-tune models on a Sinhala-specific instruction dataset.
- Evaluate using few-shot prompting with Sinhala examples.
- Build a Sinhala RAG pipeline for open-domain question answering.
