Can Sinhala NLP Help Detect Scams and Misinformation?
Sri Lanka's official language, Sinhala, is spoken by approximately 17 million people. With the rise of digital communication, Sinhala-language scams and misinformation have increased, but few NLP tools exist for this language. This post documents a classification study to assess whether Sinhala NLP can help detect scams and misinformation.
Problem Definition
The goal is to build a binary classifier that labels Sinhala messages as either scam/ misinformation or legitimate. The scope includes financial scams (e.g., fake prize notifications), political misinformation, and health misinformation. The output is a confidence score and the predicted class.
Datasets
Two datasets were created: a scam message dataset collected from Sri Lankan Facebook groups (500 messages), and a misinformation dataset fact-checked by local news outlets (300 claims). Both datasets were translated and verified by native Sinhala speakers. Messages were labeled as scam, misinformation, or neutral.
Classification
Three models were evaluated: a fine-tuned BanglaBERT adapted for Sinhala, a traditional TF-IDF + Logistic Regression baseline, and a GPT-4o zero-shot prompt. The fine-tuned model achieved the best results: precision 82.4%, recall 76.1%, F1-score 79.2%. The TF-IDF baseline scored 68.5% F1, and GPT-4o scored 61.3% F1.
Labeling
Labeling was performed by 3 native Sinhala speakers with experience in fact-checking. Inter-annotator agreement (Cohen's kappa) was 0.71, indicating substantial agreement. Disputed labels were resolved through discussion and reference to fact-checking resources.
Evaluation
Cross-validation was performed with 5 folds. Results are averaged across folds. A separate hold-out test set (20% of data) was used for final evaluation. The fine-tuned model was also evaluated for false positive rate (12.3%) and false negative rate (18.7%).
Deployment Risks
- False positives: Legitimate messages incorrectly flagged as scams, potentially silencing legitimate public discourse.
- False negatives: Scam messages missed, leading to user harm.
- Privacy: Message content is processed by the model; anonymization is required before storage.
- Bias: Classifier may perform worse on dialect variations or code-mixed Sinhala-English messages.
Human Review
All flagged messages are reviewed by a human before any action is taken. The system includes a confidence threshold (0.75) below which messages are sent for manual review. A dashboard shows the reason for classification (e.g., specific keywords, sentiment patterns).
