Mongolian speech-recognition research
A benchmark that ranks public Mongolian speech-recognition models honestly, a Whisper fine-tuning pipeline and a consent-first voice-collection portal. Still research.
- public models benchmarked
- 5
- spoken numbers transcribed correctly by public models
- 0 of 24
- lowest off-the-shelf word error rate (WER)
- 18.9%
Problem
Teachers wanted to dictate in Mongolian, but public speech models are ranked on leaderboard scores that may not hold up in real use.
Solution
We built a benchmark that ranks public Mongolian (Khalkha, Cyrillic) speech models on accuracy, speed and memory. A Whisper fine-tuning pipeline sizes each run to the available GPU and checks that no sentence appears in both the training and the test data.
A consent-first recording portal receives uploads from a teacher app and pseudonymises the speakers.
Outcome
The benchmark checks whether test data leaked into training, so inflated scores get caught. No public model transcribed even one of 24 spoken numbers correctly, and the best honest off-the-shelf result was an 18.9% word error rate.
These findings shaped a recording plan full of numbers. This is still research; there is no production model yet.