Overview
Before a model reads a sentence, a tokenizer divides it into pieces. The same Korean phrase can require very different numbers of tokens depending on that design. The EXAONE 3.0 technical report traces that choice alongside model training and instruction-following evaluation.
Key Findings
A model reads text through a vocabulary it learns before it answers a single question. If that vocabulary cuts Korean into many short fragments, a paragraph can take up more of the context window than a comparable paragraph in English. The EXAONE report starts with that design problem and follows it through the open release of a 7.8-billion-parameter model.
Tokenization is part of the bilingual design
Korean words can combine several meaningful parts. The team uses MeCab to pre-tokenize Korean data and trains a byte-level byte-pair encoding tokenizer with a vocabulary of 102,400. The report compares how many tokens per word different tokenizers produce on sampled English and Korean corpora.
| Tokenizer, Table 2 | English tokens per word | Korean tokens per word |
|---|---|---|
| EXAONE 3.0 | 1.44 | 2.46 |
| Llama 3.1 | 1.35 | 3.01 |
| Gemma 2 | 1.39 | 3.31 |
| Qwen 2 | 1.39 | 3.29 |
| Mistral | 1.55 | 5.22 |
The Korean sample uses fewer tokens per word under EXAONE’s tokenizer than under the listed alternatives. That is evidence about compression on these corpora. The full system also includes pretraining and multi-turn instruction tuning. Table 10 reports EXAONE 3.0 7.8B at 63.7 on ARC-C and 10.1 on normalized GPQA, ranking third in the compared group on both. The tokenization table alone cannot identify a cause for those downstream results.
Table 2 compares the number of tokens per word on sampled corpora. The two languages reveal different trade-offs.
| Tokenizer | English | Korean |
|---|---|---|
| EXAONE 3.0 | 1.44 | 2.46 |
| Llama 3.1 | 1.35 | 3.01 |
| Gemma 2 | 1.39 | 3.31 |
| QWEN 2 | 1.39 | 3.29 |
| Phi 3 | 1.35 | 4.69 |
| Mistral | 1.55 | 5.22 |
From predicting text to carrying a conversation
The report separately describes training on a text corpus and post-training with instruction data. Multi-turn examples teach the model to respond in context instead of treating every request as a fresh input. The released instruction model reflects those choices together. The tokenization comparison tells us how efficiently the vocabulary represents text. A question-answering table tests an entirely different outcome.
A bilingual release needs more than one scoreboard
The evaluations cover English and Korean, with settings for knowledge, reasoning, instruction following, and use of the context. Reading the compression table next to the downstream results avoids attributing every difference to the tokenizer. A favorable result in one task can coexist with a weaker result in another, as the ARC-C and normalized GPQA comparisons illustrate. The technical report gives users a reproducible starting point for evaluating the released model on their own language mix.
Questions & Answers
The opening answers are simulated author responses based on the paper.
Q: What does tokens per word measure?
A: It counts how many model tokens are used to represent words in the sampled corpus. A smaller count can let a fixed context window hold more text.
Q: Can fewer tokens establish better bilingual answers?
A: No quality score follows from compression alone. Compare answers on the same content and test whether important names, units, and technical terms survive the full pipeline.
Q: Can the report tell us how much MeCab contributes to question answering?
A: The compression comparison describes complete tokenizers. The downstream comparison describes complete models, whose training and other design choices differ. To attribute an answer-quality change to Korean pre-tokenization, compare models trained under the same data, architecture, and compute setup with and without that component.
Future Ideas
A bilingual context-budget test
Build matched English and Korean versions of technical support documents. Compare which pieces of evidence fit under a fixed token budget and verify the resulting answers with bilingual reviewers. Track latency, truncation, correct citations, and failures on mixed-language product names. Separate tokenizer-only measurements from full-model comparisons.
Sources
Original abstract
We introduce EXAONE 3.0 instruction-tuned language model, the first open model in the family of Large Language Models (LLMs) developed by LG AI Research. Among different model sizes, we publicly release the 7.8B instruction-tuned model to promote open research and innovations. Through extensive evaluations across a wide range of public and in-house benchmarks, EXAONE 3.0 demonstrates highly competitive real-world performance with instruction-following capability against other state-of-the-art open models of similar size. Our comparative analysis shows that EXAONE 3.0 excels particularly in Korean, while achieving compelling performance across general tasks and complex reasoning. With its strong real-world effectiveness and bilingual proficiency, we hope that EXAONE keeps contributing to advancements in Expert AI. Our EXAONE 3.0 instruction-tuned model is available at https://huggingface.co/LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct.