Tokenisation in Natural Language Processing
How text is segmented into machine-readable units for NLP pipelines and large language models.
Tokenisation is a core preprocessing step that converts raw text into model-readable units. It underpins modern natural language processing, neural translation, and large language models by defining how text is segmented before inference or training.
What Is Tokenisation
Tokenisation is the process of splitting text into smaller units called tokens. Depending on the algorithm, a token can be a full word, a subword fragment, punctuation, or even a byte sequence.
Why Tokenisation Is Important for AI Models
AI models do not read characters as humans do. They map token IDs to embeddings and process these vectors through neural layers. Better tokenisation improves representational efficiency, lowers unknown-token risk, and stabilises quality in multilingual and domain-specific tasks.
Types of Tokenisation (word, subword, byte-level)
- Word tokenisation: splits by word boundaries; simple but brittle for rare words.
- Subword tokenisation: decomposes words into reusable pieces and handles OOV terms better.
- Byte-level tokenisation: operates over bytes for robust multilingual coverage.
Byte Pair Encoding and SentencePiece
Byte Pair Encoding (BPE) iteratively merges frequent symbol pairs into compact subword vocabularies. SentencePiece generalises this idea with language-agnostic segmentation and unigram-style alternatives, widely used in contemporary NLP pipelines.
Tokenisation in Large Language Models
In LLM systems, tokenisation controls prompt packing, cost, context utilisation, and output fluency. It directly affects how models interpret prompts, preserve terminology, and perform tasks such as neural machine translation.
Token Limits and Their Impact on AI Systems
Every model has a finite context window. Input and output tokens share this budget, so long documents may require chunking or retrieval strategies to avoid truncation and quality loss.
Tokenisation Challenges Across Languages
Languages with rich morphology, script variation, or missing whitespace segmentation can be difficult to tokenise. Cross-language fairness depends on balanced vocabularies and multilingual evaluation, not only tokenizer speed.
Related Glossary Terms
Related Terms
Terminology Extraction
Automated or manual identification of key terms for project use.
Terminology Management
Organising, updating, and maintaining termbases for consistency.
TER (Translation Edit Rate)
A metric showing how many edits a human would need to correct a translation.
Token
A unit of text processed by a model, such as a word, subword, or punctuation mark.
Training Large Language Models
How LLMs are pretrained, optimised, and adapted using large datasets and distributed compute.
Related Resources
Speech Recognition
Technology that converts spoken language into written text using machine learning models.
Large Language Model (LLM)
A neural model trained on vast text corpora, capable of understanding and generating natural language.
Named Entity Recognition (NER)
A natural language processing method used to identify and classify entities such as names, organisations, locations, and dates in text.
Natural Language Processing (NLP)
A field of artificial intelligence focused on enabling computers to understand, analyse, and generate human language.
Semantic Similarity
A measure used in natural language processing to determine how similar two texts are in meaning.
Training Large Language Models
How LLMs are pretrained, optimised, and adapted using large datasets and distributed compute.