Amharic Normalization & Canonical Representation
Deterministic orthographic normalization framework for Amharic, featuring a reversible canonical representation (CAR), ambiguity-aware Latin decoding, and bidirectional punctuation standardization.
Problem
Amharic digital text suffers from inconsistent transliteration, mixed-script input, punctuation variation, and ambiguous Latin decoding. Existing tools silently guess interpretations, leading to incorrect or unstable downstream processing.
Approach
This note describes a layered architecture for Amharic text processing. At its core is CAR v0, a strictly bijective canonical encoding of Ethiopic characters. Every character maps to exactly one canonical token, preserving reversibility and structural stability.
On top of CAR sits AN-v0, a normalization layer for Ethiopic Unicode, Latin transliteration, and mixed-script input. Ambiguous Latin decoding is surfaced explicitly through alternatives and confidence scoring rather than silently collapsed.
A bidirectional punctuation layer standardizes Ethiopic marks and implements a context-aware period rule for abbreviations and decimals.
Impact
Establishes a canonical encoding layer for Amharic that improves dataset consistency, enables reversible processing, and provides a stable preprocessing foundation for NLP models, localization systems, and language-learning tools.
Tools and Methods
- Python
- FastAPI
- React / Vite
- Canonical encoding design
- Bidirectional transliteration
- Linguistic normalization