Definition
Unicode gives a number to every character in every writing system. An encoding, usually UTF-8, is the rule for storing those numbers as bytes. In Python 3, str holds text and bytes holds raw data, and you convert with encode() and decode().
In simple words
Text on a screen and text on a disk are different. Saving turns letters into bytes using a rulebook. If you read with the wrong rulebook, you get garbage like न. Always say encoding="utf-8".
Why it matters
Hindi and other Indian-language text breaks easily without this. In NLP you must know that len("नमस्ते") counts code points, not what you see, and an LLM's tokens are different again.
How it works
| Type | Holds | Example |
|---|---|---|
| str | Text | 'नमस्ते' |
| bytes | Raw bytes | b'\xe0\xa4\xa8...' |
In code
word = "नमस्ते"
print("Characters:", len(word))
data = word.encode("utf-8")
print("Bytes:", len(data))
print("Back to text:", data.decode("utf-8"))
print("Code of अ:", ord("अ"))
print("Character 2309:", chr(2309))
print("English bytes:", len("hello".encode("utf-8")))Output
Characters: 6 Bytes: 18 Back to text: नमस्ते Code of अ: 2309 Character 2309: अ English bytes: 5
Common mistakes
- Opening files without encoding="utf-8". On Windows the default may break Hindi text.
- Assuming one character is one byte. Hindi characters take 3 bytes each in UTF-8.
Check yourself
Summary
- Unicode numbers the characters, and UTF-8 stores them as bytes.
- str is text, and bytes is raw data.
- Always give encoding="utf-8" when opening files.
Finished reading?
Sources: The Python Tutorial, python.org