AI: Tokenisation

There is an interesting parallel between AI and written scripts.

Logographic scripts, like Chinese or Japanese Kanji, there is a separate 'glyph' to represent each word. If you can read the script, you can understand the meaning even if you don't know how to pronounce the word.

Phonetic scripts, like most Indic scripts, assign a glyph for every sound that you can make in the language. If you can read the script, you can pronounce every word, even if you never understand it's meaning.

The next article on AI has dropped and we learn how the above is related to AI 👇🏾

AI: Tokenisation

Recent Articles