Token

The basic unit of text a language model processes — often a word piece or subword.

Back to Fundamentals of Generative AI