In the context of Large Language Models, a token is the smallest unit of data that a model can understand and process. It can be as short as a single character or as long as a word, depending on the language and the specific model. Tokens are used to break down input data into manageable pieces, enabling the AI to analyze, understand, and generate text.
For most state-of the art models, one token can be statistically approximated as ~4 characters.
Tokens are also the accounting unit behind token economics: model cost, latency, context-window pressure, cache-hit rates, and usage quotas all depend on how many input and output tokens an application consumes. Informal practices such as tokenmaxxing try to increase productive AI use, but production teams should measure task outcomes rather than token volume alone.
The LLM Knowledge Base is a collection of bite-sized explanations for commonly used terms and abbreviations related to Large Language Models and Generative AI.
It's an educational resource that helps you stay up-to-date with the latest developments in AI research and its applications.