Type a sentence below and watch it break into pieces.

Working memory (8k tokens)5% full
hidden instructionsyouthe AIa file
  1. The app's hidden instructions~400 tok
  2. "Build me a login page"~30 tok

Everything still fits. Keep going and watch what happens when it doesn't.

Token counts are estimates (~4 characters per token); real tokenizers differ by model.

The model never saw your words. It saw those pieces. They're called tokens.

Anthropic's glossary: "Tokens are the smallest individual units of a language model, and can correspond to words, subwords, characters, or even bytes (in the case of Unicode)."

One sentence, cut into tokens (illustrative)

"The"

Token?
Often one token

" signup"

Token?
Maybe split: " sign" + "up"

" button"

Token?
Often one token, space included

"!"

Token?
Its own token

That table is illustrative. Every model family has its own way of cutting text, so the same sentence can be a different number of tokens in different models.

Roughly how big? Anthropic says that "for Claude, a token approximately represents 3.5 English characters," and that the exact number varies with the language.

Why you should care: limits and prices are counted in tokens. Anthropic's token-counting docs tell developers to "use token counts to manage rate limits and costs."

Try it with the rules off. In the sandbox, paste the same sentence in another language, then add emoji. Watch the token count move even though the meaning didn't.

Check yourself

0/3 got it

Saved on this device only. No account, no streaks.

Next in how the AI thinks: What is a context window?.