Tokens are word-like units recognized by a language. The compiler recognizes six classes of tokens. 

Here is the formal definition of a token:

• keyword
• identifier
• constant
• string-literal
• operator
• punctuator (also known as separators)

As the source code is scanned, tokens are extracted in such a way that the longest possible token from the character sequence is selected. For example, external would be parsed as a single identifier, rather than as the keyword extern followed by the identifier al. 

See Token Pasting with ## for a description of token pasting.