rfc2822 April 2001

3.2. Lexical Tokens

   The following rules are used to define an underlying lexical
   analyzer, which feeds tokens to the higher-level parsers.  This
   section defines the tokens used in structured header field bodies.

   Note: Readers of this standard need to pay special attention to how
   these lexical tokens are used in both the lower-level and
   higher-level syntax later in the document.  Particularly, the white
   space tokens and the comment tokens defined in section 3.2.3 get used
   in the lower-level tokens defined here, and those lower-level tokens
   are in turn used as parts of the higher-level tokens defined later.
   Therefore, the white space and comments may be allowed in the
   higher-level tokens even though they may not explicitly appear in a
   particular definition.