UTF(6)                                                     UTF(6)

     NAME
          UTF, Unicode, ASCII, rune - character set and format

     DESCRIPTION
          The Plan 9 character set and representation are based on the
          Unicode Standard and on the ISO multibyte UTF-8 encoding
          (Universal Character Set Transformation Format, 8 bits
          wide).  The Unicode Standard represents its characters in 21
          bits; UTF-8 represents such values in an 8-bit byte stream.
          Throughout this manual, UTF-8 is shortened to UTF.

          In Plan 9, a rune is a 32-bit quantity representing a
          Unicode character.  Internally, programs may store charac-
          ters as runes.  However, any external manifestation of tex-
          tual information, in files or at the interface between pro-
          grams, uses a machine-independent, byte-stream encoding
          called UTF.

          UTF is designed so the 7-bit ASCII set (values hexadecimal
          00 to 7F), appear only as themselves in the encoding.  Runes
          with values above 7F appear as sequences of two or more
          bytes with values only from 80 to FF.

          The UTF encoding of the Unicode Standard is backward compa-
          tible with ASCII: programs presented only with ASCII work on
          Plan 9 even if not written to deal with UTF, as do programs
          that deal with uninterpreted byte streams.  However, pro-
          grams that perform semantic processing on ASCII graphic
          characters must convert from UTF to runes in order to work
          properly with non-ASCII input.  See rune(2).

          Letting numbers be binary, a rune x is converted to a multi-
          byte UTF sequence as follows:

          001.   x in [00000000.00000000.0bbbbbbb] → 0bbbbbbb
          010.   x in [00000000.00000bbb.bbbbbbbb] → 110bbbbb,
          10bbbbbb
          011.   x in [00000000.bbbbbbbb.bbbbbbbb] → 1110bbbb,
          10bbbbbb, 10bbbbbb
          100.   x in [000bbbbb.bbbbbbbb.bbbbbbbb] → 11110bbb,
          10bbbbbb, 10bbbbbb, 10bbbbbb

          Conversion 001 provides a one-byte sequence that spans the
          ASCII character set in a compatible way.  Conversions 010,
          011 and 100 represent higher-valued characters as sequences
          of two, three or four bytes with the high bit set.  Plan 9
          does not support the 5 and 6 byte sequences proposed by X-
          Open.  When there are multiple ways to encode a value, for
          example rune 0, the shortest encoding is used.

          In the inverse mapping, any sequence except those described
          above is incorrect and is converted to rune hexadecimal
          FFFD.

     FILES
          /lib/ucd/UnicodeData.txt   table of characters and descrip-
                                     tions, suitable for look(1).

     SEE ALSO
          ascii(1), tcs(1), rune(2), keyboard(6), The Unicode
          Standard.