Encodings

HTML

HTML character references. Every named one in the WHATWG table, decimal and hex ones, and the Windows-1252 quirks old pages still carry
IDhtml16 / 20html · 96 characters
Encoding / HTML

HTML character references

& and friends. How a page spells what it can't type.

Overhead
text only
Padding
none
Standard
WHATWG HTML, named and numeric character references
Used by
HTML, RSS, every web archive
family 1 / 20

Sample

Out
hello world!
Back
"hello world!"

Options

asciiboolean
default false
Also write every character past ASCII as a hex reference, such as é

Access

Importimport { html } from "@agntn/encodings/html"
CLIencodings encode html 'hello world!'
Tryplayground with the sample above

A page can't just write < in its text. The parser would take it for a tag. So HTML spells it &lt;, and the ampersand spells itself &amp;. That's a character reference.

ts
const read = (text: string) => new TextDecoder().decode(decode("html", text).bytes);

encode("html", 'Tom & "Jerry" <3');  // "Tom &amp; &quot;Jerry&quot; &lt;3"
read("Universit&auml;t");            // "Universität"

Encoding escapes the five characters markup cares about: &, <, >, " and '. Everything else stays put. Want plain ASCII out? ascii: true writes the rest as hex references too.

ts
encode("html", "Zażółć gęślą", { ascii: true });
// "Za&#x17C;&#xF3;&#x142;&#x107; g&#x119;&#x15B;l&#x105;"

Ugly? Sure. It survives any charset, though.

Reading it like a browser

Decoding is the fun part. A browser never throws on a bad reference. It shrugs and keeps going. decode does the same, rule for rule from the WHATWG tokenizer.

  • Every named reference in the spec's table, from &amp; to &CounterClockwiseContourIntegral;.
  • Decimal and hex ones, &#65; and &#x1F600;.
  • The old names that still work without a semicolon. &copy 2001 is © 2001.
  • Anything that names nothing stays text. AT&T comes back as AT&T.

And &notit;? That's ¬it;. not is one of the old names, notit isn't a name at all. Browsers really do this.

Why is &#146; an apostrophe?

Because Windows said so. Code point 146 is an invisible control character. But pages from the nineties were typed in Windows-1252, where byte 146 is ’. Their authors wrote what they saw.

ts
read("it&#146;s &#150; &#128;5");  // "it’s – €5"

So browsers read 128 to 159 through Windows-1252, and this does too. Zero, surrogates and anything past U+10FFFF turn into �.

Spotting it in the wild

identify wants at least one reference that ends in ;. R&D is valid HTML too. It just never said anything. Same for a URL with &copy=2 in its query.

ts
identify("Universit&auml;t, it&#146;s 1999")[0]?.text;  // "Universität, it’s 1999"

Base64 wrapped around a page? peel takes off both layers.

It's text in and text out, so there's no overhead on random bytes. Hand encode bytes that aren't UTF-8 and it refuses. A reference names a character, never a raw byte.