Guide

Identify

Rank the encodings a string decodes in. Each score comes with the reasons behind it

Somebody sends you JBSWY3DPEBLW64TMMQ======. What is it?

ts
import { identify } from "@agntn/encodings";

identify("JBSWY3DPEBLW64TMMQ======");
// [
//   { encoding: "base32", confidence: 0.593, text: "Hello World",
//     reasons: ["padding fits the block length", "decodes to readable text"], … },
//   { encoding: "base32", options: { alphabet: "crockford" }, confidence: 0.043, … },
//   …
// ]

identify decodes the text in every registered encoding and keeps the ones that work. Then it ranks them by evidence, and the evidence is written down in reasons:

  • A matching checksum. Base58 with check and bech32 can't match by accident, so this outweighs everything else.
  • Framing only one encoding writes. Padding that fits, <~ and ~>, a begin 644 line, =XX escapes, &amp; references, a 0x prefix.
  • Decoded bytes that read as text. Valid UTF-8, mostly printable.
  • A small alphabet. If the text fits in sixteen characters, hex is likelier than base64.

It leaves out anything that decodes to nothing, and anything that decodes to the input itself. Plain ASCII is technically valid Quoted-Printable. That's not an answer. R&D is valid HTML, too, so HTML needs a reference that ends in ; first.

Every run of digits is a number, too. integer only counts when it decodes to at least eight bytes of readable text.

Any five bits are a teleprinter character. ita2 counts only when its codes print readable text. It never tries numbers: true, since digits tell you nothing about five-bit codes.

identify never tries codepoints: true on decimal or hex. Nearly every code point is a letter, digit or symbol. Hex 3432 means 42, but decimal code point 3432 is ൨, a Malayalam digit. Hex code point 31a5 is ㆥ, a Bopomofo letter. Any list of numbers could read as text. That proves nothing.

identify never tries prefix. Without a codebook, there's nothing to read. Unary would happily count any bits, which makes it a lousy detective.

The identify function tries square and can read chess square names and pairs. It never tries layout: with a8 numbering, a1 is byte 56, which happens to print as 8. That's a coincidence, not a clue.

A FEN gets the same treatment. identify reads it as fen, castling and clocks in details. It never tries another layout. Every layout holds the same board, so you'd get one position three times.

The number is a ranking

confidence runs from 0 to 1 and it sorts candidates. It isn't a probability. A base64 string of random bytes scores low, because there's no text and no checksum to vouch for it, and it's still the right answer. Read the reasons. Then decide.

Two readings of one text

Some encodings read the same text in several alphabets, or with and without a checksum. identify tries every one, and a candidate that needed an option says so in options:

ts
identify("CPNMUOG=").slice(0, 2);
// [{ encoding: "base32", options: { alphabet: "hex" }, text: "foob", confidence: 0.593, ... },
//  { encoding: "base32", confidence: 0.243, ... }]

Hand options to decode and you get the same bytes. A reading that gives the same bytes as the default one adds nothing, so it's left out. aGVsbG8gd29ybGQh reads the same in both base64 alphabets, and base64 shows up once. The base58 alphabets don't agree on anything, so all three show up, and only one says hello world.

When two candidates score the same, the registry order breaks the tie. Ties are honest, not bugs.

Long text

Base58 decoding is quadratic in the length. Real base58 strings are short, addresses and keys and CIDs, so identify skips the base58 family above 1024 characters. The tools cap base58 at 10000 characters either way.

Encoded twice? Peel it

Puzzles rarely stop at one layer. Base64, then hex on top, then binary for fun. peel takes them off one at a time.

ts
import { peel } from "@agntn/encodings";

peel("5958523059574e7249474630494752686432343d");
// [
//   { encoding: "hex", confirmed: true, text: "YXR0YWNrIGF0IGRhd24=", … },
//   { encoding: "base64", confirmed: true, text: "attack at dawn", … },
// ]

Every layer is the best candidate something backs. A checksum, framing, or at least eight bytes of readable text. Why eight? Plain words decode to four readable bytes by accident far too often. Peeling goes on while the layer is text.

And when the bottom isn't text? Base64 over encrypted bytes has nothing to back it. Then the best guess comes last, marked confirmed: false.

ts
peel("U2FsdGVkX1+xlr6h");
// [{ encoding: "base64", confirmed: false, … }]  // "Salted__" and four more bytes

The guess only shows up when the text still looks encoded. It needs a digit, and no spaces unless the guess is hex or another small alphabet. Prose gets no guess. peel("Hello world") is empty, and that's the right answer.

limit caps the layers, ten by default. encodings works like below. The CLI has identify --peel, the tool takes peel: true.

Only some encodings

ts
identify(text, { encodings: ["base64", "base32"], limit: 1 });

Your own registered encodings join the line-up without extra wiring.