Character-Set Blind Spots

In 2017 a researcher registered a domain that displayed, in most browsers of the day, as apple.com. It was not apple.com. The letters were Cyrillic, code points that render identical to their Latin twins and resolve to a completely different address. The demonstration was harmless and the point was not: to a human the string was one thing, to the machine it was another, and the attacker chose which gap to stand in. The industry calls it the homograph attack, and it is a special case of a much older lesson that safety classifiers are now learning again from scratch.

A classifier is, underneath the sophistication, a system that reads text and judges it. Which quietly assumes something untrue, that there is one canonical version of the text to read. There is not. The same meaning can be carried by an enormous number of different byte sequences, and the attacker, not the defender, picks which one arrives.

That is the whole seam. Human meaning is stable across encodings; the eye reads through them without noticing. Machine classification is not stable across encodings, because it works on bytes and tokens rather than on the meaning a person reconstructs. Slide the same intent from one script to another, mix character sets mid-word, reach for a homoglyph that looks identical and tokenizes differently, and you have changed nothing a human registers and everything a pattern matcher depends on.

Correct the obvious mistake up front, because treating it as a single-language problem is exactly the underweighting that lets it work. This is not about Chinese. Arabic script, Cyrillic, mixed-Unicode composition, invisible joiners, zero-width characters, decomposed versus precomposed forms, they all open the same class of gap. A Latin a and a Cyrillic а are one pixel apart and two unrelated code points. A word split across two scripts is one word to a reader and two fragments to a tokenizer. The evasion is not a language. It is a property of the space between how text is rendered and how it is processed, and that space exists in every script.

Why classifiers underweight it comes down to training and runtime. Detection leans on patterns learned from a corpus, and the corpus skews toward the well-formed, single-script, canonically encoded text that makes up most writing. The adversarial encodings are, by construction, the long tail the training data barely held. And at inference the classifier works a sliding window over tokens, so a payload fragmented across scripts, or padded with characters that tokenize into noise, falls into the blind spots between what any single window sees as coherent. The meaning reassembles in the reader's head. It never reassembles in the classifier's window.

The defense is old, boring, and correct, and it has a name a generation of web developers can recite in their sleep: canonicalize before you validate. Path traversal, SQL injection, cross-site scripting, half the vulnerability classes of the 2000s came down to a system judging raw input the attacker had encoded around, and the fix was always the same. Normalize first, decide second. A detection layer must not judge whatever raw byte sequence happened to arrive. It has to collapse the text to a canonical form first, Unicode normalization, homoglyph and confusable folding, zero-width and formatting stripping, and then read the result. You defend the meaning, not the encoding. If two strings mean the same thing to a human, they must look the same to your classifier before it decides, or the attacker keeps handing you the version you have not folded yet.

The corollary is that heavy encoding is itself a signal, not just a decoding chore. Legitimate text is overwhelmingly consistent in its character sets. A message that stitches three scripts together mid-token, or leans on confusables and invisible characters, is doing something an ordinary sentence never needs to do. That anomaly deserves weight. Not an automatic block, but a reason to look harder and trust less, because effort spent encoding around detection is effort spent for a reason.

The attacker's advantage here is pure asymmetry. They choose the encoding, you inherit it, and there are far more ways to write a thing than there are ways you have remembered to normalize. Close the gap the only way it closes. Read the meaning, not the bytes, and the clever encoding collapses back into the plain sentence it was always hiding.

GhostInThePrompt.com // The attacker picks the encoding, so read the meaning. Canonicalize first or judge the string they chose.