Big5 encoding mishandles some trailing bytes, with possible XSS

[mihnita]

There are some sequences of bytes that are valid lead-trailing according to the description at https://encoding.spec.whatwg.org/#big5-decoder, but don't have a corresponding Unicode codepoint in the index-big5.txt mapping table.

In this case the first byte is converted to U+FFFD, but the second one is left "as is". In some cases that second byte can be backslash (\, 5C), which can be used to "escape" the ending quote of strings in JavaScript and potentially resulting in XSS exploits.

Example (attached): the 83 5C sequence
According to the algorithm at https://encoding.spec.whatwg.org/#big5-decoder

• the first byte is a lead byte (case 5, byte is in the range 0x81 to 0xFE, inclusive)
• for the second byte we have case 3
  • 3.1. Let offset be 0x40 (byte is less than 0x7F)
  • 3.2. byte is in the range 0x40 to 0x7E, inclusive => set pointer to (lead − 0x81) × 157 + (byte − offset). The result of that is (0x83 - 0x81) * 157 + (0x5C - 0x40) which is 0x156
  • there is no mapping for 0x156 in index-big5.txt, and because the byte is an ASCII byte we prepend byte to stream (case 3.6) and Return error (case 3.7)

The end result is a U+FFFD (from the error) followed by a 5C (the trailing byte, "as is")

You can see this in the attached file.
When opened in both Chrome and Firefox the text is rendered as the "Unicode REPLACEMENT CHARACTER" (correct) followed by a back-slash (incorrect).

This is a valid lead-trail byte sequence that should either be replaced by one single U+FFFD character, or by two U+FFFD characters, whatever the policy is (I think the second case).

But the definitely the trailing byte should not be left "as is"

The possible exploit can use the trailing byte (which is backslash) to escape the end of a string, for example.
Checking the console of Firefox you will see the 'SyntaxError: "" string literal contains an unescaped line break' message. In Chrome the message is 'Uncaught SyntaxError: Invalid or unexpected token'

I did not check, but this might also happen in other DBCS (Double Byte Characters Sets) that have the second byte in the ASCII range (for instance in Shift JIS?).

big5.zip

[annevk]

Heya, welcome!

This is by design to avoid allowing attackers to use this to prevent ASCII delimiters from being seen.

I took some time to find the rationale to make sure we're all on the same page going forward:

• https://bugzilla.mozilla.org/show_bug.cgi?id=690225
• https://www.w3.org/Bugs/Public/show_bug.cgi?id=19961

[mihnita]

Thank you for the quick answer.

==== TLDR ====

82 22 which is an invalid Shift JIS byte sequence.
83 5C is a valid Big 5 sequence, with not Unicode mapping.

Because the 83 5C is a valid sequence but has no mapping, the whole sequence should be replaced with U+FFFD (one or two of them, TBD), not just the lead byte.

==== The long version ====

I have tried to carefully read the links provided, but I think this is kind of the opposite...
The bugs are about 82 22 which is indeed an invalid bytesequence for Shift_JIS.

That handling is correct, because it is a 'valid lead' followed by 'invalid trailing'

If byte is in the range 0x40 to 0x7E, inclusive, or 0x80 to 0xFC, inclusive, set pointer to (lead − lead offset) × 188 + byte − offset.
0x22 is not in the [0x40, 0x7E] range

This is different, as 83 5C is a valid Big 5 sequence, valid lead + valid trailing, but no Unicode mapping, at least not in the table included with the spec (index-big5.txt).

This can lead the an exploit similar to the one quoted in the bug (from 2011):

a shift_jis lead byte 0x82 was used to “mask” a 0x22 trail byte in a JSON resource
of which an attacker could control some field. The producer did not see the problem even though
this is an illegal byte combination.

In this case 83 5C can be used to mask a 0x22 (double quote) coming AFTER 5C

One might not see the problem, because 83 5C is a valid Big 5 byte sequence,
So 83 5C 22 maps (sometimes :-) to U+F00E" (PUA followed by a double quote).
And with some other mappings is converted to U+FFFD\" which escapes the quote.

To make things more interesting that sequence is actually mapped to a PUA character (U+F00E) in the Microsoft implementation (https://en.wikipedia.org/wiki/Code_page_950)
Not really relevant to the bug...

---

Considering that there are many Big 5 extensions (and tables), this is quite likely to happen.
(https://en.wikipedia.org/wiki/Big5). And most don't even have a IANA registration.

In fact, I discovered this exactly from a client side JSON parsing (using this algorithm), with data produced server side (using the Big 5-HKSCS table by default).

None of these extensions are registered with IANA, so there is no standard way to communicate that information to another client.

[annevk]

Thanks for that analysis, I guess that is indeed a novel angle that I'm not sure was fully considered. That would affect most legacy encodings to some extent, including Shift_JIS I suspect (we tend to unwind for ASCII bytes as a general principle).

Since most implementations are now aligned I wonder to what extent they want to change this again.

At the very least we should add a warning somewhere or maybe add a paragraph to the security considerations that suggests that if you're using a legacy encoding, you have to be sure that it's identical to those defined and that otherwise you need to account for the difference in behavior being exploited.

[mihnita]

Since most implementations are now aligned I wonder to what extent they want to change this again.
...
At the very least we should add a warning somewhere

If there is agreement that this is a possible security problem waiting to happen, then a fix is a lot better than a warning. Plus, it really "feels" incorrect: a valid lead-trailing byte sequence gets only half-converted to U+FFFD...

It is not an intrinsic problem with legacy encodings, it is a problem with the algorithm as described in this spec.

Let's say we add a warning.
What is an implementer supposed to do to mitigate the security problem?
They can either implement some fix, making them non-WATWG compliant, or they can ignore it.
None of the options sounds like a good one.

"you have to be sure that it's identical to those defined": there is no way to do that. You get some content from somewhere, you don't even control it, and it is tagged as Big 5. There is no way to tag it as some variant of Big 5, because none of them is IANA registered.
The only thing one can do is detect the error and convert the invalid sequences to FFFD, which is what this spec is supposed to detail.

---

Would it help if I get some "chime in" from someone in the Chrome team saying they would implement this if the spec changes?

Thank you very much,
Mihai

[annevk]

The warning would be for content producers, who also must use UTF-8 per the standard already, so they are already in violation of sorts.

If Chrome were willing to drive this work (including the change proposals and test changes required) that would certainly help, but we'd need one more implementer (Apple or Mozilla) to also be on board.

[jungshik]

I'd rather not tweak legacy encoding implementations any further in Chromium's copy of ICU unless it's absolutely necessary.

[hsivonen]

Thanks for that analysis, I guess that is indeed a novel angle that I'm not sure was fully considered.

While novel angle for the Encoding Standard, my understanding is that a similar structural concern kept Shift_JIS unsupported as a system encoding in Fedora so that the transition was direct from EUC-JP to UTF-8. (I don't know how DOS and pre-NT Windows dealt internally with 0x5C in two-byte characters in file paths under the Japanese locale. I also don't how if Fedora supported Big5 as a system encoding previously.)

In fact, I discovered this exactly from a client side JSON parsing (using this algorithm), with data produced server side (using the Big 5-HKSCS table by default).

Interesting both because JSON is not allowed to be Big5-encoded (irrelevant as far as providing security even for people who don't conform with specs goes) and because the spec is trying to be able to decode HKSCS.

While a backslash may have different implications than other ASCII-range bytes as the second byte of an unmappable sequence, I'm reluctant to make a special case for 0x5C in the the general ASCII unmasking policy with the level of evidence offered so far.

Before debating special-casing 0x5C as the trail byte when an index lookup fails, I'd be interested in learning what Big5-HKSCS generator can generate byte pairs that the index in the Encoding Standard does not have mappings for. We have mappings for the 0x5C trail byte for every lead byte from 0x87 onwards. We have no mappings for any byte pair, whose lead is in the range 0x81 to 0x86, inclusive. What software produced the 0x83, 0x5C byte sequence and what Big5 extension does it belong to? (CC @foolip)

you have to be sure that it's identical to those defined

That's relatively useless advice especially in the case of Big5, which, as defined in the spec, is a WHATWG synthesis that is unlikely to be an exact match for any legacy implementation.

[hsivonen]

The old Big5 study emails suggest have evidence of lead bytes in the 0x81 to 0x86 (inclusive) range in pages whose URLs seem no longer to work. How did those items of evidence not result in mappings for lead bytes in that range?

[mihnita]

The warning would be for content producers, who also must use UTF-8 per the standard already,
so they are already in violation of sorts.

The trouble here is that XSS attacks are not coming from "friendly" producers. If we say "don't do this, it is a possible vulnerability", we are just inviting bad actors to abuse it.

I'm reluctant to make a special case for 0x5C in the the general ASCII unmasking policy

I am not advocating special treatment for 5C.
I thing that a valid lead-byte - trailing byte sequence, recognized as such, should be treated as one "unit". And either converted to a Unicode character (if there is a mapping), or convert to FFFD if there is no conversion.
But not convert half (to FFFD) and keep the other half.

I'd be interested in learning what Big5-HKSCS generator can generate byte pairs that the index in
the Encoding Standard does not have mappings for

Anything Windows Code page 950. So probably most (all?) Windows APIs.
I can run some tests, if you want.

And also ICU:

  @Test
  public void testBig5() {
    byte [] bytes = { (byte) 0x83, (byte) 0x5C };
    String charsetName = "Big5";

    java.nio.charset.Charset cs = java.nio.charset.Charset.forName(charsetName);
    System.out.println(cs + " : " + cs.aliases());
    System.out.println(hex(new String(bytes, cs)));

    cs = java.nio.charset.Charset.forName("cp950");
    System.out.println(cs + " : " + cs.aliases());
    System.out.println(hex(new String(bytes, cs)));

    cs = com.ibm.icu.charset.CharsetICU.forNameICU(charsetName);
    System.out.println(cs + " : " + cs.aliases());
    System.out.println(hex(new String(bytes, cs)));
  }

The code above produces:

Big5 : [csBig5]
 FFFD 005C
x-IBM950 : [cp950, ibm950, 950, ibm-950]
 F00E
Big5 : [windows-950, csBig5, x-big5, Big5, x-windows-950, windows-950-2000, ms950]
 F00E

(the hex method does a simple char by char hex dump, I did not include it here)

We notice that Java considers Big5 and cp950 to be different charsets, but ICU4J considers them aliases.

---

I totally understand the reluctance to change a spec, and to change implementations.
We would all like to produce code / specs that are perfect, bug free, and never need updating.
But this is the reality of things: we find problems, we should fix them.

I would really hate to see some exploit based on something this a few months down the line...
Double byte character sets (DBCS) have been a source of bugs and exploits for a long time.

[mihnita]

If it helps you can also diff the mapping tables from the Unicode site:
http://www.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/OTHER/BIG5.TXT
http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP950.TXT

[hsivonen]

The code above produces:

Interesting. So it maps to the PUA. Maybe we should map byte pairs with leads in the 0x81 to 0x86 (inclusive) range to the PUA. It seems worthwhile to check what Microsoft does for codepage 950.

If it helps you can also diff the mapping tables from the Unicode site:
http://www.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/OTHER/BIG5.TXT
http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP950.TXT

These tables have mappings only starting from lead byte 0xA1 upwards.

[hsivonen]

https://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WindowsBestFit/bestfit950.txt says "EUDC" for those lead bytes (but also for 0x87 to 0xA0, inclusive).

The Unicode.org vendor repo does not appear to have a Microsoft submission for the Hong Kong "951" shadow codepage.

[foolip]

Before debating special-casing 0x5C as the trail byte when an index lookup fails, I'd be interested in learning what Big5-HKSCS generator can generate byte pairs that the index in the Encoding Standard does not have mappings for. We have mappings for the 0x5C trail byte for every lead byte from 0x87 onwards. We have no mappings for any byte pair, whose lead is in the range 0x81 to 0x86, inclusive. What software produced the 0x83, 0x5C byte sequence and what Big5 extension does it belong to? (CC @foolip)

I'm afraid that I don't know much about the software which produced the content that @annevk, I and others analyzed back in the day. I feel like the best way to answer questions about what the spec should say now would be to perform a new scrape looking for certain patterns, perhaps starting from the list of URLs in httparchive.

I'd rather not tweak legacy encoding implementations any further in Chromium's copy of ICU unless it's absolutely necessary.

@jungshik are we now in alignment with the Encodings spec for Big5? If not, do you know where the differences are?