HTML escaping in JSON+LD

[tisdall]

I came across some JSON+LD on a site that contained a & and I assumed that I accidentally escaped something somewhere. However, I found that that was what was actually in the content and also that the standard says that it should be there.

For my application, I would like that & to be a &, but I was wondering if extruct should be doing this already?

[Gallaecio]

I’m not sure what the expected behavior is.

The standard that you link says:

• That only <script>, </script>, <!-- and --> need escaping.
• That the escaping will remain after processing.

To me it sounds like extruct is currently acting according to the standard.

I am unsure, though. Also, even in that case, I wonder if such decoding should be an optional feature at least.

[tisdall]

The "Note" seems to imply encoding the "usual suspects" of HTML entities:

The reasoning behind it is specifically to avoid having <script> and/or <!-- in the middle of the script block without escaping. However, it seems they decided to cover all the characters commonly escaped when within an HTML attribute.

The standard is about rendering the JSON+LD into an HTML document, but extruct is for extracting the data back into python data structures so it should be doing the opposite. Right now extruct is not unescaping back into regular text.

Here's an example:

from extruct.jsonld import JsonLdExtractor
from extruct.xmldom import XmlDomHTMLParser
from lxml.html import fromstring
domparser = XmlDomHTMLParser(encoding='UTF8')
content = """
<script type="application/ld+json">
{
  "@context": "http://schema.org/",
  "@type": "Thing",
  "name": "&lt;script&gt; &lt;!-- &amp;&quot;&apos;"
}
</script>
"""
tree = fromstring(content, parser=domparser)
jsonld = JsonLdExtractor().extract_items(tree)
print(jsonld[0]['name'])  # expected output *should* be  `<script> <!-- &"'`

[Gallaecio]

I wonder what it means for us that it says there “the content will remain escaped after processing through the JSON-LD API”.

[tisdall]

Excellent point. Yeah, I was wondering about that too...

Here's some documentation on the JSON-LD API. The gist seems to be an API that "restructuring data according to the defined transformations" using methods compact, expand, andflatten. The end result is still JSON-LD, but reformatted structurally. extruct's JsonLdExtractor is not implementing that API.

I found an example of expand (example 2 going to example 3). That API looks like it'd be really useful to have to transform JSON-LD into a more standardized format before processing, but that's a separate issue. expand would make the output closer (the same, even?) to the output from RDFaExtractor.

I guess that note is there because after processing JSON-LD into other JSON-LD, the end result may or may not be in an HTML context. I don't think it applies to the output of extruct.

[gk2go]

Is this the same issue?

view-source:https://www.aljazeera.com/news/2019/07/argentina-designates-hezbollah-terrorist-organisation-190719120713627.html

    jslde = JsonLdExtractor()
    schema = jslde.extract(article.html)
Expecting ',' delimiter: line 7 column 54 (char 281)

[tisdall]

@gk2go - The issue is about unescaping escaped HTML entities. The issue you're having is about malformed JSON because the person didn't quote the " as either \" or ".

"headline": "Argentina designates Hezbollah "terrorist" organisation",

Please use https://search.google.com/structured-data/testing-tool/ to test structured data first before suspecting a bug with extruct.