<li> tags without surrounding <ul> or <ol>

[Rodeoclash]

I'm trying to automatically clean up some malformed HTML content, example:

<br>
<li>Bachelor of Arts</li>
<li>Bachelor of Biomedical Science</li>
<li>Bachelor of Commerce</li>
<li>Bachelor of Commerce Specialist</li>
<li>Bachelor of Computer Science</li>
<li>Bachelor of Education (Honours)</li>
<li>Bachelor of Engineering (Honours)</li>
<li>Bachelor of Global Studies</li>
<li>Bachelor of Information Technology</li>
<li>Bachelor of Laws (Honours)</li>
<li>Bachelor of Music.</li>
<br>

The <li> tags are not correctly enclosed in a ordered / unordered list. My assumption is that tidy-html5 would have caught this. I've been through the documentation a few times, am I missing a switch somewhere or am I using the tool in an incorrect way :(

[geoffmcl]

@Rodeoclash thanks for the issue...

Yes, unfortunately, current tidy will not add say a <ul> around your sample block... but maybe it should... and I know of no option to cause this...

Maybe there should be one, or maybe it should be done by default... the W3C validator seems to agree...

And the W3C specs, like https://dev.w3.org/html5/spec-preview/the-li-element.html, indicates "Contexts in which this element can be used:" as <ol>, <ul>, <menu> parents... but it does go on with Otherwise ...! Need to read and examine more W3C specs... any more definitive pointer really appreciated...

I hesitate to call this a bug, but maybe it is, but it is certainly a request to change tidy's behaviour, namely a Feature Request...

I will try to look at this, over time, but as always, feedback, a patch, or PR, would be very welcome... thanks...

[Rodeoclash]

Thanks for the response @geoffmcl - I'd hazard it sits somewhere between feature request and bug.

[LWillms]

But how should Tidy decide where to put the <ul> and </ul> tags?

Before or after the <br>? Or instead?

These are questions of the actual meaning, i.e. semantic questions. Tidy should not try to find out what the original author had meant, should not deal with semantics, only with the syntax.

My recommendation is to do the correction to semantic errors by hand, with a sweeping change in some editor, or with a self written script using Regular Expressions. I use as editor UltraEdit, and with that I can do changes in all open files. Or I use the "Inforapid search and replace" to make changes in a whole set of files. For the upgrading project (from HTML 3.2 and 4.1 to HTML5) which has be brought here, I have written a VBscript using Regular Expressions which changes e.g. headers marked only by presentation oriented HTML tags as semantically marked headers with <H1> to <H6>. The script recognizes the meaning by elements which Tidy would not detect and should not take into account. Only then I throw this improved HTML on Tidy to correct syntax errors.

[geoffmcl]

@LWillms the short answer is, as often written, Tidy can not read authors' minds ;=))

But in this case it would put the <ul> after the firsts <br>, that is before the first <li>, since the W3C spec says an <li> can only be a child of <ol>, <ul>, <menu> parents, and the </ul> before the second <br>... Tidy has good internal structures and can easily do this...

Since the general idea is Tidy should try to not output invalid html, then at least it should flag this as an error, and not produce html...

But then we have a --force-out yes option, saying do your best, regardless, and understand no guarantee of the result... but in such a forced output it could add the <ul>...</ul> pair... with additional appropriate warning that it is doing this...

My whole point is, at the moment, given the above sample, and --show-body-only yes will output No warnings or errors were found. which is wrong in my view... effectively a bug... Tidy should do something...

And yes, the user, any user, can apply pre and/or post processing, by a myriad of apps, as they see fit...

Thanks for your feedback... always welcome...

[BBrey]

Well, to make some more fuss about this issue, we received an update on tidy for php 5.6.31 and this now breaks behaviour. Version 5.6.30 used a very old tidy lib (2009) which surrounded <li> with an <ul>.

We are now looking for a workaround.

[geoffmcl]

I have just done some more research, and find that up to, and including, tidy 5.3.10, circa Sept 11, 2016, tidy would add a <ul> ... </ul> around that block of <li> tags, using my in_525-2.html test, with the warnings and output -

line 11 column 1 - Warning: <li> isn't allowed in <body> elements
line 9 column 1 - Info: <body> previously mentioned
line 11 column 1 - Warning: inserting implicit <ul>
... skipped ...
<body>
<br>
<ul>
<li>Bachelor of Arts</li>

Then it was gone in tidy 5.3.11, and remains missing through to the current 5.7.0, so should be possible to track down the commit that changed this behaviour... and restore it...

Hmmm, how to review all commits between these two versions? Maybe checkout and diff, or something...

Will try to find time to look into this, unless someone beats me to it... but meantime moving out the milestone...

Should not be too difficult to fix this bug... thanks...

[geoffmcl]

Well the patch that broke this behaviour was easy to find -

diff -ur tidy-html5-5.5.10\src\parser.c tidy-html5\src\parser.c
--- tidy-html5-5.5.10\src\parser.c	Thu Nov 30 14:39:53 2017
+++ tidy-html5\src\parser.c	Thu Nov 30 14:41:12 2017
@@ -4022,9 +4022,9 @@
         */
         lexer->excludeBlocks = no;
         
-        if ( nodeIsINPUT(node) ||
+        if (( nodeIsINPUT(node) ||
              (!TY_(nodeHasCM)(node, CM_BLOCK) && !TY_(nodeHasCM)(node, CM_INLINE))
-           )
+           ) && !TY_(IsHTML5Mode)(doc) )
         {
             /* avoid this error message being issued twice */
             if (!(node->tag->model & CM_HEAD))

Effectively, this allows the <li> to be in the <body>, if in HTML5 mode... should still add implicit <ul> if a legacy doctype...

I added this when addressing issue #428... and merged the issue-428 branch... I did add a BIG caveat -
"And maybe this is wrong"!!!

Well, now I have the feeling to undo this change, which maybe re-opens #428, and find a better, perhaps more restrictive, patch to be added to again fix it...

Will experiment a little on this... be back soonest... feedback very welcome... thanks...

[jidanni]

Not a murmur from tidy,

$ echo '<LI>X</LI>'|tidy -eq
line 1 column 1 - Warning: missing <!DOCTYPE> declaration
line 1 column 1 - Warning: inserting implicit <body>
line 1 column 1 - Warning: inserting missing 'title' element
$ tidy --version
HTML Tidy for Linux version 5.6.0

One would hope this would make errors,
and not get cleansed on a second run,

$ tidy x.htm | tidy