There is not much that needs to be said about this. After HTML 4, it seemed that the future of HTML was XHTML, "a reformulation of HTML 4 in XML" (instead of SGML). The W3C came out with a few specifications for XHTML 1.0 (with 3 different versions: strict, transitional, and frameset) and 1.1. It seemed for a while that there would be XHTML 2.0.
Instead we got HTML 5, which allows both an XML syntax and a non-XML syntax.
Of course, the non-XML syntax is not radically different from XML, like the compact syntax for RELAX NG. It's just more like traditional HTML.
So in XML syntax we might have a list of numbered items followed by a paragraph:
<ol> <li>First</li> <li>Second</li> </ol> <p>I came, I saw, I conquered.</p>
In non-XML syntax we could break (some of) the rules (the "five commandments") of XML, such as case-sensitivity and the closing tags:
<OL> <li>First <li>Second</LI> <P>I came, I saw, I conquered.
Personally, I prefer the XML syntax in HTML 5.