html

HTML Reading and Writing — machines/html.shoddy

Summary

html reads a web page into a tree you can walk, and writes a tree back out as HTML. It tolerates what real pages are actually like: missing end tags, unquoted attributes, uppercase names, a stray < in the prose, a <script> full of characters that would otherwise be markup. It does not give up on a page because one thing in it is wrong.

A parsed page is an xml tree. Everything that machine can do — XFind, XPath, XTextOf, XAttrOr, XEqual — works here unchanged. html adds only what is specifically about HTML.

A Brief History of HTML

Tim Berners-Lee wrote the first version in 1990 at CERN, the European physics laboratory. It was a short list of tags borrowed from an in-house dialect of SGML, an older markup language. There was no specification for several years. There was a browser, and whatever it accepted was the language.

That accident set the character of everything since. Pages were written by people, not compilers, and they were full of mistakes. A browser that refused a page simply lost to a browser that rendered it anyway — so every browser rendered it anyway. Error recovery became the language's central feature, and it was completely undocumented. Two browsers guessed differently about the same broken page. So pages were written to work in one browser and not the other, which is most of what the 1990s were about.

XHTML, around 2000, tried to fix this by declaring HTML to be XML: one mistake, and the page does not render. Authors did not want that. Neither, in the end, did anyone else, and the effort collapsed.

What replaced it was the opposite idea, and the better one. HTML5 began around 2004 and became a Recommendation — a finished web standard — in 2014. It specified the error recovery itself: a tree-construction algorithm precise enough that every browser now builds the same tree from the same broken page. It is a remarkable document, and it is roughly the size of this entire repository.

That is why this machine is honest about being something smaller. It implements the forgiving parts a hand-written page actually relies on. It says plainly which of the clever parts it does not.

Why It's Useful

HTML is the most widely produced structured text in the world, and almost all of it is written by hand or emitted by something in a hurry. Suppose you want to know what is on a page: the links, the headings, the table of numbers, the text without the navigation. Then you need something that reads what is there, rather than what ought to be there.

The demonstration that comes with this machine is a real one. tst/html-demo.shoddy reads every page in this repository's own documentation. It reports dead links, anchors pointing at ids that do not exist, ids used twice on a page, and images with no alt text. It runs in a few seconds, and it has already earned its keep. The machines catalog wires twenty-six pages into a previous/next chain by hand, and a chain like that fails silently. A page can be in the catalog, on the home page and in the sitemap, and still be unreachable by paging — because two of its neighbours agree with each other and not with it. That had happened. A person did not notice. A program noticed immediately.

Include "html.shoddy"

Def Main()
    Let page = HtmlLoad("docs/index.html")
    Print(HTitle(page))
    Print(Str(Length(HLinks(page))) & " links")
    Print(HtmlTextOf(page))          ' the readable text, without the script

User's Guide

The tree is xml's

There is no Html type. HtmlParse returns an XDocument holding XElem, XText, XComment and XDoctype nodes. That is the same tree xml builds, so the same words walk it:

Let page = HtmlLoad("index.html")
Print(XTextOf(XFind(page, "h1")))                 ' xml's XFind
Print(XAttrOr(HById(page, "main"), "class", ""))  ' xml's XAttrOr

That is the whole design, and it is the same debt json owes dict: when the data structure is already right, use it rather than writing a second one that is nearly the same.

The four questions that are about HTML

You want	Word
an element by id	`HById(page, "main")`
elements by class	`HByClass(page, "mast")`
elements by tag	`HByTag(page, "td")`
every link, image or id	`HLinks`, `HImages`, `HIds`
the title	`HTitleOr(page, "")`
the readable text	`HtmlTextOf(page)`

HByClass searches the class attribute by membership, not by equality: an element with class="mast has-icon" is found by either name. HtmlTextOf is the text a reader would see, which means the <script> and the <style> are left out. XTextOf is still there when you want everything.

Under the Hood

What "forgiving" means, exactly

The test suite pins it down, rather than a description. tst/html/soup.html is written badly on purpose, once for every rule. tst/html/soup-clean.html is what the writer makes of it, compared byte for byte. What is handled:

Case folding. <DIV> and <div> are one tag. HREF and href are one attribute.
Void elements. The fourteen tags that take no content. They parse whether closed or not, and are written back bare — <br>, never <br/>.
Raw text. Inside <script> and <style> a < is not markup, which is the only way a page can carry if (a < b).
Three attribute spellings. Quoted with either quote, unquoted, or bare — a name with no value, which HTML gives the empty string. A repeated attribute keeps the first. That is the opposite of what a JSON object does with a repeated key.
Implied end tags. A <li> closes the one before it. A <p> closes when a block element opens. A table cell closes when the next cell or the next row begins.
Stray markup. A < that begins nothing is text, and an end tag matching nothing that is open is dropped.

The open-element stack

That last rule needs more than it looks. Suppose an end tag arrives that does not match the innermost open element. There are two possibilities, and they want opposite treatment. It may belong to an ancestor: then everything down to that ancestor should close. Or it may belong to nothing: then it should be discarded, and parsing carries on where it was.

Without knowing which elements are open, you cannot tell those apart. And guessing wrong is not a small error. Consider a stray </p>, left over from a paragraph some other rule had already closed. Guessed wrong, it would close the <body> and the <html> around it. Every element after that point would be torn off the tree and left dangling at the top level. So the list of open tags is threaded down through the reader, and the two cases are told apart by asking it.

The entity table, and why it holds bytes

HTML5 names 2231 character references — spelled-out names for characters, such as — for an em dash. This table holds about eighty, chosen by counting what hand-written pages actually use: the arrows, dashes, quotes and symbols. Numeric references also work, in either base — decimal or hexadecimal. An unknown name is left standing as its own text, never an error. Refusing to render a page over one bad ampersand is precisely the behaviour HTML spent thirty years getting rid of.

The values are UTF-8 bytes, not code points (a code point is a character's number in Unicode), and xml explains at length why. ReadFile is one byte to one character, so — has to decode to the same three bytes that a literal em dash already is in the file. Decoding it to Chr(8212) would make the entity and the literal two different strings. WriteFile would then turn one of them into a question mark.

The writer is not xml's

HTML is not XML, and serialising it as though it were — writing the tree back out by XML's rules — produces pages that break. A void element goes out bare. A raw-text element's content goes out unescaped, because escaping a < inside a script would change the program. An empty non-void element goes out as a pair: <div></div>, never <div/>. A browser reads <div/> as an open div that then swallows the rest of the page.

WHAT THIS IS NOT

This is a forgiving tag-soup reader, not an HTML5 parser, and the difference is not a formality. (Tag soup is the nickname for the messy HTML real pages are made of.) HTML5's tree-construction algorithm exists to make every browser agree on what a broken page means. It includes the list of active formatting elements, the adoption agency, foster parenting, and the twenty-odd insertion modes. None of that is here.

Concretely: misnested formatting is not repaired, so <b><i>x</b></i> does not come back as the two overlapping runs a browser would build. A <tbody> a browser would invent is not invented. Content that HTML5 would move out of a table and place before it is left where it was written. Such a page will still parse — this reader does not abort easily. But the tree will be the one the bytes describe, not the one the browser shows.

Word Reference

Reading and writing

Word	What it does
`HtmlParse(s)`	Parse a page; aborts only on the depth budget.
`HtmlRead(s)`	Total: `XOk(doc)` or `XErr(why, at)`.
`HtmlReadDepth(s, n)`	As above with your own nesting budget.
`HtmlLoad(path)`	`HtmlParse(ReadFile(path))`.
`HtmlText(x)`	Compact HTML, with HTML's serialisation rules.
`HtmlPretty(x, n)`	Indented, leaving mixed content and scripts alone.
`HtmlSave(path, x)`	Compact, to a file.

Escaping

Word	What it does
`HtmlEscape(s)`, `HtmlEscapeAttr(s)`	Text and attribute values, safely.
`HtmlUnescape(s)`	References to text; an unknown name stays as it is.
`HtmlEntity(name)`	What a name stands for, or `""`.

Finding things

Word	What it does
`HById(x, id)` / `HByIdOr` / `HAllById`	By id; the third finds the duplicates.
`HByClass(x, c)`	By class, searching the list by member.
`HByTag(x, t)`	By tag name, anywhere below.
`HLinks(x)`, `HImages(x)`, `HIds(x)`	Every href, src and id, in document order.
`HTitle(x)` / `HTitleOr(x, d)`	The document title.
`HtmlTextOf(x)`	The readable text, without script or style.

Asking about tags

Word	What it does
`HIsVoid(t)`	Does this tag take no content?
`HIsRawText(t)`	Is its content text rather than markup?
`HVoids()`, `HRawText()`, `HEntities()`	The tables themselves, should you want to read them.

Who Uses It

No machine and no mill includes it yet — its words are already at the reckoner's prompt through its seed, and a mill that puts them to work will appear here.

The Machines It Uses

	Machine	Why
	dict	`DictHas` and `DictGet` read an element's attribute list, which is an association list like any other.
	seq	`Contains` tests a tag against the void and raw-text sets.
	str	`Split` and `Join` under the attribute scanner and the serializer.
	xml	The whole tree, and every word that walks it. `html` adds a reader, a writer and an entity table, and redefines nothing.