json

JSON Reading and Writing — machines/json.shoddy

Summary

json reads and writes JSON — the format nearly every web service, configuration file and data export speaks. Hand it a stretch of text and it hands you back a value you can walk into. You can look up a key, index an element, or follow a path down through the nesting. Hand it a value and it hands you back text, either packed tight for a machine or laid out with indentation for a person. It goes both ways. What comes out of one end goes back into the other unchanged.

It follows the standard exactly. That standard is RFC 8259 — the internet document that defines JSON — and this machine does the whole grammar and nothing but. Every escape works, including \uXXXX. Numbers work in all their spellings, and nesting works to any depth. You may have seen relaxations elsewhere: comments, trailing commas, unquoted keys, single quotes, NaN. Those are not JSON, and they are refused here.

Reading comes in two forms, because the text you parse — read and turn into a value — is so often someone else's. JsonParse is the strict one: it gives you the document or stops the program. JsonRead is the total one: it always returns a value, either the document or a note saying what went wrong and at which character. Shoddy has no way to catch a program once it has stopped, so the second form is how a program survives input it did not write.

A Brief History of JSON

JSON is a small idea that got very large. In the early 2000s Douglas Crockford wanted a way for a browser and a server to pass structured data back and forth, without inventing a format for each application. He noticed that JavaScript already had a perfectly good notation for writing values down — objects in braces, arrays in brackets, strings in quotes. The notation was easy for a person to read and trivial for a machine to parse. So he did not invent anything. He wrote down the subset that was already there, called it JavaScript Object Notation, and put it on a one-page website.

That restraint is why it won. The competition at the time was XML: angle brackets, namespaces, schemas, entities — a specification you could not hold in your head, and a parser you would not write yourself. JSON had six kinds of value — object, array, string, number, true, false, null — and a grammar that fits on a postcard. Anyone could implement it in an afternoon, in any language, and thousands of people did. By the end of the decade it had displaced XML for almost everything that talks over a network.

It was standardised twice over, which tells you something about how widely it had spread. One standards body published it as ECMA-404 in 2013. The internet standards body, the IETF, published it as RFC 7159 and then RFC 8259. The specifications are famously short. They are also famously strict about things people assume are allowed. A JSON document may not contain a comment. It may not have a trailing comma after the last element. It may not quote its keys with apostrophes, may not write NaN or Infinity, and may not leave a key unquoted. Every one of those is something a hand-written parser tends to accept by accident. This machine refuses all of them, deliberately, and has a test for each.

The looser dialects do exist — JSON5 and its relatives add back the comments and the trailing commas. But they are a different format wearing the same name. Mixing them silently is how two programs come to disagree about what a file says. If you want them, you want a different machine.

Why It's Useful

Sooner or later a program has to talk to something that isn't it. A weather service, a price feed, a configuration file someone edits by hand, a data set a colleague exported, the saved state of your own program from last Tuesday. Overwhelmingly, the thing on the other side speaks JSON. Without a reader you are reduced to hunting through the text with Instr and Mid. That works right up until a value contains a comma, or a quote, or a newline. Then it quietly stops working, in a way you won't notice for a month.

Reach for json whenever data crosses the boundary of your program. Reading: a response you fetched, a file of records, a settings file, a fixture for a test. Writing: results another program will read, a document you'll diff against tomorrow's, a saved structure you mean to load back. It is also the least surprising way to store something structured on disk. You can open the file and read it, which is more than can be said for most binary formats. And isam is there for when you need keyed lookup instead of a whole document at once.

User's Guide

A document is a Json value: one of JNull, JBool, JNum, JStr, JArr or JObj. You get one by parsing text, and you take it apart with the navigation words.

Include "json.shoddy"

Def Main()
    Let doc = JsonParse(ReadFile("forecast.json"))

    ' walk down through the nesting by naming the keys
    Let periods = JPath(doc, { "properties", "periods" })
    Print(JCount(periods))                       ' 14

    ' index an element, then read fields out of it
    Let today = JNth(periods, 1)
    Print(JStrOr(JGet(today, "name"), "?"))      ' Tonight
    Print(JNumOr(JGet(today, "temperature"), 0)) ' 63

    ' build a document and write it back out
    Let out = JPut(JPut(JObj({ }), "when", JStr("tonight")), "temp", JNum(63))
    Print(JsonText(out))                         ' {"temp":63,"when":"tonight"}
    Print(JsonPretty(out, 2))                    ' the same, laid out over lines

A few things worth remembering:

Two ways to read, and you want both. JsonParse stops the program if the text is malformed — not shaped the way JSON requires. That is what you want for a file you wrote yourself. JsonRead hands back JOk(doc) or JErr(why, where) and never stops anything. That is what you want for text that arrived from somewhere else. The same pairing runs through the whole machine: JGet and JGetOr, JPath and JPathOr, JText and JStrOr.
The Or words never fail. JStrOr(v, "?") gives you the string if v is one, and your default if it is anything else — including if it's a number, and including if it's null. They do not convert. Asking JNumOr for the number in "7" gives you the default, because a JSON string is not a JSON number. Pretending otherwise is how data gets quietly corrupted.
Compare with JEqual, not =. Two documents that say the same thing are not = each other. Arrays and objects are built on lists, and a list compares by identity — is it the very same list? — rather than by contents. JEqual is the deep comparison: it looks inside. It treats objects as unordered, the way JSON says they are.
Everything is pure. JPut and JDel hand you back a new document with the change made; the one you passed in is untouched.
Member order is the document's order, because an object here is a list of pairs rather than a set. Putting a key that is already there moves it to the front — a consequence of how DictPut works. If you need a stable order for comparing files, JsonCanonical sorts every object's members by key, all the way down.

Under the Hood

You don't need any of this to use the machine — it's here for the curious.

The representation. Json is a sum type — a value that is always exactly one of a fixed set of shapes — with six variants. An object's members are a List Of Pair: the key in Fst, the value in Snd. That is not a coincidence. It is exactly the association list — a plain list of key-and-value pairs — that dict is built on. So DictGet, DictPut, DictDel and DictHas work on an object's members with no adapter at all. JGet, JPut, JDel and JKeys are thin wrappers over dict rather than a second implementation of the same idea. Every field name is prefixed with J for a duller reason. Accessor words share one flat namespace with the builtins, and a builtin quietly outranks an accessor. A field called Str or Len would therefore never be reached.

The reader is recursive descent — one small reading routine per kind of value, each calling the others as the text demands. It walks the string with a 1-based index, and it never slices the input. Slicing would mean taking the tail of a 13 KB document at every character, which turns a linear parser into a quadratic one — one whose work grows with the square of the input rather than in step with it. Each step returns both what it read and where it stopped. Positions therefore thread through the whole descent, and an error can say exactly which character defeated it.

The split that matters is between the two kinds of recursion. The character scanners — skipping whitespace, running through digits, reading a string body — call only themselves, in tail position: the call is the last thing the word does. The mill compiles such calls to a loop, so they cost no stack — the memory that tracks calls in progress — however long the input. Values, arrays and objects genuinely call one another. That does cost stack, but only in proportion to how deeply the document nests. The depth is small for real data and unbounded for hostile data: [[[[[ repeated is two bytes per stack frame. A .NET stack overflow cannot be caught, so the program would die with no message at all. Hence a depth limit of 200 by default, spent one level at a time and reported as an ordinary error, with JsonReadDepth to raise it when a document genuinely needs it.

Numbers are scanned by hand before Val ever sees them. Val is C's strtod, and it is far more permissive than JSON — it will happily take +1, .5, 5. and hexadecimal. So the grammar -? ( 0 | [1-9][0-9]* ) ( "." [0-9]+ )? ( [eE] [+-]? [0-9]+ )? is validated first, and Val is called only on a slice already known to match it.

The writer never builds its output by concatenating onto a growing string. That approach is quadratic in the size of the result — exactly the wrong shape for a serializer, a program that turns values into text. Instead, every writer word prepends fragments onto an accumulator, at constant time each, and the join happens once, at the top. The escaper goes further and tracks runs: an ordinary string with nothing to escape produces a single fragment rather than one per character.

What it costs. Reading is linear in the length of the text — double the text, double the work — and writing is linear in the size of the output. Looking a key up inside an object is a linear scan of that object's members, inherited straight from dict. That is fine for the tens of members real documents have, and it is the same honest trade dict makes. Collapsing duplicate keys at parse time costs a scan per member, for the same reason and with the same caveat.

Where it stops. Numbers are written with Str, which gives ten significant digits. A document full of arbitrary doubles therefore does not survive a round trip bit-for-bit. That is the language's number formatting rather than anything this machine chooses, and values that fit in ten digits come back exactly. And a string cannot carry NUL — the character whose code is zero — because Chr(0) is the empty string in Shoddy. A @BS@u0000 escape is therefore refused by name rather than dropped silently. Dropping it would hand you a string shorter than the document said it was.

Word Reference

Reading, writing, walking and building

Reading

Word	Description
JsonParse(s)	The document in `s`. Stops the program with the reason and the character offset if the text is not valid JSON. The one to reach for when the text is yours.
JsonRead(s)	Total — it always returns a value. Hands back `JOk(doc)` or `JErr(why, at)`, where `at` is the 1-based character the parse gave up on. Nothing it is given can stop the program, which is what makes it safe for input you did not write.
JsonReadDepth(s, maxDepth)	As `JsonRead`, with the nesting limit given explicitly. `JsonRead` is this with the default of 200.
JsonLoad(path)	Reads the file and parses it, strictly.

Writing

Word	Description
JsonText(j)	Compact text — no whitespace anywhere outside strings. Stops the program if the document holds a number that is not finite, since neither `nan` nor `inf` is JSON.
JsonPretty(j, indent)	Laid out over lines, `indent` spaces per level, following the same convention as `python -m json.tool` so the output diffs against other tools. `indent` of 0 means compact.
JsonSave(path, j)	Writes the compact text to a file.
JsonEscape(s)	The inside of a JSON string — the escaping without the surrounding quotes. Useful on its own.
JsonCanonical(j)	The same document with every object's members sorted by key, recursively. For output you mean to compare byte-for-byte against yesterday's.

Walking a document

Word	Description
JGet(j, k)	The value filed under key `k`. Stops the program if `j` isn't an object or the key isn't there.
JGetOr(j, k, dflt)	Same, but hands back `dflt` instead of stopping.
JHas(j, k)	True or False. False for a non-object rather than an error, so you can ask without checking first.
JNth(j, n)	The `n`th element of an array, counting from 1 like everything else in the language.
JPath(j, path)	Follows a list of keys down through the nesting: `JPath(doc, { "properties", "forecast" })`.
JPathOr(j, path, dflt)	Same, giving back `dflt` the moment a key along the way is missing.
JKeys(j)	Every key of an object, in document order.
JCount(j)	How many members an object has, or how many items an array has.
JEqual(a, b)	Deep comparison — the one to use, since `=` cannot see inside a list. Objects compare as unordered, arrays as ordered.

Reading values out

Word	Description
IsJNull(j)	Is this the JSON `null`?
JStrOr(j, dflt)	The string inside, or your default if it isn't a string. Never converts.
JNumOr(j, dflt)	The number inside, or your default.
JBoolOr(j, dflt)	The boolean inside, or your default.
JText / JNumber / JFlag / JItems / JMembers	The strict halves, straight off the type: the value inside, or the program stops because it was the wrong kind. These are what the `Or` words guard.

Building

Word	Description
JPut(j, k, v)	A new object with `v` filed under `k`. Replaces the key if it was already there — and moves it to the front, as `DictPut` does.
JDel(j, k)	A new object without that key. Harmless if it wasn't there.
JNumList(xs)	A list of numbers as a JSON array.
JStrList(xs)	A list of strings as a JSON array.
IsFinite(x)	True unless `x` is an infinity or not a number. The guard the writer uses, since there is no such builtin and `Abs(x) > 1e308` would reject perfectly good numbers.

Who Uses It

	User	How
	weather-glass	Every response. Object keys with spaces in them, coordinates that arrive as strings, and a `null` precipitation that has to read as zero.

The Machines It Uses

	Machine	Why
	dict	An object's members are a dict — the same `List Of Pair` — so `DictGet`, `DictPut` and `DictDel` do the key work unchanged.
	seq	`Zip` pairs keys with values, and `All` checks a document before it is written.
	sinq	`OrderBy` puts an object's members in key order for `JsonCanonical`. The `Sort` builtin will not order a `List Of Pair`, so this machine used to sort the keys and look each one back up.
	str	`Join` puts the writer's fragments together at the end, and `StrRep` makes the pretty printer's indentation.