https

An HTTP Client over TLS — machines/https.shoddy

Summary

https fetches a page or an API response over TLS and gives you words to take the reply apart: the status, the headers, the body. TLS is the encryption layer that keeps web traffic private — the S in HTTPS. HTTP is the request-and-reply language the web speaks. This machine is the HTTP; it is built on net, which supplies the sockets and the encryption.

Let reply = HttpsGet("api.weather.gov", "/points/41.88,-87.63",
    { Pair("User-Agent", "my-mill (me@example.com)"),
      Pair("Accept", "application/geo+json") })
If HttpStatus(reply) = 200 Then
    Print(HttpBody(reply))

A Brief History of HTTP over TLS

HTTP arrived in 1991 as a page of text: a request line, a response, no headers, no status codes. It said nothing about privacy because the network it ran on was a few thousand people who mostly knew each other.

Netscape added SSL — TLS's ancestor — in 1994 and got it wrong twice. Version 1.0 never shipped, and 2.0 was broken within a year. SSL 3.0 in 1996 became the thing the web actually ran on for a decade. The IETF, the body that writes internet standards, took it over and renamed it TLS to avoid Netscape's trademark. TLS 1.0 shipped in 1999 as what was essentially SSL 3.1 with the serial numbers filed off. Everything before TLS 1.2 is now deprecated — retired as unsafe — and everything before 1.0 is broken beyond repair.

The interesting part for a language like this one is what didn't change. TLS is a layer under HTTP, not a modification of it. The bytes above the encryption are the same bytes HTTP always spoke, which is why this machine is small. The transport got harder; the protocol did not. What the layering does cost is the ability to ask a socket whether data is waiting, and net's page explains what that costs in turn.

The other thing that changed is the default. HTTPS was for payment pages and login forms until roughly 2014. Then Snowden, Let's Encrypt, and browsers marking plain HTTP as "not secure" turned it into the ordinary case within about five years. That is why a machine that speaks only port 80 — the numbered door plain HTTP answers on — is now a teaching aid rather than a tool.

Why It's Useful

The plain HttpGet in net proves the socket stack works and cannot talk to the modern web. Almost nothing answers on port 80 any more except to say "go and ask on 443" — HTTPS's door. A good deal of what does answer refuses a request that arrives without a User-Agent, the header that says who is asking.

The concrete target that shaped every decision here is api.weather.gov, which was probed live. Port 80 returns 301 Moved Permanently with an empty body and a strict-transport-security header carrying preload — there is no plaintext (unencrypted) path at all. An empty User-Agent gets 403. Forced to HTTP/1.0 it answers cleanly, with a Content-Length, Connection: close, no chunked encoding (a body sent in pieces) and no gzip (compression). Every choice below follows from those three facts.

User's Guide

Pure where it can be

This is the design rule, and it is worth stating first. Building a request and reading a reply are functions on Strings. HttpRequestText, HttpStatus, HttpReason, HttpHeaders, HttpHeader and HttpBody touch no socket, need no --allow-net, and are tested offline against fixture text in tst/nettest.shoddy. Only HttpsGet and HttpsRequest reach the network.

The practical effect is that the interesting half of your program is testable without a server, and the half that needs one is four lines.

Headers are a dictionary

Headers come back as a List Of Pair — which is dict's association list, so DictGetOr works on them with no help from this machine:

Let hs = HttpHeaders(reply)
Print(DictGetOr(hs, "content-type", "unknown"))
Print(HttpHeader(reply, "Content-Type", "unknown"))   ' the same answer

Names are lowercased on the way in. HTTP header names are case-insensitive, and normalising at the boundary is what lets that lookup just work. You never have to guess whether this particular server wrote Content-Type or content-type. The name you ask HttpHeader with is lowercased for you too, so both sides of the question are case-blind.

Sending a request

HttpRequestText emits Host first and always; everything else is yours. Send a User-Agent. It is courteous, it is how an operator tells your traffic from a scraper's, and more hosts than you would expect return 403 without one.

Under the Hood

HTTP/1.0, deliberately

Every request this machine sends says HTTP/1.0, and that is a choice rather than an oversight. It makes the peer closes when it has finished the correct end-of-body rule. So RecvAllTls reads to end of input and stops. There is no chunked decoder here, because a server answering 1.0 does not send Transfer-Encoding: chunked.

Speaking 1.1 would buy keep-alive — reusing one connection for several requests — and cost a chunk decoder, a Content-Length reader and a connection cache. That is a fair trade one day, and an entire second machine's worth of work. It is not this one.

What it will not do for you

Redirects are reported, not chased. A 301 comes back as a 301 with its location header intact. Following one means deciding about loops, cross-host hops and whether a POST stays a POST. Those are your policy rather than a library's.
No compression. Nothing sends Accept-Encoding, so nothing arrives gzipped. Do not add the header without adding a decoder.
The body is bytes. One per character, exactly as ReadFile hands them over, so a UTF-8 page arrives as its bytes. Writing it back out with WriteFile reproduces the file exactly. Turning those bytes into characters is a separate concern this machine does not have.
Client only. Server-side TLS is not something Shoddy has. See emley-moor for a server, which necessarily speaks plaintext on the loopback.

A secured socket blocks

Worth knowing even if you never touch the transport directly: once Secure has upgraded a handle, Recv on it blocks — stops and waits — and Ready is refused outright, aborting the program. net's page explains why a poll cannot answer honestly through TLS records. Nothing in this machine calls Ready, so you will not meet the refusal by using HttpsGet. You will meet it if you build your own loop on a secured socket.

Word Reference

Building a request — pure

Word	What it does
`HttpRequestText(method, path, host, headers)`	The request exactly as it goes on the wire: request line, `Host`, your headers, blank line, CRLF line endings (carriage return then line feed) throughout.

Reading a reply — pure

Word	What it does
`HttpStatus(reply)`	The status number, or `0` if this is not a reply.
`HttpReason(reply)`	The reason phrase — `OK`, `Not Found`.
`HttpHeaders(reply)`	Every header as a `List Of Pair`, names lowercased, values trimmed.
`HttpHeader(reply, name, dflt)`	One header, asked for in any case.
`HttpBody(reply)`	Everything after the blank line; a bare LF gap is tolerated.
`HttpHead(reply)`	The header block without the body.

Fetching — effectful, and gated

Word	What it does
`HttpsGet(host, path, headers)`	One GET over TLS on port 443, returning the whole raw reply.
`HttpsRequest(method, host, path, headers)`	The same for any method.
`HttpsPort()`	443.

Who Uses It

	User	How
	emley-moor	The server reads its incoming requests with `HttpHeaders` — a request has the same header block a reply does, so the words read one without knowing the difference.
	weather-glass	Four HTTPS GETs: the ZIP lookup, then the points call and the two forecasts.

The Machines It Uses

	Machine	Why
	dict	Headers are an association list, so `HttpHeader` is `DictGetOr` with the name lowercased first.
	net	The sockets and the TLS upgrade: `RequestTls` does the connecting, securing, sending and draining.
	seq	`Append` accumulates the reply as fragments, so a drained body is joined once rather than copied per chunk.
	str	`Split`, `Join`, `Trim` and `StartsWith` do the parsing; there is no other machinery in here.