Application Layer: HTTP; DNS; Email


We now move to the top of the stack, where the protocols you use every day live. The remarkable thing about Layer 7 is how little it has to know: an application developer picks a transport protocol (TCP or UDP) and a destination, sets a message format, and the network takes care of the rest. Today we look at three of the most important application protocols — DNS, which turns names into addresses; SMTP, which moves mail; and HTTP, which moves the web — plus NAT, which is how a home full of devices shares a single public address. A theme to watch for: all three of these protocols are, at bottom, just text messages exchanged in an agreed format, which is why you can craft them by hand.

Slides from class

Files:

Additional information:

Agenda

These notes cover the application layer: how processes address each other, what applications need from the layers below, and then NAT, DNS, email, and the web in turn. Transport-layer mechanics (how TCP actually achieves reliability, how congestion control works) are still ahead of us, as is TLS in the security unit.

Notes

Processes, sockets, and protocols

An application-layer program runs on end systems and communicates over the network — web server software talking to browser software, for instance. Note what you don't have to write: software for network-core devices. Routers and switches do not run user applications. That restriction is a feature: because applications live only at the edges, anyone can write and deploy one without permission from or changes to the network itself, which is why new applications can appear and spread so quickly.

We'll assume the client/server paradigm throughout. A server is an always-on host with a permanent IP address, often in a data center. Clients contact the server, may be connected only intermittently, may have dynamic IP addresses, and do not communicate directly with each other. HTTP, IMAP, and FTP all work this way.

More precisely, the things that communicate are processes — programs running within a host. Two processes on the same host communicate using inter-process communication provided by the OS; processes on different hosts communicate by exchanging messages over the network. The client process is the one that initiates communication; the server process is the one waiting to be contacted.

A process sends and receives messages through its socket, and the door analogy is worth keeping in mind: the sending process pushes a message out its door and relies on the network infrastructure to carry it to the socket at the receiving process. There are always two sockets involved, one at each end. The socket is the boundary of responsibility — everything above it is controlled by the application developer, everything below it by the OS.

For a process to receive messages it needs an identifier. Is the host's IP address enough? No — many processes may be running on the same host. The identifier must include both the IP address and the port number associated with the process. To send an HTTP message to Dartmouth's web server you need IP address 129.170.224.138 and port 80.

An application-layer protocol defines three things: the types of messages exchanged (request, response); the message syntax, meaning what fields exist and how they are delineated (often in a standard format like JSON or XML); and the message semantics, meaning what the fields mean and the rules for when and how processes send and respond to messages.

Protocols come in two flavors. Open protocols are defined in Requests For Comments (RFCs) that anyone can read, which is what allows independently written software to interoperate — HTTP and SMTP are examples, and it's why your mail client works with someone else's mail server. Proprietary protocols, like Zoom's, are closed to outsiders.

What applications need from the transport layer

Different applications need different things from the network, along four dimensions:

Some representative cases: file transfer and email need no loss, are elastic, and are not time sensitive. Web documents are the same. Real-time audio/video is loss-tolerant, needs 5 Kbps–1 Mbps for audio and 10 Kbps–5 Mbps for video, and is sensitive to delays on the order of tens of milliseconds. Interactive games are loss-tolerant, need only a few Kbps, and are sensitive to tens of milliseconds. Text messaging needs no loss and is elastic, with timing requirements somewhere in between.

Against those needs, the Internet offers two transport services. TCP provides reliable transport between the sending and receiving processes, is connection-oriented (setup is required before data flows), and adds flow control so a sender can't overwhelm a receiver plus congestion control to throttle senders when the network is overloaded. What TCP does not provide: timing guarantees, minimum throughput guarantees, or security. UDP provides unreliable data transfer and none of the rest — no reliability, flow control, congestion control, timing, throughput guarantee, security, or connection setup.

Given that list, why does UDP exist at all? Because sometimes you want speed and can tolerate loss, and because everything TCP adds costs either time or flexibility. In practice: FTP, SMTP, and HTTP run over TCP; streaming audio/video typically uses TCP; Internet telephony uses TCP or UDP; interactive games use UDP or TCP.

In practice: FTP, SMTP, and HTTP run over TCP; stored video streaming (Netflix, YouTube) is generally fetched over HTTP and so over TCP, since a playout buffer absorbs the retransmission delay, though it is increasingly moving to QUIC over UDP; real-time audio and video conferencing uses UDP, because a retransmission that arrives after its playout deadline is useless.

Note the security gap. Vanilla TCP and UDP sockets provide no encryption at all, which means a cleartext password written into a socket traverses the Internet in cleartext. Transport Layer Security (TLS) fills that gap: it provides encrypted connections, data integrity, and endpoint authentication. Despite the name, TLS is implemented in the application layer — applications call TLS libraries, which use TCP in turn. Cleartext goes into the socket, ciphertext crosses the Internet, and the receiver decrypts. We'll cover TLS properly in the security portion of the course.

IP addresses and CIDR

An IPv4 address such as 129.170.224.138 is a fixed 32 bits, or 4 bytes, written as four octets separated by dots, each ranging 0 to 255. Addresses are hierarchical, read left to right: the left portion gives the network number and the right portion identifies a specific host on that network.

Where's the boundary? Under CIDR (Classless Interdomain Routing, 1993 onward), wherever we would like. You specify how many bits make up the network ID and the rest identify a device:

The number after the slash is the subnet mask. Compare this to the classful scheme it replaced (e.g., pre-1993), where an organization could have only 224, 216, or 28 addresses and nothing in between; CIDR lets an allocation be sized to actual need.

Some address ranges are reserved for private use:

10.0.0.0/8 — 224 ≈ 16.7 million devices
172.16.0.0/12 — 220 devices
192.168.0.0/16 — 216 ≈ 65 thousand devices

Private addresses are non-routable on the Internet, which is precisely what makes them useful: many different homes and offices can use the same private addresses inside without conflict. Also reserved is the loopback range 127.0.0.0/8, of which 127.0.0.1 (localhost) is the familiar member — traffic sent there loops back inside your own machine, so you can talk to yourself without knowing your own address. Your machine will show an interface named lo for this.

NAT: sharing one address

Why do private addresses exist? Because we ran out of IPv4 addresses. Thirty-two bits gives 232 ≈ 4 billion addresses total, and there are far more connected hosts in the world than that.

Consider a home network with a handful of hosts (and in reality, a few dozen once you count phones, TVs, thermostats, light bulbs, and so forth). One approach would be to give each host its own globally unique address, so a single home would consume many public addresses. That doesn't scale. Instead, the ISP issues one address for the whole home and the home uses private addresses inside. That arrangement is Network Address Translation (NAT).

Concretely: the router hands out internal addresses on one of the private ranges, say 192.168.60.0/24, using DHCP, so hosts are at 192.168.60.2, .3, .4 and so on, up to 254 of them (remember 0 and 255 are reserved). The router itself has two addresses — 192.168.60.1 on the inside, and 75.69.94.78 (issued by the ISP) on the outside. Now follow one request to a web server at 129.170.171.44:

  1. The host at 192.168.60.5 requests a page: source 192.168.60.5 port 9052 (OS assigned unused port), destination 129.170.171.44 port 80 (well-known web server port). Since the destination is outside, the request goes to the gateway router at 192.168.60.1.
  2. The router records the mapping — that 192.168.60.5 using source port 9052 is talking to 129.170.171.44 port 80 — then rewrites the source address to its own outside address, 75.69.94.78, and forwards the request.
  3. The web server sees a request from 75.69.94.78 and replies to it: source 129.170.171.44 port 80, destination 75.69.94.78 port 9052. The web server has no idea the private address exists.
  4. The router receives the reply, looks up which internal host was using port 9052 for a conversation with that server, finds 192.168.60.5, rewrites the destination accordingly, and delivers it.

That's NAT's big idea: every host inside shares one outside address, and the router uses port numbers to keep the conversations straight. This is also why the Day 3 walkthrough carried a caveat — the private addresses in that example were not what actually appeared on the public Internet.

DNS: names to addresses

Every host on the Internet has an IP address, and humans are terrible at remembering numbers. Introduce yourself as "172.217.165.142" and the other person will have forgotten it before you finish the sentence; introduce yourself as "Bob" and they'll probably retain it. Likewise you can remember google.com, amazon.com, and dartmouth.edu, but not 172.217.165.142, 205.251.242.103, and 129.170.224.138.

The Domain Name System (DNS) closes that gap: given a hostname such as google.com, it returns an IP address. DNS is an application-layer protocol — hosts and DNS servers exchange messages to resolve names — running over UDP on port 53, implemented as a distributed database spread across a hierarchy of many name servers.

Why distributed rather than one central server? Four reasons, each fatal on its own: a single server is a single point of failure; the traffic volume is enormous (Comcast alone handles around 600 billion queries a day, Akamai about 2.2 trillion); one location would be physically distant from much of the planet, adding latency we know we can't recover; and there would be no way to do maintenance without taking the Internet down.

The design constraints are worth appreciating. DNS is a humongous distributed database of billions of individually simple records, handling trillions of queries a day with far more reads than writes, decentralized so that millions of organizations each control their own records, and it is expected to be bulletproof. And it has to be fast: essentially every Internet transaction begins with a DNS lookup, so milliseconds count. That performance requirement is exactly why DNS uses UDP rather than TCP — a connection handshake before every name lookup would be intolerable.

What happens when you type a URL

Type www.dartmouth.edu into a browser and the following happens before any web traffic flows. The browser extracts the hostname from the URL and passes it to the client side of the DNS application, which runs on your machine. The DNS client sends a query containing the hostname to a DNS server over UDP port 53, and eventually receives a reply containing the IP address, 129.170.224.138. Only then can the browser initiate a TCP connection to port 80 at that address. On Linux and macOS you can invoke this from code with gethostbyname(<hostname>), or the more modern version getaddrinfo, which calls a DNS server on your behalf. This command is useful for parts of Lab 1.

Other things DNS does

Name-to-address translation is the headline service, but not the only one.

Host aliasing. A server with an unwieldy real name, say relay1.west-coast.enterprise.com, can have friendlier aliases like enterprise.com and www.enterprise.com; DNS returns the same address for any of them. The real name is called the canonical name.

Mail server aliasing. Bob's address is bob@enterprise.com even though the mail server is relay1.west-coast.mail.enterprise.com. DNS supplies an MX record naming the actual mail server for the domain.

Load distribution. An organization may run several web servers behind one name. DNS hands out the addresses in round-robin fashion — first query gets the first server, second query the second, and so on — spreading load without any special equipment.

The DNS hierarchy

DNS is organized as a tree, with four kinds of participant.

Root servers sit at the top. There are about 2,000 deployed worldwide, each a copy of one of 13 logical servers managed by IANA. Root servers provide the addresses of Top-Level Domain servers, and act as the official contact of last resort for a name server that cannot otherwise resolve a name.

Top-Level Domain (TLD) servers handle .com, .org, .net, .edu, .gov, and the country domains (.uk, .fr, .ca, .jp). Each TLD has a server or server cluster run by a designated company: Verisign Global Registry Services maintains .com, Educause maintains .edu. TLD servers provide the addresses of authoritative DNS servers. The Internet started with six TLDs — .com, .org, .gov, .mil, .net, .edu — and now has more than 1,500.

Authoritative DNS servers hold the definitive records. Every organization with publicly accessible hosts must provide records mapping those hostnames to addresses, and its authoritative server is where those records live. An organization can run its own or pay an ISP to host the records; most universities and large companies run their own primary and secondary (backup) servers.

Local name servers don't strictly belong to the hierarchy at all. Every ISP — residential, corporate, university — has at least one, also called a default name server. Your host most likely got the address of the local name servedr from DHCP and sends its queries here. The local server keeps a cache of recent name-to-address pairs and answers from cache when it can; when it doesn't know an address, or its cached entry has expired, it acts as a proxy and forwards the query into the hierarchy.

Iterated and recursive queries

Suppose a host at dartmouth.edu wants the address of gaia.cs.umass.edu. There are two ways to work through the hierarchy, and the difference is about who does the legwork.

In an iterated query, the local server does all the work and each server it contacts answers with a referral rather than an answer. The host asks dns.dartmouth.edu; the local server asks a root server, which says "I don't know, but ask the .edu TLD server"; the local server asks the TLD server, which says "ask dns.cs.umass.edu"; the local server asks the authoritative server, which returns the address; the local server passes it to the host. Eight messages, and the burden stays with the local server. The phrase to remember is "I don't know this name, but ask this server."

In a recursive query, the local server asks the contacted server to resolve the name on its behalf, so the query is passed down the hierarchy and the answer comes back up through the same chain. This puts the burden on the contacted servers, which is why the root and TLD servers generally don't accept recursive queries from everyone. User applications — browsers and the like — normally make recursive queries to their local server, which then iterates on their behalf.

Caching

Once any name server learns a host name/IP address mapping, it caches it. Cache entries carry a TTL (Time To Live) set by the server administrator, typically minutes to hours; 86,400 seconds (24 hours) is a common value. A local server with an unexpired cached entry answers immediately, even though it isn't authoritative for that name. In particular, the addresses of TLD servers are usually cached locally, so root servers are rarely visited in practice.

The cost is staleness: DNS is best-effort name translation. If a host changes its IP address, the change may not be known Internet-wide until every cached entry's TTL expires. Update and notify mechanisms exist (RFC 2136), but the basic model is "correct eventually."

Record types

A few record types cover most of what you'll see:

Querying DNS by hand

nslookup lets you ask for a specific record type. The A record for Dartmouth: 

$ nslookup -type=A dartmouth.edu
Server:  129.170.17.4
Address: 129.170.17.4#53

Name:    dartmouth.edu
Address: 129.170.224.138

The Server line is the name server that answered; the Address under Name is the A record. You can sanity-check the result by pinging 129.170.224.138, or by typing the bare address into a browser — the Dartmouth home page comes up, since the name dartmouth.edu was never anything but a convenience.

The MX record shows where mail goes: 

$ nslookup -type=MX dartmouth.edu
dartmouth.edu  mail exchanger = 5 dartmouth-edu.mail.protection.outlook.com.

Priority 5, and only one server listed. Note what this reveals: Dartmouth's mail is handled by Outlook. MX records are public, so a domain's mail provider is public too.

Get the mail server's IP address 

$ nslookup -type=A dartmouth-edu.mail.protection.outlook.com 
Server:   129.170.17.4
Address:  129.170.17.4#53

Non-authoritative answer:
Name: dartmouth-edu.mail.protection.outlook.com
Address: 52.101.41.192
Name: dartmouth-edu.mail.protection.outlook.com
Address: 52.101.11.15
Name: dartmouth-edu.mail.protection.outlook.com
Address: 52.101.9.20
Name: dartmouth-edu.mail.protection.outlook.com
Address: 52.101.50.7

Note: there are multiple email servers (for redundancy in case of a server failure), presented in round-robin order. Run this command again and you will likely get a different ordering of Dartmouth's email servers.

And the NS records name the name servers: 

$ nslookup -type=NS dartmouth.edu
dartmouth.edu  nameserver = dns2.dartmouth.edu.
dartmouth.edu  nameserver = dns1.dartmouth.edu.
dartmouth.edu  nameserver = west-campus-dns.dartmouth.edu.

Three of them. Why? Because DNS is critical infrastructure — if the name server is unreachable, everything in the domain is effectively unreachable — so you want redundancy. Run the command again and you'll see them listed in a different order: round-robin again, this time spreading load across the name servers themselves.

dig (Domain Information Groper) does the same job with more power and a different output format: 

$ dig dartmouth.edu
;; ANSWER SECTION:
dartmouth.edu.  300  IN  A  129.170.224.138

$ dig dartmouth.edu MX
;; ANSWER SECTION:
dartmouth.edu.  3600  IN  MX  5 dartmouth-edu.mail.protection.outlook.com.

Note the numbers 300 and 3600 — those are the TTLs in seconds, so you can read a record's caching lifetime straight out of the answer.

whois queries registration data rather than DNS records, and is often more revealing than people expect. whois dartmouth.edu returns the registrant (Dartmouth College, Dartmouth Network Services, with a street address), an administrative contact including a name, phone number, and email address, and the dates the record was activated (March 1986), last updated, and expires. For a network administrator this is how you find who to call about a misbehaving host; for an attacker it's reconnaissance and a starting point for social engineering.

DNS message format

Queries and replies use the same message format, which is convenient when you write your own client. The header carries a 2-byte ID, a flags field whose bits include QR (0 in a query, 1 in a reply), and counts of the questions, answers, additional records, and authority records that follow. A query for nytimes.com looks like: 

ID=0xced1  QR=0
# questions=1   # answers=0
# additional=0  # authority=0
questions = nytimes.com: type A

And the reply repeats the same ID (so the client can match it to the outstanding query, the same trick DHCP's transaction ID played), sets QR to 1, echoes the question, and adds answers: 

ID=0xced1  QR=1
# questions=1   # answers=3
questions = nytimes.com: type A
nytimes.com  type A  addr 151.101.65.164
nytimes.com  type A  addr 151.101.1.164
nytimes.com  type A  addr 151.101.129.164

Three answers, because nytimes.com has several servers at different addresses — load distribution, visible in the reply.

Getting into DNS in the first place

Suppose you start a company, Network Utopia. You register the name networkutopia.com with a DNS registrar (there was once only one registrar: Network Solutions; now there are many), supplying the names and IP addresses of your primary and secondary authoritative name servers. The registrar inserts an NS record and an A record into the .com TLD servers — the NS record saying networkutopia.com is served by dns1.networkutopia.com, and the A record giving that server's address, 212.212.212.1. You then stand up that authoritative server yourself and put in it an A record for www.networkutopia.com pointing at your web server and an MX record for networkutopia.com pointing at your mail server.

DNS security

DNS is critical and therefore a target. Two families of attack:

Denial of service. Bombard the root servers with traffic — attempted in 2002, when a botnet aimed ICMP pings at all 13 root servers, without success, partly because ICMP was filtered and partly because local servers cache TLD addresses and so bypass the roots anyway. Bombarding TLD servers is more dangerous and has partly worked: in 2016 the Mirai botnet directed DNS query traffic at Dyn, a managed DNS provider based here in New Hampshire, and took down access to Amazon and Twitter among others. DNS can also be turned into a weapon against someone else: send queries with a spoofed source address set to the target, and the replies — larger than the queries, hence amplification — flood the victim.

Redirection. A man-in-the-middle can intercept DNS queries and answer them however it likes. Or, in DNS cache poisoning, an attacker sends bogus replies to a DNS server, which caches them and hands the wrong address to everyone who asks — for as long as the TTL says. Note the pattern from ARP and DHCP repeating: a protocol built for speed with no authentication, where the reply is believed because it arrived.

Email: SMTP, IMAP, and POP3

Email has three main components. User agents are what people use to compose, read, and send mail — Outlook, Apple Mail, Gmail. Mail servers hold a mailbox for each user, receive and store messages for the agents, and send mail on to other mail servers (dealing along the way with problems like nonexistent mailboxes). And SMTP, the Simple Mail Transfer Protocol, is what mail servers use to send messages to each other.

Alice sending mail to Bob, in six steps:

  1. Alice uses her user agent to compose an email message to bob@someschool.edu.
  2. Her user agent sends the email message to her mail server, where the email message is placed in a queue.
  3. The client side of SMTP on her server opens a TCP connection to Bob's mail server.
  4. The SMTP client sends Alice's message over that TCP connection.
  5. Bob's mail server places the message in Bob's mailbox.
  6. Bob invokes his user agent to read it.

Several things about that flow are worth drawing out. SMTP uses TCP — reliably, since nobody wants a message delivered with pieces missing — on port 25. Transfer is direct between servers: the sending server acts as the client and connects to the receiving server, with no intermediate store-and-forward email servers. If the receiving server is down, the sending server simply tries again later. Each transfer has three phases: handshaking (greeting), transfer of messages, and closure. The interaction is command/response, like HTTP: commands are ASCII text, responses are a status code and a phrase. And messages must be in 7-bit ASCII, so anything that isn't text has to be encoded into text first — which is why attachments exist in the form they do.

Reading the RFC as a dictionary

How do you know what to say to a mail server? Think of calling a service line. You dial, they ask how they can help, you identify yourself, say what you want, deliver your message, and hang up. That works because both parties share a language and a rough script. If you speak English and they speak French, you need a dictionary mapping one to the other — "I am Bob" to "Je suis Bob," "How can I help you?" to "Puis-je vous aider?"

Talking to an SMTP server is the same problem, and the RFC is the dictionary. SMTP is defined in RFC 5321 and RFC 5322. Translating our intentions into SMTP:

Connect → port 25
Introduce myself → HELO
Inform of sender → MAIL FROM
Inform of recipient → RCPT TO
Send content of message → DATA
End connection → QUIT

And the server's replies are numeric codes: 220 service ready, 250 OK, 221 closing, 552 or 451 for errors.

Speaking SMTP by hand

Because the protocol is ASCII text over TCP, you can be the mail client yourself using telnet. Against the class's simulated server: 

$ telnet smtp.kiewit.dartmouth.edu 25
Trying 10.28.55.168...
Connected to smtp.kiewit.dartmouth.edu.
220 mailhog.example ESMTP MailHog
HELO test.com
250 Hello test.com
MAIL FROM: <tjp@test.com>
250 Sender tjp@test.com ok
RCPT TO: <you@test.com>
250 Recipient you@test.com ok
DATA
354 End data with <CR><LF>.<CR><LF>
This is line 1
this is another line
I'll end with just a period
.
250 Ok: queued as sQ8bP0_qeuegsw...@mailhog.example
QUIT
221 Bye

Read that exchange closely, because two things should bother you. First, the message body ends with a line containing a single period — that's the delimiter, which is why a line of your message that consists only of a period needs special handling. Second, and more important: nothing verified that the sender was really tjp@test.com. SMTP as originally specified simply takes your word for it, which is the root of email spoofing and much of the spam problem, and the reason for the layers of add-on authentication (SPF, DKIM, DMARC) that real mail systems now use.

(smtp.kiewit.dartmouth.edu is not a real mail server, so your plans for using it as the base of a spam empire are dashed.)

When you write this in code, you can send the same thing as one string, with \r\n (carriage return, newline) ending each line: 

"HELO localhost\r\n\
MAIL FROM:<alice@cs60.com>\r\n\
RCPT TO:<bob@cs60.com>\r\n\
DATA\r\n\
From: Alice <alice@cs60.com>\r\n\
To: Bob <bob@cs60.com>\r\n\
Subject: Test email using netcat\r\n\
\r\n\
Hello cs60 student, this is a test email.\r\n\
.\r\n\
QUIT\r\n"

Note the structure inside DATA: the From:, To:, and Subject: lines are part of the message content, followed by a blank line and then the body. Those headers are what your mail client displays — and they are entirely separate from the MAIL FROM and RCPT TO commands that actually determine where the mail goes. They need not agree, which is worth thinking about next time you look at a suspicious message.

Getting mail back out

SMTP pushes mail to a server; a different protocol is needed to pull it off from the server. So there are three legs to the journey: Alice's user agent sends to her mail server using SMTP (or HTTP, for webmail like Gmail); Alice's server sends to Bob's server using SMTP; and Bob's user agent retrieves messages from his server using IMAP, HTTP, or the older POP3. Short version: SMTP sends, IMAP receives.

IMAP and HTTP leave messages on the server and download copies, while POP3 downloads to one device and erases the server's copy. That difference is why IMAP won: most people now read mail on a laptop and a phone and a tablet, and want the same mailbox on all of them. POP3 is uncommonly used these days.

One design question worth asking: could Alice's user agent just send directly to Bob's user agent, skipping the servers? Theoretically yes, but Bob's agent would have to be online constantly, and you'd have to know which of his devices to use. Mail servers are always on and have stable addresses; laptops and phones are neither.

The web: HTTP

The Internet was mainly a scientific and academic tool until the web browser, which was the killer app that opened it to millions of people. Part of what made it different from broadcast media is that it operates on demand: clients request pages rather than receiving whatever is being transmitted. Clients get those pages using HTTP, the HyperText Transfer Protocol. Note the division of labor: HTTP handles the delivery of a page, not how the browser renders it — that's HTML's job.

A URL (Uniform Resource Locator) identifies an object on the Internet: a page, an image, a file. Dissecting one:

http://tjp.cs.dartmouth.edu/cs60/index.html

http is the Layer 7 protocol; tjp.cs.dartmouth.edu is the host, whose address DNS supplies via an A record; /cs60 is a directory on the server; and index.html is a file. Web pages are just text files written in HTML, and they may reference other objects by URL — a page with 10 images is 11 separate files, potentially stored on different servers.

HTTP follows the client/server model. The client is a browser that requests, receives, and displays objects; the server sends objects in response to requests. Popular web servers include Apache, IIS, and Nginx. The server has a fixed IP address, which the client finds with DNS. Web servers typically listen for connections on port 80.

HTTP uses TCP and is stateless

The client initiates a TCP connection (creating a socket) to the server on port 80; the server accepts; Layer 7 HTTP messages are exchanged; the connection closes. Because TCP is underneath, HTTP doesn't have to worry about missing or out-of-order packets — Layer 4 handles that. This is layering paying off: HTTP can be a simple text protocol precisely because someone else (Layer 4) guarantees the bytes arrive in order.

HTTP is also stateless: the server maintains no information about past client requests. Every request is independent. This is a deliberate simplification, and the reason is that stateful protocols are hard. If the server tracks the state of a multi-step transaction and then the client or the connection dies partway through, the two sides' views of the state disagree and someone has to reconcile them. Statelessness means there is nothing to reconcile.

Non-persistent and persistent connections

Early HTTP was non-persistent: open a TCP connection, send at most one object, close the connection. For a page with text and 10 images, that means eleven separate connections. Walking through one of them: the client initiates a TCP connection to port 80; the server accepts and notifies the client; the client sends a request naming the object; the server forms a response containing the object and sends it; the server closes the connection; the client receives and displays the HTML, parses it, and finds 10 referenced JPEGs — and repeats the whole sequence for each image.

The cost is easy to quantify using RTT, the time for a small packet to travel to the server and back. Per object you pay one RTT to set up the TCP connection, one RTT for the request and the first bytes of the response, plus the time to actually transmit the file:

non-persistent HTTP response time = 2 × RTT + file transmission time

Two RTTs per object, plus OS overhead for each TCP connection. Browsers partially worked around this by opening multiple parallel TCP connections to fetch objects at once.

Persistent HTTP, introduced in HTTP/1.1, leaves the connection open after sending a response. Subsequent messages between the same client and server reuse it, and the client can send a request as soon as it encounters a referenced object rather than waiting. In the best case that's as little as one RTT for all the referenced objects, cutting response time roughly in half.

Request and response messages

HTTP has two message types, a request and a response. Both are ASCII and human-readable. The general format is a request line (method, space, URL, space, version, then carriage return and line feed), then header lines of the form field name: value, then a carriage return and line feed on a line by itself to mark the end of the headers, then an optional entity body. That bare blank line is the delimiter that tells the server the headers are finished — the same structural trick as the lone period in SMTP.

HTTP has several methods, or "verbs":

A GET request example: 

GET /index.html HTTP/1.1\r\n
Host: vibrantcloud.org\r\n
<a browser may include other header info here>
\r\n

A response starts with a status line giving the protocol version, a status code, and a status phrase, then headers, then the data: 

HTTP/1.1 200 OK\r\n
Date: Wed, 23 Sep 2026 20:09:20 GMT\r\n
Server: Apache/2.0.52 (CentOS)\r\n
Last-Modified: Wed, 10 Sep 2026 17:00:02 GMT\r\n
ETag: "17dc6-a5c-bf716880"\r\n
Accept-Ranges: bytes\r\n
Content-Length: 2652\r\n
Keep-Alive: timeout=10, max=100\r\n
Connection: Keep-Alive\r\n
Content-Type: text/html; charset=ISO-8859-1\r\n
\r\n
data data data data data ...

Status codes you'll meet: 200 OK (request succeeded, object follows); 301 Moved Permanently (new location given in the Location: header); 400 Bad Request (server didn't understand the request); 404 Not Found; and 505 HTTP Version Not Supported.

Speaking HTTP by hand

As with SMTP, you can be the client. Open a TCP connection to port 80 on a server, then type a minimal but complete GET request — note the blank line at the end, so hit return twice: 

$ telnet vibrantcloud.org 80
Trying 173.201.179.249...
Connected to vibrantcloud.org.
Escape character is '^]'.
GET /index.html HTTP/1.1
Host: vibrantcloud.org

Telnet is a network protocol that lets you connect to and control a remote computer or server over a network using a text-based command-line interface. Above we connect to a site called vibrantcloud.org, then we issue an HTTP GET request for a web page called index.html on a server named vibrantcloud.org. This request is formatted according to the HTTP spec. The spec says:

The server responds with the following: 

HTTP/1.1 200 OK
Date: Wed, 23 Sep 2026 19:51:33 GMT
Server: Apache
Upgrade: h2,h2c
Connection: Upgrade
Last-Modified: Fri, 16 Apr 2021 15:17:30 GMT
ETag: "1b210a5-137-5c0187a783680"
Accept-Ranges: bytes
Content-Length: 311
Vary: Accept-Encoding
Content-Type: text/html

<!doctype html>

<html lang="en">
<head>
  <meta charset="utf-8">

  <title>VibrantCloud</title>
  <meta name="description" content="VibrantCloud">
  <meta name="author" content="Anon">

</head>

<body>
  <h1>VibrantCloud</h1>
  <p>This is the famous VibrantCloud you've heard so much about.</p>
</body>
</html>

The server's response comes back, headers followed by the HTML of the page. Do this on the Linux VM; macOS's telnet sends only the first line. You can also watch the same exchange in Wireshark.

Conditional GET and caching

If your browser already has a copy of an object, re-downloading it wastes time and bandwidth. A conditional GET avoids that. The client includes the date of its cached copy in the request:

If-modified-since: <date>

If the object hasn't changed since that date, the server replies 304 Not Modified with no object attached at all — no transmission delay, no network resources used. If it has changed, the server replies 200 OK with the new data.

On a larger scale, web caches (proxy servers) do the same thing for a whole organization. The browser is configured to send its requests to the cache; if the cache has the object it returns it directly, and if not it fetches it from the origin server, stores a copy, and passes it along. This reduces load on the origin server and on the organization's access link — which is often the real bottleneck, and much cheaper to relieve with a cache than by buying a faster link. This is the same idea as the CDNs from Day 1, applied at the other end of the path.

Cookies

HTTP is stateless, but plenty of applications need to remember things: a shopping cart, a login, a set of preferences. Cookies add state without making the protocol stateful, and they have four components: a cookie header line in the HTTP response; a cookie header line in the next HTTP request; a cookie file kept on the user's machine and managed by the browser; and a back-end database at the website.

The mechanism: on a first visit, the site creates a unique ID and a database entry for it, and returns the ID in a set-cookie: header. The browser stores it and includes it in every subsequent request to that site, letting the site recognize the returning visitor. Note where the state lives — the HTTP messages carry it, so the protocol itself stays stateless.

Cookies are used for authorization, shopping carts, recommendations, and session state. They also permit a site to learn a great deal about your behavior on that site, and third-party persistent cookies allow a common identifier to be tracked across many different sites, which is the machinery of behavioral advertising. Europe's GDPR treats a cookie identifier that can identify an individual as personal data, subject to data-protection rules, which is why you now click through consent banners.

HTTP/1.1 to HTTP/3

Everything since HTTP/1.1 has been chasing the same goal: less delay when fetching many objects.

HTTP/1.1 introduced pipelined GETs over a single TCP connection, with the server responding in order, first come first served. That ordering creates head-of-line (HOL) blocking: request one large object and three small ones, and the three small objects wait behind the large one even though they could have been delivered almost immediately. Loss recovery makes it worse, since retransmitting a lost TCP segment stalls everything behind it.

HTTP/2 (RFC 7540, 2015) keeps the methods, status codes, and most headers unchanged but gives the server flexibility in how it sends objects: transmission order can follow client-specified priorities rather than strict FCFS (first come, first served), the server can push objects the client hasn't requested yet, and objects are divided into frames whose transmission can be interleaved. With interleaving, the three small objects arrive quickly and the large one is only slightly delayed.

But HTTP/2 still runs over a single TCP connection, so recovery from packet loss still stalls all object transmissions, browsers still have an incentive to open parallel connections, and vanilla TCP provides no security. HTTP/3 addresses all three by moving to UDP and adding security plus per-object error and congestion control.

Underneath HTTP/3 is QUIC (Quick UDP Internet Connections), a Google-developed application-layer protocol built on UDP and deployed widely on Google's servers and apps. QUIC reimplements, at the application layer, much of what TCP does — its own specification notes that readers who know TCP's loss detection and congestion control will find familiar algorithms. The layering is worth pausing on: HTTP/2 over TCP puts TLS between the application and transport; HTTP/3 puts QUIC there instead, over UDP, over IP.

The headline benefit is connection setup. TCP-plus-TLS requires two serial handshakes, one for the transport connection and one for security. QUIC establishes reliability, congestion control, authentication, encryption, and connection state in a single handshake, one RTT. Better still, a client that cached a TLS session ticket from a previous QUIC connection can reuse it to encrypt and authenticate a new one, giving 0-RTT connection establishment — data in the very first packet. Given that we know RTT is bounded below by the speed of light, eliminating round trips is the only way left to make things faster. More on QUIC when we reach the transport layer.

Putting it all together

The Wireshark capture vibrantcloud_dns_tcp_http.pcapng shows the entire process of a web browser fetching the web page index.html on a server called vibrantcloud.org. I turned off TLS encryption so you can see each step:

Exercises

To do before/around the next class:

Terms

Not a comprehensive reference — the textbook has much more — but these are the terms from Day 4 that will keep coming back.

process
A program running within a host. The client process initiates communication; the server process waits to be contacted. A process is identified on the network by an IP address and a port.
application-layer protocol
A definition of the message types exchanged, their syntax (fields and delimiters), and their semantics, plus the rules for when to send and how to respond.
RFC (Request for Comments)
The published specification of an open protocol. SMTP is RFC 5321 and 5322; HTTP is RFC 7230 and 9110; DHCP is RFC 2131. Reading the RFC is how you learn what to say to a server.
open vs. proprietary protocol
Open protocols are publicly specified, allowing independent implementations to interoperate; proprietary ones are closed to outsiders.
elastic application
One that makes use of whatever throughput it gets, as opposed to applications needing a minimum rate to function.
flow control / congestion control
TCP services that keep a sender from overwhelming a receiver, and throttle senders when the network is overloaded, respectively.
TLS (Transport Layer Security)
Provides encryption, data integrity, and endpoint authentication over TCP. Implemented in the application layer via libraries, despite the name.
CIDR (Classless Interdomain Routing)
Notation and scheme, from 1993, allowing the network/host boundary in a 32-bit address to fall anywhere. a.b.c.d/n means the leftmost n bits are the network ID.
subnet mask
The /n portion of a CIDR address, specifying how many leading bits identify the network.
private (non-routable) address
Ranges reserved for internal use — 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 — which cannot be routed on the public Internet and so may be reused in every home and office.
loopback address
127.0.0.0/8, commonly 127.0.0.1 (localhost). Traffic sent here never leaves the host.
NAT (Network Address Translation)
A router service letting many hosts on private addresses share one public address, by recording which internal host and port each outbound conversation belongs to and rewriting addresses in both directions.
DNS (Domain Name System)
The distributed database that maps hostnames to IP addresses. An application-layer protocol over UDP port 53, chosen for speed.
root / TLD / authoritative / local name server
Root servers (13 logical, ~2,000 deployed) point to TLD servers; TLD servers (.com, .edu, country codes) point to authoritative servers; authoritative servers hold an organization's definitive records; local (default) name servers receive host queries, cache results, and forward into the hierarchy.
iterated vs. recursive query
In an iterated query each contacted server returns a referral and the local server does the work. In a recursive query the contacted server resolves the name on the asker's behalf. Applications normally make recursive queries to their local server, which iterates.
TTL (Time To Live), DNS
How long a cached DNS record stays valid, set by the administrator; minutes to hours, commonly 86,400 seconds. Why an address change takes time to propagate.
A / AAAA / MX / NS / CNAME records
A maps a hostname to an IPv4 address (AAAA for IPv6); MX names a domain's mail servers with priorities; NS names its authoritative name servers; CNAME maps an alias to a canonical name.
canonical name
A host's real name, as opposed to the aliases that point to it.
round robin
Handing out several addresses (or listing several servers) in rotation across successive queries, to distribute load.
nslookup / dig / whois
Command-line tools for querying DNS records by type, for querying DNS with more power and detail (including TTLs), and for querying domain registration data respectively.
DNS cache poisoning
Sending bogus replies to a DNS server so it caches and serves a wrong address until the TTL expires. Possible because DNS replies are unauthenticated.
user agent
The mail client a person uses to compose, send, and read mail.
SMTP (Simple Mail Transfer Protocol)
Moves mail between servers over TCP port 25, directly from sender to receiver with no intermediate hops. ASCII command/response: HELO, MAIL FROM, RCPT TO, DATA, QUIT, with numeric replies (220, 250, 221). Message bodies must be 7-bit ASCII and end with a line containing a single period.
IMAP / POP3
Protocols for retrieving mail from a server. IMAP (and HTTP webmail) leaves messages on the server and downloads copies, suiting multiple devices; POP3 downloads to one device and deletes the server copy.
HTTP (HyperText Transfer Protocol)
The web's application-layer protocol, over TCP port 80. Handles delivery of web objects, not their rendering.
URL (Uniform Resource Locator)
An identifier for an object on the Internet, naming a protocol, host, directory, and file.
stateless
A protocol in which the server keeps no record of past requests, so every request is independent. Chosen for HTTP because reconciling inconsistent state after a crash is hard.
non-persistent vs. persistent HTTP
Non-persistent uses one TCP connection per object, costing 2 × RTT plus transmission time each. Persistent (HTTP/1.1) keeps the connection open for multiple objects, as little as one RTT for all of them.
GET / POST / PUT / DELETE
HTTP methods (verbs): retrieve data (parameters in the URL query string), send form input in the entity body, upload an object replacing what's at a URL, and delete a resource.
status codes
200 OK, 301 Moved Permanently, 304 Not Modified, 400 Bad Request, 404 Not Found, 505 HTTP Version Not Supported.
conditional GET
A request carrying If-modified-since: so the server returns 304 Not Modified, with no object, when the client's cached copy is current.
web cache (proxy server)
A server that satisfies client requests from its own store when it can, reducing load on origin servers and on the organization's access link.
cookie
A unique ID a site returns in a set-cookie: header, stored by the browser and returned in later requests, letting a stateless protocol support state. Third-party persistent cookies allow tracking across sites; identifying cookies are personal data under GDPR.
head-of-line (HOL) blocking
Delay caused when a small object waits behind a large one because responses are sent in request order. Mitigated in HTTP/2 by dividing objects into interleaved frames.
QUIC / HTTP/3
QUIC is an application-layer protocol over UDP implementing reliability, congestion control, authentication, and encryption in a single 1-RTT handshake (0-RTT when reusing a cached session ticket). HTTP/3 is HTTP over QUIC.

Credit: Most information on this page is from Computer Networking: A Top-Down Approach, 9th edition by Kurose and Ross.
This page was generated by Claude from in-class slides.