How the Internet Works

If you’re new to Internet technologies and want a comprehensive understanding from just one read, this article is for you. Here, you’ll delve into the four layers of the TCP/IP model, understand what MAC and IP addresses are, and why we need two types of digital addresses. You’ll learn how DNS works, why switches and routers are essential, how NAT operates, how secure connections are established, what public key infrastructure is, why TLS certificates are crucial, the differences between the three versions of the HTTP protocol, how HTTP authentication functions, and finally, a bit about VPNs.

4 Layers of the TCP/IP Model

When you connect multiple computers to a switch, you create a local network. This is a simple, flat-structured network where complex routing algorithms aren’t necessary. At this level, the data link (the fourth and lowest layer) of the TCP/IP model, also known as the Network Access Layer, operates. This layer handles physical data delivery. For your computer to send data to another device on an Ethernet or Wi-Fi network, it must know the MAC address of the destination device. It forms a message, like an Ethernet frame, includes the necessary data, specifies the recipient’s MAC address, and sends it through an Ethernet cable. A MAC address is a unique identifier of a network device, composed of six octets, for example, 00-50-B6-5B-CA-6A.

The Internet is a vast and intricate structure of numerous local networks. The method of transmitting data by MAC address isn’t scalable for such a large network, hence the need for routing. The third layer of the TCP/IP model, the Internetwork or Internet layer, is responsible for this. Here, data is encapsulated in IP packets, with each packet containing the IP address of its intended destination. To send an IP packet to a recipient outside your local network, you must pass it through a chain of devices. Physical delivery of data occurs only at the data link layer, so the process involves alternating between the data link and network layers until the packet reaches its destination. At each intermediate device, the decision is made based on the IP address to determine the next hop for the packet, aiming for an optimal route.

Next is the transport layer, where protocols like TCP, UDP, TLS, and QUIC operate. These protocols facilitate data exchange between applications or, more precisely, between ports. For instance, an application may request the operating system to allocate TCP port 80. If granted, the application can listen to this port. Upon receiving a TCP packet with port 80, the OS notifies the application that new data is available on this port, which the application can then process. With UDP, the process differs as it has a separate socket table. When working with TCP, programmers don’t need to worry about breaking data into TCP packets, IP packets, or Ethernet frames, nor about the packets taking different routes or getting lost. A TCP connection between two applications on different devices ensures that an array of bytes loaded into a TCP socket on one device is received intact on another.

IP packets can be lost due to interference, network device buffer overflow, radio channel obstacles, damaged cables, etc. Since it’s not their responsibility, network or data link layer protocols don’t address this issue. If necessary, lost packets are retransmitted at the transport layer, such as with TCP, a reliable data transmission protocol. TCP also reorders data, as IP packets may be sent in one order and received in another. UDP, on the other hand, doesn’t solve these problems, but it is faster.

TLS operates on top of TCP, ensuring reliable data delivery and providing encryption to establish a secure communication channel. Previously, the SSL protocol was used for this purpose, but it’s now outdated. Finally, the application layer, the highest level of the TCP/IP model, includes well-known protocols like HTTP, HTTPS, FTP (for file transfers), SMTP (for email), DNS, and more. This layer isn’t about data delivery but about structuring and parsing it. While TCP connections ensure the secure transmission of data, protocols like HTTP define how data should be structured and exchanged, enabling communication between clients and servers worldwide using standardized message formats.

Connecting to the Network

An Ethernet cable for your computer is an interface that can receive or send a structured sequence of bytes, equipped with the recipient’s MAC address, to its destination. When you connect an Ethernet cable, your computer recognizes the potential to go online but needs an IP address, prompting it to send a DHCP request over the Ethernet connection. This request includes the computer’s MAC address, asking for an IP address.

Earlier, we discussed how MAC addresses are used for physical data delivery on a local network. So, to which MAC address is the DHCP request sent? Since the computer is unaware of other devices on the network, it sends a broadcast request to the address FF-FF-FF-FF-FF-FF-FF. This request is received by all devices in the broadcast domain (the local network). Devices without a DHCP server discard the request, while those with a DHCP server, like a home router, respond with an IP address assignment.

The DHCP server’s role is to maintain unique IP addresses within its subnet. After receiving a DHCP response, the computer confirms its acceptance of the IP address, and the DHCP server provides final confirmation. This additional exchange is necessary if multiple DHCP servers are on the network, allowing the computer to select one IP address and reject the others. The initial DHCP response also includes the default gateway’s IP address, one or more DNS servers, and the subnet mask.

The default gateway is the device that allows you to connect to the Internet from your local network. Typically, this is your router at home. When your computer needs to send an IP packet to a device outside your local network, it sends the packet to the default gateway, which routes it accordingly. The subnet mask helps determine whether an IP address belongs to the same local network. If not, the packet is sent to the default gateway.

DNS servers translate domain names into IP addresses. When accessing a device on the Internet, you usually use a domain name, not an IP address. The DNS system converts the domain name to an IP address for routing. When entering a URL in your browser, it checks its cache for the domain name. If not found, it makes a system call, and the OS handles the DNS request, searching its cache before contacting the DNS server.

Returning to the Ethernet-connected computer with an empty cache, let’s say it wants to access example.com. After making a DHCP request and obtaining the DNS server’s IP address, it can now request the IP address for example.com from the DNS server, caching the result for future use.

For the request, a MAC address is required for physical delivery, which happens at the data link layer. The computer uses the IP address of the server or the default gateway’s MAC address to send the request. However, since the DHCP server’s response didn’t include any MAC addresses, the ARP protocol helps by broadcasting another request: “The device with IP address such-and-such and MAC address such-and-such asks the device with this IP address to provide its MAC address.” This broadcast request is received by all devices on the network. Devices that don’t match the IP address discard the request, but they may cache the sender’s information for future communication. The correct device responds with its MAC address, which your computer then uses.

DNS

If the DNS server doesn’t know the required address, it initiates a recursive search through upstream DNS servers until the record is found. The root DNS server, for instance, returns the DNS servers of domain zones (.ru, .com, etc.) rather than the requested IP address. This hierarchical system ensures that DNS requests are efficiently resolved without overwhelming any single server.

To balance server load across different regions, service owners can use GeoDNS, where a DNS provider can return different IP addresses based on the user’s region. For example, a North American user accessing example.com might receive the IP address 192.0.2.1, while a European user might receive 198.51.100.2, ensuring faster response times by directing users to the nearest server.

Another load-balancing method is using a CDN (Content Delivery Network). A CDN caches data on servers in various regions, reducing latency by delivering content from the closest server. CNAME records play a crucial role in this process, allowing domain aliases to point to different servers in different regions. This setup means that a user in Europe might be directed to a different server than a user in Asia, ensuring efficient content delivery without requiring the company to maintain servers in every region. The CDN provider manages caching and server maintenance, streamlining content distribution.

URL

To access a specific resource on the Internet, you don’t just use a domain name or IP address but a URL. A URL’s structure is as follows:

scheme://username:password@host:port/path?query#fragment        

The scheme specifies the protocol (e.g., http, https, ftp, mailto) used to make the request. The URL structure varies by protocol, but for HTTP, it follows this pattern. The host is usually a domain name, but an IP address can also be used. After the domain name, a path typically follows, sometimes preceded by a port number (e.g., :8080). The path may be followed by a fragment, which indicates a specific section of a web page, and query parameters (key=value pairs separated by &).

When accessing a resource via a browser, the port is usually omitted, and the browser defaults to port 80 for HTTP or 443 for HTTPS. The host and port number are used to establish a TCP connection, while the host name is included in the headers, and the path and query parameters are specified in the request’s start line. The host name is essential because multiple sites can share the same IP address, and the server needs to know which site to present to the client.

The fragment, however, is client-side information that the browser uses to display a specific part of the web page.

Switches and Routers

Switches and routers are both essential for network communication, but they operate differently.

A switch is a device that receives Ethernet frames and forwards them to the appropriate device. It uses its MAC address table to determine where to send each frame. If the switch doesn’t know the destination MAC address, it sends the frame to all devices (broadcasting) and updates its MAC address table when the correct device responds.

In contrast, a router forwards IP packets from one network to another. It uses a routing table to determine the best path for each packet. Routers can connect different types of networks, like Ethernet and Wi-Fi, and handle traffic between local networks and the Internet.

Switches operate at the data link layer (layer 2), while routers operate at the network layer (layer 3). This means switches use MAC addresses to forward data, while routers use IP addresses.

When your computer sends data over a network, the data is encapsulated in layers: the application layer (e.g., HTTP), transport layer (e.g., TCP), network layer (IP), and data link layer (Ethernet). Each layer adds its header information, with the Ethernet frame at the outermost layer. The frame is sent to a switch, which forwards it to the correct device based on the MAC address.

If the data needs to be sent to a different network, the switch sends the frame to a router. The router removes the Ethernet header, processes the IP packet, and forwards it to the next hop, adding a new Ethernet header for the next network. This process repeats until the data reaches its destination.

Transport Protocols

Transport layer protocols like TCP, UDP, and others are crucial for directing packets from the sender’s application port to the recipient’s. For example, an HTTP request is formed at the application layer, then passed to the transport layer where a TCP or UDP header is added. This header includes information about ports, creating what is known as a datagram or segment. The segment is then handed off to the internetwork layer, where an IP header is attached. This new combination is called a packet, which is subsequently passed to the data link layer where an Ethernet frame header is added, in case of an Ethernet connection.

This frame travels over the network, and at each node, the IP packet is extracted, the next destination is determined, and a new frame is formed and sent on. Throughout the process, protocols from all levels of the TCP/IP stack are engaged at the start and end of the transmission, while intermediate nodes mainly interact with the data link and internetwork layers.

A UDP packet header, for example, is composed of 8 bytes, including the sender port, recipient port, datagram length, and checksum. The checksum detects packet damage during transmission, discarding any compromised packets. UDP limits the message size to slightly under 64 kilobytes, though practical limits are often much lower depending on network restrictions. UDP’s simplicity offers only the basic functionalities expected of a transport protocol.

TCP packets are more complex, including headers with sender and recipient ports, sequence numbers, acknowledgment numbers, header length, 9 flags, window size, checksum, and additional options. These features enable TCP to provide a reliable communication channel with mechanisms for resending lost packets and restoring their order. While UDP allows for simple message sending and receiving, TCP involves establishing a connection, exchanging data, and then closing the connection. This process begins with a handshake, where the first device sends a packet with a SYN flag, the second responds with SYN-ACK, and the first device confirms with an ACK. The connection is then established, with each packet containing a sequence number to track data integrity and order.

Most internet traffic uses TCP due to its reliability, but UDP is favored for tasks like DNS queries where speed and minimal delay are essential. DNS queries, often under 512 bytes, fit comfortably within a single UDP packet. If a packet is lost, the client simply retries after a timeout. If multiple attempts fail, alternative DNS servers may be queried, and if unsuccessful, an error is returned to the user. When DNS queries exceed 512 bytes, TCP may be used.

NAT (Network Address Translation)

You’ve likely seen IP addresses like 192.168.0.1, common in local networks. These are private network addresses defined by the RFC 1918 standard and are not routable on the global internet. This allows different organizations and individuals to use the same addresses within their local networks without risking conflicts on the internet. This system was created to address the shortage of IPv4 addresses, as the demand far exceeded what was anticipated in the early days of the internet.

For a device to access the internet, it needs a global IP address, but most devices only have local IP addresses. NAT is used to replace the local IP address with a global one as packets traverse the network, and then replace the global address with the local one on the return trip. This process effectively extends the IP address space by utilizing the port number at the transport level.

While IPv6 was designed to solve the IP address shortage with its 128-bit address space, NAT has allowed organizations to continue using IPv4, delaying the transition to IPv6. NAT employs four translation mechanisms: Static NAT, Static PAT, Dynamic NAT, and Dynamic PAT.

Static NAT translates a specific private address to a predetermined public address, but it doesn’t conserve IPv4 address space, limiting its usefulness. Dynamic NAT, on the other hand, assigns a pool of public IP addresses to devices on the internal network, allowing for translation on demand. If more devices need to access the internet than there are available public addresses, packets from the extra devices are dropped until a connection is freed.

Static PAT translates different private addresses to the same public address but with different port numbers, effectively conserving public IP addresses. Dynamic PAT, the most commonly used type, maps an entire local subnet to a single public IP address, dynamically assigning ports as needed. This allows multiple devices to share one public IP address, with NAT tracking the mappings in a table.

TLS (Transport Layer Security)

TLS, an evolution of the earlier SSL protocol, works on top of TCP, adding security to the transport layer. While SSL is now outdated due to security flaws, TLS is widely used today, often referenced under the name SSL for historical reasons (as seen with tools like OpenSSL).

When establishing a TLS connection, a handshake occurs where the client requests a TLS certificate from the server. To simplify, let’s start with the basics of establishing a secure connection without a certificate, though this method isn’t used in practice.

Encryption algorithms can be symmetric or asymmetric. Symmetric encryption uses a single key for both encryption and decryption. The key length must be chosen carefully to ensure it can’t be cracked in a reasonable time, even by powerful computers. In TLS, symmetric keys are typically 128, 192, or 256 bits long, making them practically unbreakable with current technology.

However, both the client and server need the same key, and transmitting this key over an insecure network isn’t safe. Asymmetric encryption solves this by using a pair of keys: a public key, which can be shared openly, and a private key, kept secret. If you have the server’s public key, you can encrypt a message that only the server can decrypt with its private key. This ensures that only the intended recipient can read the message, even if it’s intercepted.

The process works as follows: the client contacts the server over an open channel, the server sends its public key, and the client generates a session key for symmetric encryption. The client then encrypts this session key with the server’s public key and sends it back. The server decrypts the session key with its private key, and from that point, both can communicate securely using the session key for encryption.

This system relies on the secrecy of the server’s private key. Website owners must protect these keys, as their compromise could expose all communications.

To prevent “Man In The Middle” (MITM) attacks, where an attacker intercepts and manipulates communications between the client and server, TLS uses a chain of trust involving public keys and certificates. Certification Authorities (CAs) issue TLS certificates to website owners, which verify that a given public key belongs to a specific server. Clients trust certificates from CAs whose public keys are pre-installed in their operating systems.

When a client receives a certificate, it verifies the CA’s digital signature using the CA’s public key. If valid, the client can trust that the certificate and corresponding public key belong to the server. The client then uses this key to establish a secure session.

The Diffie-Hellman algorithm is often used during this process, providing forward secrecy. Even if a server’s private key is compromised later, past sessions remain secure because ephemeral keys, used to generate session keys, are deleted after each session.

In cases where a root CA issues certificates to other CAs, which then issue certificates to websites, the entire chain of certificates is sent to the client during the TLS handshake. The security of the connection depends on the integrity of each entity in this chain.

The security landscape is continually evolving, especially with the potential future threat of quantum computers, which could break current cryptographic methods much faster than conventional computers. As a result, organizations are beginning to adopt post-quantum encryption algorithms, designed to be resistant to quantum attacks. While symmetric encryption remains robust, requiring only longer keys, asymmetric encryption is more vulnerable and requires new quantum-resistant algorithms.

HTTP (Hypertext Transfer Protocol)

HTTP requests and responses are plain text, beginning with a start line and headers that contain standard information, followed by a message body with the data to be transmitted. In HTTP/1.1, servers can transmit data in chunks using the Transfer-Encoding: chunked header, which helps manage server memory usage over time. For example, when a user accesses a website, the HTTP response might include an HTML document that structures the web page’s content—text, images, tables, etc. The HTML document includes URLs pointing to these resources, prompting the browser to make additional HTTP requests to load them.

HTTPS, the secure version of HTTP, operates on top of TLS instead of TCP. Versions 2 and 3 of the HTTP protocol were developed to increase data exchange speed. HTTP/2, for example, allows parallel loading of resources in one connection using streams, with headers transmitted in binary form and compressed to eliminate redundant data. The server push function lets servers send resources immediately after a client’s initial request, reducing latency.

HTTP/3 goes further by operating over UDP/QUIC, a protocol developed to enhance speed and reliability. QUIC combines the features of TCP/TLS, like connection reliability and encryption, with faster setup and data transfer. It supports Zero-RTT (zero round-trip time), allowing clients to send requests immediately, and facilitates smoother IP migration during a session.

VPN (Virtual Private Network)

A VPN creates a virtual network that connects devices over the internet as if they were directly connected to the same local network. This connection is facilitated by network tunnels, which provide encrypted communication between devices. This encrypted tunnel ensures that data transmitted over the internet is secure and private, even though it may pass through multiple networks and routers.

WireGuard VPN Example

On Linux, network devices are represented by network interfaces like eth0, eth1 (for Ethernet) or wlan0, wlan1 (for wireless). With VPN protocols like WireGuard:

• Setup:

1. WireGuard creates virtual network interfaces on each end device, often named wg0, wg1, etc.
2. Each device has a unique public and private key pair used for encryption.
3. The devices exchange their public keys and configure their virtual network interfaces to establish a secure tunnel.

• Connection:

1. Data sent through the VPN is encrypted using the devices’ keys before being transmitted over the internet.
2. Upon reaching the other end of the tunnel, the data is decrypted and delivered to the intended destination.

• Advantages:

1. Security: Data is encrypted end-to-end, making it unreadable to anyone who intercepts it.
2. Privacy: Users’ real IP addresses are hidden behind the VPN server’s IP address.
3. Flexibility: VPNs can connect devices across different geographical locations as if they were on the same local network.

• WireGuard Specifics:

1. WireGuard is known for its simplicity and high performance compared to older VPN protocols.
2. It uses modern cryptographic methods and has a small codebase, which enhances its security and efficiency.

VPNs are widely used for secure remote access, privacy protection, and bypassing geographic restrictions.