How Docker manages its image layers using Layered FileSystem

Docker Layered Filesystem

Why do we complicate the simple things so much and then find it super difficult to understand and use in the real tech world?

Lets bake a pan cake and understand one of the most important concepts in Docker and it is related to its file system called Docker Layered File System.

What is Docker?

By now, I am sure, we all know what is docker, if still it needs more understanding, then here we go.

Imagine you are baking a cake. You need a recipe (lets call it a Dockerfile) that tells you step by step what to do. Docker is like your super smart kitchen helper that follows the recipe and builds the cake for you. But instead of a cake, Docker builds something called a container, which is like a tiny, self contained computer that runs an app.

?

What’s a Layered Filesystem?

Now, that’s the real thing, in our Linux host we deal with file system like NTFS, XFS, EXT4 etc, but do we use the same layer of file system in docker as well? lets talk about the layered filesystem. Think of it like a stack of pancakes. Each pancake is a layer, and when you stack them up, you get a delicious stack of pancakes. Similarly, Docker uses layers to build its containers. Each layer is like a step in the recipe, and when you stack them up, you get a complete container.

Example: Building a Cake (or a Container.)

Lets say your recipe (Dockerfile) looks like this:

1. Start with a plate (this is like the base layer, FROM ubuntu).
2. Add a pancake (this is like installing something, RUN apt update).
3. Add some syrup (this is like adding more stuff, RUN apt install nginx).
4. Top it with whipped cream (this is like the final touch, CMD ["nginx", "-g", "daemon off;"]).

Btw, in Dockerfile the 1st part is called the Instruction and the 2nd part is called as the Arguments to that instruction.

Here, each step adds a new layer to your stack. Docker does the same thing when building a container and so thease each layers are stored as a separate file or folder /var/lib/docker/overlay2

Or it can be other storage driver, get to know about storage driver from “docker info” command.

?

Why Use Layers?

Now, the question is, why do we need to use Layered File System and what benefit does it provide.

Layers are super cool because they save time and space. Let me explain:

1. Its Reusable

Imagine you have a stack of pancakes, and you want to make another stack. Instead of making all the pancakes from scratch, you can reuse the ones you already have. Docker does the same thing. If two containers need the same base layer (like ubuntu), Docker reuses it instead of making a new one. This saves a lot of time and space.

2. Its Read Only

Once a pancake is made, you can’t change it. It’s read only. If you want to add more syrup or whipped cream, you just add a new pancake on top. Docker works the same way. The layers in an image are read only, so if a container needs to change something, it adds a new writable layer on top.

What’s a Writable Layer?

But, if docker each layer is only Read Only, the how can we use that for container, as in the container there are configuration files, log files and many other files, which needs to have write access to do some modification. Lets get the concept here as the container adds a writable layer above the read only layer of docker image and that’s how we get the container super flexible to run an application on that.

Copy-on-Write (CoW)

Lets say we need a layer file to be used to write for a specific container, For that, we make a copy of that layer and then assign that to that container(process) and it can be used to do any modification on that layer without disturbing the original layer, in that case, we call that layer as a Copy on Write.

?

How Does Docker Store Layers?

Docker stores layers as special files and folders on host(Linux). It uses something called a storage driver (overlay2) to manage these layers. Think of the storage driver as a librarian who organizes all the books (layers) in a library (your computer).

• The librarian knows where each book is stored.
• When you need a book, the librarian finds it and gives it to you.
• If you want to write in a book, the librarian gives you a copy instead of the original.

Now, we know that each Read Only layers are an individual files/folders as mentioned above, so they should have a specific size and to see those layers size, we can use

docker history <image_id>

Lets understand all agin with an example of Building a LEGO Castle

Lets use LEGO blocks to explain Docker’s layered filesystem:

1. Base Plate: This is like the base layer (FROM ubuntu). Its the foundation of the LEGO castle.
2. First Layer: Add walls and doors (RUN apt update). This is the second layer.
3. Second Layer: Add windows and a roof (RUN apt install nginx). This is the third layer.
4. Final Touch:Add flags and decorations (CMD ["nginx", "-g", "daemon off;"]). This is the top layer.

Now, we have a complete LEGO castle (Docker image). If we want to play with the castle (run a container), we can add a new layer of LEGO blocks (the writable layer) on the top of that caste to make any changes without breaking the original castle.

Why Is This Cool?

1. Efficiency: Docker reuses layers, so it does not have to build everything from scratch every time. This saves time and space.
2. Isolation: Each container has its own writable layer, so changes in one container dont affect others. Its like having separate LEGO sets for different castles.
3. Consistency: Since the layers are read only, the base image stays the same no matter how many containers you run. This makes apps run reliably.

?

If this makes it easy to understand, write your comments and I will break down complex concepts like this to make it easy to understand, which would ease the work with this technology, and if any issue comes we know exactly where to look for the issue.