Android: A Unique Linux Distribution.

As I mentioned two weeks ago, with this article we are embarking on a journey into the world of Android. In this first stage we begin to see the android's architecture.

But I want to start a bit differently from the usual refrain, "here's the layered architecture diagram of Android, and now let's explain the first layer". We've seen that approach many times when opening a book about Android.

This time, let's try something different. In this first stage of our journey, I want to address a fundamental question: "Can Android be considered a true operating system, or is it just another Linux distribution?" Finding the right answer to this question will set our compass in the right direction, making it easier to navigate the next steps of our journey.

When we think of Linux distributions often come to mind names like Ubuntu, Fedora, or Debian. However, Android stands out as a unique member of the Linux family, with features that distinguish it from traditional desktop distributions. At its core, Android is indeed a Linux distribution, but several components, especially the Dalvik (or ART) runtime, make it special.

Android's foundation lies in the Linux kernel. This kernel is responsible for managing hardware resources, providing security, and enabling communication between software and hardware. Android uses a modified version of the Linux kernel, optimized for mobile devices, which includes enhancements such as "Low memory management" tailored for devices with limited RAM, "Power management" essential for battery-operated devices and the "Binder Driver", a low-level kernel component that manages inter-process communication (the next stage of this journey will be entirely focused on the "Binder" mechanism).

So Android is fundamentally based on the Linux kernel. Although it has evolved to meet the unique requirements of mobile devices, many of its core principles and mechanisms are rooted in traditional Linux. One such aspect is the way Linux handles user management and how Android manages applications. While they serve different purposes, there are interesting parallels between the two systems.

Let's analyze them:

Linux is a multi-user operating system, designed to support multiple users simultaneously. There are three elements on which multi-user management in Linux is based: user accounts, file permissions, and security:

• User Accounts: There are three types of users, Root User (The super-user with unrestricted access to all commands and files), Regular Users (accounts created for general use, with specific permissions and limitations) and User Groups (collections of users with shared permissions to streamline access control).
• File Permissions: Linux uses a permission model to control access to files and directories. Each file or directory has three sets of permissions (read, write, execute) assigned to three types of users (owner, group, others). This system ensures that users can only access resources they are authorized to use.
• Isolation and Security: User accounts in Linux are isolated from each other. This means that one user’s processes and files are not accessible to another user without proper permissions. This isolation enhances security and stability.

Although Android is not a multi-user system, it applies similar concepts to manage applications. Each app in Android is treated as a separate user in a Linux system.

Here's how:

• Application Sandboxing: When an app is installed on Android, it is assigned a unique user ID (UID). This UID is used to run the app's processes and to manage permissions.
• Isolation: Just like user accounts in Linux are isolated, apps in Android are sandboxed. This means that each app runs in its own isolated environment, with its data stored in a separate directory. An app cannot access another app's data without explicit permission.
• App-specific Directories: Each app in Android has its own private directory on the file system where it stores its data. This directory is accessible only by the app itself, just like a user's home directory in Linux is private.
• Shared Resources: For shared resources, such as SD card storage, Android uses a permission model. Apps must explicitly request permission to read from or write to these shared areas, similar to how Linux users must have the appropriate permissions to access shared files.
• Permission System: Android uses a permission system that users must grant for apps to access certain features or data. These permissions are similar to the access control lists (ACLs) in Linux.
• SELinux: Android incorporates SELinux (Security-Enhanced Linux) to enforce additional security policies. SELinux in Android provides fine-grained control over the access of apps to system resources, enhancing the security model.

So we can say that although Linux user management and Android app management have different purposes, the basic concepts are surprisingly similar. Both systems rely on isolation, permissions, and security to maintain order and protect resources. Understanding these parallels highlights how Android, although a unique and specialized operating system, remains deeply rooted in the Linux principles, leveraging its robust and proven mechanisms to create a secure and efficient environment for mobile applications.

In summary, Android is a distinctive Linux distribution tailored for mobile devices, combining the robustness of the Linux kernel with a user-friendly interface and a powerful application framework. We will soon see how the Dalvik (and ART) runtime plays a crucial role in this ecosystem, enabling efficient application execution in resource-constrained environments.

This combination of Linux strengths with Android's unique elements has made it the dominant force in the mobile operating system market.

As we explore Android, it will become clear how this "particular" Linux distribution has reshaped our understanding of what a Linux-based operating system can be, highlighting the versatility and adaptability of the Linux ecosystem.

The architecture of Android can be considered as a stack comprising applications, an operating system, a runtime environment, middleware, services, and libraries. This architecture is illustrated in the following diagram, where each layer of the stack corresponds to a specific set of components that work together to perform specific tasks for that layer.

Positioned at the bottom of the stack, there is the Linux kernel that allows access to the hardware, through the appropriate drivers, to the layers above it.

Based on a specific version of the Linux kernel, but with numerous modifications and additions by Google to meet the needs of mobile devices. The differences are due to a set of new features, called "Androidisms", that were originally added to support Android. One of the most important Androidism is the Binder. Binder implements IPC (inter-process-communication) and an associated security mechanism, which we will discuss extensively in the next article.

On top of the kernel is the native layer, consisting of the init process (the first process started, which starts all other processes), several native daemons, and a many native libraries that are used throughout the system. While the presence of an init and daemons is reminiscent of a traditional Linux system, both init and the associated startup scripts have been developed from scratch and are quite different from their Linux counterparts.

In the diagram, above the native layer, we can see the Android Runtime (1.),(2.). As previously mentioned, the multitasking capability of the Linux kernel allows multiple processes to run concurrently. Every application running on an Android device owns its own instance of the Virtual Machine.

Running applications inside a virtual machine offers several advantages. For instance, applications are essentially sandboxed, meaning they cannot directly communicate with the operating system or other applications, nor can they access the hardware directly. Another advantage of this abstraction is that application code is hardware-independent, ensuring that the basic libraries function the same way on any device.

The Dalvik Virtual Machine was developed by Google and relies on the fundamental features of the Linux system. In terms of memory usage, it is much more efficient than commonly used Java virtual machines. Additionally, it was designed so that multiple instances can be launched efficiently without significantly slowing down a mobile device.

To run a Dalvik VM, the application code must be transformed from a Java class to a Dalvik executable (.dex). With Android 5.0 (Lollipop), Google introduced ART as the new runtime environment, replacing Dalvik. ART uses Ahead-Of-Time (AOT) compilation, which compiles the bytecode of an app into native machine code during installation, rather than at runtime. This fundamental shift brought several advantages.

But to understand how these virtual machines work it is useful to see them in action on a simple function, so let's consider the following java function.

public static int add(int i, int j) {
    return i + j;
}        

Let's start to see how the standard Java Virtual Machine (JVM) works:

The Java compiler (javac) compiles the Java source code into bytecode. This bytecode is stored in .class files. The JVM reads the bytecode and executes it. The JVM uses Just-In-Time (JIT) compilation to convert the bytecode into native machine code at runtime. The JIT compiler optimizes the code during execution for better performance.

The JVM executes this bytecode, translating it into machine code for the host machine.

If we use IntelliJ IDEA as an IDE, we can view the bytecode of the java function by selecting View | Show Bytecode from the main menu.

The compiled bytecode for the "add" function might look something like this:

public static add(II)I
   L0
    LINENUMBER 4 L0
    ILOAD 0
    ILOAD 1
    IADD
    IRETURN
   L1        

Here it is the explanation of the produced bytecode:

• ILOAD 0: Load the first argument (i) onto the stack.
• ILOAD 1: Load the second argument (j) onto the stack.
• IADD: Add the two integers on the stack.
• IRETURN: Return the result.

The JVM uses a stack-based architecture, meaning it operates on a stack for its instructions.

Similar to the JVM, the Java compiler compiles the Java source code into bytecode. The Android SDK includes a tool called dx that converts the .class bytecode files into a Dalvik Executable (DEX) file. The DEX file format is optimized for minimal memory footprint and is designed to be suitable for resource-constrained environments like mobile devices.

While the JVM is stack-based, Dalvik uses a register-based architecture. This means that instead of pushing and popping operands onto a stack, Dalvik uses a fixed number of registers to hold intermediate values. This approach reduces the number of instructions required for computation, potentially increasing execution speed.

To see the generated bytecode we open the APK in Android Studio and use the APK Analyzer to view the DEX files and their contents.

.method public static add(II)I
    .registers 3
    .param p0, "i"    # I
    .param p1, "j"    # I
    .annotation system Ldalvik/annotation/MethodParameters;
        accessFlags = {
            0x0,
            0x0
        }
        names = {
            "i",
            "j"
        }
    .end annotation

    .line 6
    add-int v0, p0, p1

    return v0
.end method        

As you can see, there is a single instruction to add the parameters in registers p0 and p1, with the result stored in register v0. Dalvik uses fewer instructions, but the resulting code is larger compared to code from a stack-based virtual machine. In most architectures, loading the code is less expensive than dispatching instructions, so register-based VMs can be interpreted more efficiently.

Another optimization was made within the dx tool. I remember you that the Java source files are converted to Java class files by the Java compiler. Then the dx tool converts Java class files into a .dex (Dalvik executable) file. All class files of the application are placed in this .dex file. During this conversion process, redundant information in the class files is optimized in the .dex file. For example, if the same java.lang.String is found in different class files, the .dex file contains only one reference of this. These .dex files are therefore much smaller in size than the corresponding class files. The ".dex" file and the resources of an Android project, e.g., the images and XML files, are packed into an ".apk" (Android Package) file.

Moreover the compiler has been modified to become a JIT compilation (Just In Time), a type of dynamic compilation. In a JIT environment, the first phase involves compiling the bytecode, transforming the source code into bytecode. Subsequently, the bytecode is installed on the target system. When the bytecode is executed, the runtime environment's compiler translates it into native machine code.

All heavy operations are done during the building phase, resulting in a longer compilation time. During the post-bytecode compilation phase (during the transformation into native code), all time-consuming operations, such as syntactic and semantic analysis of the source code and initial optimization, are performed, reducing subsequent execution times. The generated native code is saved for future executions, thus improving performance with subsequent startups.

In most devices, the system libraries and applications do not directly contain the device-independent DEX code but rather an optimized platform-dependent code called .ODEX. This optimization process is executed during the application installation phase. The .ODEX file represents the optimized part of an application, allowing for a faster startup.

An ODEX file is created using a software tool called dex2opt, which performs optimization by slimming down the .dex file: virtual methods are replaced with optimized versions containing the method's virtual table index, avoiding the need to search the Virtual Table during execution. Additionally, empty methods are removed, among other optimizations. The dex2opt software also uses indexing to bypass initial symbolic resolution and allow for rapid execution.

Below we can see the compilation and execution scheme of the Dalvik Virtual Machine.

Among the various optimizations implemented to improve Dalvik virtual machine performance, one is achieved with the zygote process. Zygote is a daemon process initialized by the operating system during startup as a system service, designed to create Dalvik instances. This service also opens a socket /dev/socket/zygote to listen for startup requests from various applications. Each time a request arrives on the Zygote socket, it performs a fork() to create a new Dalvik instance with the minimum necessary resources for an application. To optimize this startup when Zygote performs a fork, it does not actually copy the memory but shares it. This makes application processes lighter.

We will discuss Zygote in detail when we cover the Android startup process in one of the upcoming articles.

Another topic worth mentioning is Dalvik garbage collection mode. A garbage collector (GC) is an automatic tool for managing unused resources, such as memory blocks that have been allocated and used but are no longer referenced by active processes. The garbage collector allows the programmer not to worry about resource allocation and deallocation.

Within Dalvik, the memory management process by the Garbage collector includes two pauses during which the application execution is suspended. In the first pause, memory blocks that can be freed are enumerated, and in the second, they are marked. When the system starts to run low on memory, the GC is called.

The Android Runtime (ART) replaced the earlier Dalvik virtual machine to provide a more efficient and powerful execution environment for Android applications. ART became the default runtime from Android 5.0 Lollipop onwards.

Unlike Dalvik, which primarily used Just-In-Time (JIT) compilation, ART uses Ahead-Of-Time (AOT) compilation, translating bytecode into native machine code upon installation of the app. This significant shift in approach results in improved performance, reduced startup times, and better battery life.

When an Android application is installed, ART compiles the app's bytecode into native machine code for the target device. This process is done during installation, which means that the app is fully compiled and optimized before it is executed. This eliminates the need for JIT compilation during runtime, thus reducing execution overhead.

ART compiles apps using a tool called dex2oat. This utility takes DEX files as input and generates a compiled app that can be run on the target device

Below we can see the compilation and execution scheme of the ART Virtual Machine.

ART optimizes memory allocation and management. By pre-compiling bytecode into native code, ART reduces the memory overhead associated with JIT compilation. This leads to more efficient memory usage, which is especially beneficial for devices with limited resources.

ART also introduces a more efficient garbage collector (GC) that reduces pause times during memory management, improving the overall responsiveness of applications.

To summarize, after this analysis about JVM, Dalvik and ART we can say that while JVM, Dalvik, and ART all serve the purpose of executing Java code, their underlying architectures and execution methodologies differ significantly, each tailored to their respective environments. When we build our app and generate APK, part of that APK are?.dex files. Those files contain the source code of our app including all libraries that we used in low-level code designed for a software interpreter the bytecode. When a user runs our app the bytecode written in?.dex files is translated by Android Runtime into the machine code a set of instructions, that can be directly understood by the machine and is processed by the CPU. The compilation of the bytecode into machine code can be done using various strategies and all those strategies have their trade-offs. And to understand how Android Runtime works today we went back in time several years.

Here is a bit of Android history, which I have been living since the beginning.

In the early days of Android smartphones were not so powerful as they are now. Most phones at a time have very little RAM. So the first Android Runtime, known as Dalvik,?was implemented to optimize exactly this one parameter: RAM usage. Therefore instead of compiling the whole app to machine code before running it, it used the strategy called?Just In Time compilation. In this strategy, the compiler works a little bit as an interpreter. It compiles small pieces of code while the app is running. And because Dalvik only compiles the code it needs and does so at runtime, it saves a lot of RAM. But this strategy has a serious drawback: because all this happens at runtime, it obviously has a negative impact on runtime performance. Meanwhile, mobile phones have become more and more powerful and have more and more RAM. And since also apps were getting bigger the JIT performance impact was becoming more of a problem. That's the reason why Android Lollipop introduced ART a new Android Runtime. ART instead of using Just in Time compilation as was done in Dalvik used a strategy called Ahead of Time compilation. In ART instead of interpreting the code at runtime the code was compiled before running the app and when the app was running the machine code was already prepared. This approach greatly improved runtime performance since running native machine code is much faster than just in time compilation.

I would like to dedicate the last part of this article to a brief description of a unusual use of DEX files. We can call it "Dynamic Loading of DEX Files in Android". This approach, which I have used on a few occasions, simulates dynamic loading of plugins even in a Java-Kotlin environment,

As we saw before a DEX file is a compiled version of Java bytecode that has been transformed specifically for the Dalvik Virtual Machine (DVM) and, subsequently, the Android Runtime (ART). When developers write applications in Java, the code is first compiled into standard Java bytecode (.class files). The Android SDK's dx tool then converts these .class files into a single .dex file. This .dex file contains all the necessary bytecode for the application to run on an Android device.

Dynamic loading of DEX files in Android allows us to load classes at runtime, similar to how dynamic-link libraries (DLLs) in Windows or shared objects (SO) in Linux are used. This capability can be useful in several scenarios, such as modular applications, plugin systems, or updating parts of an app without requiring a full update.

To load a DEX file dynamically in Android, we can use the DexClassLoader. This class allows us to load classes from DEX files that are stored in our app's private storage or any accessible directory.

Steps to Load a DEX File Dynamically:

1) Prepare the DEX File:

Ensure this DEX file is stored in an accessible directory. This can be our app's internal storage, external storage, or downloaded from a server.

2) Use DexClassLoader to Load the DEX File:

import dalvik.system.DexClassLoader;
import java.io.File;
import java.io.IOException;
import java.lang.reflect.Method;

public class DynamicDexLoader {

  public static void loadDexFile(Context context, String dexPath, String outputDir) {
      File optimizedDexOutputPath = new File(outputDir);
      DexClassLoader dexClassLoader = new DexClassLoader(
            dexPath,
            optimizedDexOutputPath.getAbsolutePath(),
            null,
            context.getClassLoader()
      );

      try {
        // Load the class dynamically
        Class<?> loadedClass=dexClassLoader.loadClass("com.example. DynamicClass");
        // Create an instance of the loaded class
        Object instance = loadedClass.newInstance();
        // Get the method to be invoked
        Method method = loadedClass.getMethod("dynamicMethod");
        // Invoke the method
        method.invoke(instance);
    } catch (Exception e) {
         e.printStackTrace();
    }
  }
}
        

3) Call the Dynamic Loader in the Activity:

public class MainActivity extends Activity {
    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        // Path to the dex file
        String dexPath = "/path/to/dexfile.dex";
        // Output directory for optimized dex files
        String outputDir = getDir("dex", Context.MODE_PRIVATE).getAbsolutePath();

        DynamicDexLoader.loadDexFile(this, dexPath, outputDir);
    }
}        

The ability to dynamically load DEX files in Android provides a powerful mechanism for creating modular, flexible, and updatable applications. By leveraging DexClassLoader, developers can implement advanced features such as plugin systems or even apply the creational micropattern "Indexed Creation Method" that we saw in the article on "Systematic Object Oriented Design" (3.).

We reached the end of this first episode of our journey inside Android operating system.

In this episode we explored what makes Android unique compared to typical Linux distributions. We examined the similarities between user management in Linux and application management in Android. Then we looked at the most distinctive element that differentiates Android from Linux: the Android Runtime. Lastly, we looked at an interesting and uncommon application of dynamically loading Android bytecode (.dex) to implement a plugin mechanism.

That's all for this first stage.

See you on the next episode where we will continue to talk about the native level of the Android stack.

I remind you my newsletter "Sw Design & Clean Architecture"? : https://lnkd.in/eUzYBuEX where you can find my previous articles and where you can register, if you have not already done, so you will be notified when I publish new articles.

Thanks for reading my article, and I hope you have found the topic useful,

Feel free to leave any feedback.

Your feedback is very appreciated.

Thanks again.

Stefano

References:

1. Nikolay Elenkov, “Android Security Internals” No Starch Press (October 2014).

2. G.Meike, L.Schiefer, “Inside the Android OS” Addison Wesley (August 2021).

3. S.Santilli: "https://www.dhirubhai.net/pulse/from-testability-excellence-systematic-approach-design-santilli-lbo3f "