Android and Jenkins: what is the limit?

Thanks a lot to Kevin Martens for his review and suggestions.

After reading the title, you may be thinking "Wait, what? Is Jenkins somehow limited in building Android apps?" You can relax, as I may have phrased it incorrectly. We’re not talking about building Android apps with Jenkins, which has no limitations as far as I know. We’re talking about building?something?with Jenkins, using an Android device as a Jenkins node, or potentially as a Jenkins controller. Does this sounds appealing or strange enough to you? Continue reading to learn more about the relationship between Android and Jenkins!

Jenkins and?aarch64

I joined the Jenkins project in April 2022. At that time, we could already find?aarch64 docker?images, for the agents or the controller, and regular installers for?aarch64?Linux. The?oldest image?for a controller I found was from August 2021, and the?oldest image?for an agent was from February 2022. This is nothing new, as Jenkins works on?aarch64?Linux and has been running on that CPU architecture for years.

It’s pretty easy when using Linux on an?x86_64?machine, but it can be more difficult on an?aarch64?machine. This is because the tools needed to build Android applications were not available until late 2021, with the release of Android Build Tools?31.0.0. Of course, you can use?Rosetta?to build your applications, and even combine it with?Docker.

In my experience with Jenkins and?aarch64, I have several?aarch64?Jenkins agents and controllers. Some of them are using?docker?and some of them are installed directly on the Linux machine thanks to the?standard instructions. Thankfully, there has been nothing outstanding to worry about, as Jenkins works fine for me with?aarch64.

Android and?aarch64

Until recently, it was difficult to build Android applications on an?aarch64?machine. The main reason was that the?Android build tools?were not compatible with?aarch64?machines. Before version?31.0.0, there was a bug in the Android Build Tools that caused the?aapt2?tool to crash when building resources on?aarch64?machines. This issue was resolved in version?31.0.0, which added native support for?aarch64?and fixed the?aapt2?crash on these machines. Thanks to this, Android Build Tools are now natively compatible with?aarch64.

Isn’t that fantastic? This meant there was no need to use Rosetta to build Android apps on?aarch64?anymore. Until I started writing this post, I was actually using build-tools?30.x. I didn’t need to build Android apps on?aarch64?machines, so upgrading was unnecessary.

However, a friend of mine works on an?M1 Mac, which happens to be an?aarch64?machine, and wanted to build Android apps on his Mac. He was working with Docker, which translates?x86-64?to?aarch64, as long as you specify that you’re using an?x86-64?image to begin with. I know it’s strange that?x86-64?is called?amd64?in Docker, but that’s not the point.

[...]
android-agent:
    platform: linux/amd64
    build: ../
    restart: unless-stopped
    volumes:
        - android-agent-data:/home/jenkins:rw
        - ../adbkey.pub:/home/jenkins/.android/adbkey.pub:rw
        - ../adbkey.txt:/home/jenkins/.android/adbkey:rw
    environment:
        - JENKINS_AGENT_SSH_PUBKEY=ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBpNqXQ4x7fPPUBbYPxKF77Zqq6d35iPCD2chg644OUD
        - STF_HOST_NAME
[...]        

We could theoretically build Android apps on?Raspberry PI 3B+?with?Jenkins?from now on.

In the introduction, I stated that I wanted to build something with Jenkins on an Android device, and not the other way around. The goal is to run parts of Jenkins (first an agent, then a controller, then a whole Jenkins instance thanks to Docker) on an Android device. So why am I telling you about building Android apps on?aarch64?machines with Jenkins? Because it would be kind of ironic, and tons of fun, to build an Android app with Jenkins on an Android device! That would even allow me to introduce you to the?Jenkinsception?concept. More on that in a future post.

Get?sshd?working on an Android device

What does?sshd?have to do with installing Jenkins? Well, we have a few different ways to start and connect a Jenkins agent to a Jenkins controller, but the easiest one for me is to use an ssh agent. The first step is to get?sshd?working on an Android device. There are various ways to access an Android device through SSH. Here are some of them:

• ADB: You can use the Android Debug Bridge (ADB) to connect to your Android device over SSH. This requires you to enable USB debugging on your device and have ADB installed on your computer. The following guide may not work any longer, but it’s a good starting point:?How to enable ssh on Android.
• SimpleSSHD: SimpleSSHD is a lightweight and easy-to-use SSH server app for Android. It supports key-based authentication, and you can configure it to run at startup.
• Termux: Termux is a terminal emulator and Linux environment app that allows you to start an SSH server on your Android device. You can then use any SSH client to connect to your device over SSH. Unfortunately, updates for Termux are not available on the Google Play Store anymore, but you can still find it on?GitHub?or?F-Droid.

It is important to note that SSH access to your device can pose security risks, so it is recommended to use caution and?only?enable SSH access when necessary.

Termux is my go-to choice when it comes to using some kind of Linux on Android. There are packages available that will allow you to install new software, and package updates, like a "real" Linux distribution. I almost feel at home when using it.

While reading the?Termux documentation, I discovered that Termux has an SSH server (OpenSSH) built-in. It’s not enabled by default, but it’s easy enough to configure. The following instructions are available on the?Termux wiki, and I’ve added some details to make it easier to follow.

Starting and stopping the OpenSSH server

Since Termux does not use an?initialization system, services are started manually from the command line.

To start the OpenSSH server, you need to execute this command:?sshd. If you need to stop?sshd, just kill its process:?pkill sshd.

SSH daemon logs to the Android system log, and you can view them by running?logcat -s 'sshd:*'. This is possible from either Termux or ADB.

Setting up password authentication

Password authentication is enabled by default, making it easier to get started. Before proceeding, make sure that you understand that password authentication is less secure than a pubkey-based one.

Ensure that everything is up-to-date and the?openssh?package is installed:

 pkg upgradepkg install openssh        

Please note that?$PREFIX?is a variable that points to the Termux installation directory. It is usually?/data/data/com.termux/files.

Password authentication is enabled by default in the configuration file. You can review the file at?$PREFIX/etc/ssh/sshd_config, and it should contain this data:

 PrintMotd yes
 PasswordAuthentication yes
 Subsystem sftp /data/data/com.termux/files/usr/libexec/sftp-server        

If your file does not look like this, you will have to edit the file. Note that?vi?is not installed by default, but?nano?is. You can use?nano?to edit the file.

Set a new password by executing the command?passwd. While the program allows a minimal password length of one character, the recommended password length is more than eight to ten characters. Passwords are not printed on the console.

 $ passwd New password:
 Retype new password:
 New password was successfully set.        

Setting up public key authentication

Public key authentication is the recommended way for logging in using SSH. You need to have a public/private key pair to use this type of authentication. For a successful login, the public key must exist in the authorized keys list on the remote machine, while the private key should be kept safe on your local host.

In the following example, it is assumed that you want to establish public key authentication between your PC (host) and your future Jenkins agent, which happens to be an Android device running Termux (remote). It is also assumed that you’re running a Linux distribution on your PC,?WSL2, or even?Cygwin. It would be better if both machines were using the same network, for example both are connected to the same Wi-Fi network. It is also assumed that you know your Android device’s IP address. If you have access to your router webpage, you should be able to see which IP has been assigned to your Android device. If you don’t have access to the router webpage, you can find your IP address on an Android device by following these steps:

• Open the Settings app on your Android device.
• Scroll down and tap on "About phone" or "About device".
• Look for the "Status" or "Network" section and tap it.
• Find the "IP address" or "Wi-Fi IP address" option, which will display your device’s IP address.

Alternatively, you can also find your IP address within Termux by typing the following command:?ip addr show. Be aware that if the package is not installed yet, you will need to issue?pkg install iproute2?first. Look for the?inet?line next to the?wlan0?line that has your IP address given by your Wi-Fi router.

If you do not have ssh keys, you can generate them. In this example, we will generate an?RSA?key. On the PC, execute the command:?ssh-keygen -t rsa -b 2048 -f id_rsa, replacing?id_rsa?with the name of your key. For me it would be?ssh_key_for_jenkins_agent_2023-03-10. The command shown above generates a private RSA key with a 2048-bit key length and saves it to the file?id_rsa. In the same directory, you can find a file named?id_rsa.pub, and this is a public key.

For me, the command was:

 ssh-keygen -t rsa -b 2048 -f ssh_key_for_jenkins_agent_2023-03-10
 Generating public/private rsa key pair.
 Enter passphrase (empty for no passphrase):
 Enter same passphrase again:
 Your identification has been saved in ssh_key_for_jenkins_agent_2023-03-10
 Your public key has been saved in ssh_key_for_jenkins_agent_2023-03-10.pub
 The key fingerprint is:SHA256:yoykbWyCHuqrANFBkO41vuXMC7kLhsVfe8caLWQEUqk user@PC
 The key's randomart image is:
 +---[RSA 2048]----+
 |.+o ..o.         |
 |.. . ...         |
 |o .  .  .        |
 | + oE  .         |
 |o = o . S        |
 |o+ B.* = o       |
 |++oo& = + +      |
 |= o=o+ . =       |
 |=+.o... .        |
 +----[SHA256]-----+        

The key was generated in the current directory, not in?$HOME/.ssh. I tend to move the generated key in that?$HOME/.ssh?directory (mv ssh_key_for_jenkins_agent_2023-03-10* ~/.ssh?for me). I then change the directory to?$HOME/.ssh?(cd ~/.ssh) and change the permissions of the key (chmod 600 ssh_key_for_jenkins_agent_2023-03-10).

2048 bit is the minimal key length that is considered safe. You can use higher values, but do not use a higher than 4096, as the remote server may not support that big of a key.

Copy the key to the remote machine (your Jenkins agent wannabe running Termux). Password authentication must be enabled to install a public key on the remote machine. Now execute:?ssh-copy-id -p 8022 -i id_rsa IP_ADDRESS, making sure to replace?id_rsa?with the name of your key and?IP_ADDRESS?with the IP address of your Android machine.

Alternatively, you can manually copy the content inside?id_rsa.pub?(public key), which is already on the PC, and looks like?ssh-rsa <A LOT OF RANDOM STRINGS> user@host. After copying, paste this content to the Termux file?$HOME/.ssh/authorized_keys?(remote machine). Before copying and pasting, you must connect through?ssh user@IP_ADDRESS -p 8022, replacing?IP_ADDRESS?with the IP address of your Android machine. Doing so enables you to copy the content of the public key, using any text editor available on PC, and paste it inside an ssh session handled by Termux.

What looks strange to me is that?user?could be just about anything. I tried to log in without supplying a user, which means I was using my PC username, and it worked! I tried to log in with a different username and this also worked. When issuing the?whoami?command inside Termux, it shows the username of the Termux user, which is?u0_a504?in my case.

If everything went fine, you will see a message like this one:

 Number of key(s) added: 1        

If your system has an ssh-agent, you should now?add your newly generated key to the agent. After adding the key, try logging into the machine with:?ssh -p '8022' 'IP_ADDRESS'?Be sure to replace?IP_ADDRESS?with the IP address of your Android machine and check to make sure that only the key(s) you wanted were added. If you don’t have an agent running, you will have to use a slightly different command:?ssh -i id_rsa -p '8022' 'IP_ADDRESS'?Here, you need to replace?id_rsa?with the name of your key and?IP_ADDRESS?with the IP address of your Android machine. That would display for me as:

 ssh -i ssh_key_for_jenkins_agent_2023-03-10 -p 8022 192.168.1.xx
 Welcome to Termux!        

At this point, password authentication can be disabled. Using?nano, edit the file?$PREFIX/etc/ssh/sshd_config, and replace the line beginning?PasswordAuthentication?with?PasswordAuthentication no. Back in the Termux app, execute the command?pkill sshd && sshd?to restart the?sshd?server with the updated configuration file. Of course, if you were to do that from your PC, you would be disconnected and the ssh server would not be restarted.

Now you can log in to the remote machine without a password. Just execute the command?ssh -p '8022' 'IP_ADDRESS'?For this command, replace?IP_ADDRESS?with the IP address of your Android machine, or with the more complex command?-i, if your machine does not use an ssh agent.

Installing Java on Termux

We all know that Jenkins is written in Java. We also know Android apps are written in Java or Kotlin, so while we could hope that we magically skip this step, I’m afraid we can’t. The virtual machine that runs Android apps is not the same as the one that runs on your PC. Later on, we’ll detail the main differences between the two. The Android virtual machine (called dalvik) is available on Termux, but it is not capable of executing our agent.jar file, since the?java?command is not available yet.

$ dalvikvm -showversion
ART version 2.1.0 arm64
$ java --version
bash: /data/data/com.termux/files/usr/bin/java: No such file or directory        

For the time being, let’s assume that we need to install Java on Termux. Let’s find out which java versions are available on Termux:

pkg update && pkg search openjdk
Checking availability of current mirror:
[*] https://packages-cf.termux.dev/apt/termux-main: ok
Sorting...
Done
Full Text Search...
Done
openjdk-17/stable 17.0-25 aarch64
  Java development kit and runtime
openjdk-17-source/stable 17.0-25 all
  Source files for openjdk-17
openjdk-17-x/stable 17.0-25 aarch64
  Portion of openjdk-17 requiring X11 functionality        

Nice. Jenkins supports Java 17 since the?2.355?and?2.346.1 LTS?releases, so let’s go with OpenJDK 17.

pkg install openjdk-17        

Now the?java?command is available:

java --version
openjdk 17-internal 2021-09-14
OpenJDK Runtime Environment (build 17-internal+0-adhoc..src)
OpenJDK 64-Bit Server VM (build 17-internal+0-adhoc..src, mixed mode)        

Creating a Jenkins ssh agent

You should now be able to connect via?ssh?to your Android device running Termux if you have issued the?sshd?command. Your?ssh?server also knows about the?ssh?key you generated on your PC. We will now create a credential based on that key within Jenkins, that will allow you to connect to your Android device running Termux from Jenkins later on.

Creating a Jenkins ssh credential

For this part, there is almost nothing specific to Android. You can follow the?official documentation, and there are instructions on how to?create a Jenkins credential.

Setting up a Jenkins ssh agent

It’s now time to?set up your agent.

You can use?Android?as a label for your agent. Choose the?Launch agent via SSH?option. The hostname should be your phone’s IP address, which was named 'IP_ADDRESS' in the previous steps.

The credentials should be the ones you created in the previous steps. The remote root directory should be?/data/data/com.termux/files/home. The host key verification strategy should be?Non-verifying Verification Strategy. The?Launch method?should be?Launch agent via SSH.

Don’t forget to select the?Advanced?option and change the port to?8022. You could also specify the path of the?java?executable you installed in the previous steps, which happens to be?/data/data/com.termux/files/usr/bin/java. Since I have installed the 'Platform Labeller' plugin, I have also checked the 'Automatic Platform Labels' checkbox. We’ll see later on if it can cope with Android devices that don’t use the?lsb_release?command.

The very last thing to do is to select?Save. You should now see the complete list of your defined agents. While the agent has been created, it may have not started yet. If that’s the case, select the name corresponding to your newly created agent ('Android Phone' for me) and select?Launch?to start the agent. After some time, you should see in the logs?Agent successfully connected and online, which means you can now use this agent to run your builds.

Using a Jenkins ssh agent

Let’s create a new job and use our newly created agent to run it.

The simplest job that comes to mind is a?Freestyle project?that runs the?uname -a?command. That should give us some information about the Android device we are running on, while proving that the agent is working. Once again, there is nothing specific to Android for this step, so you can follow the?official documentation. The only changes to the documentation I have made are:

• I have used the?Android?label to make sure the job is run on the Android agent.
• I have used the?uname -a?command instead of the?echo $NODE_NAME?command.

Started by user admin
Running as SYSTEM[EnvInject] - Loading node environment variables.
Building remotely on Android Phone (aarch64 aarch64-unknown+check_lsb_release_installed aarch64-unknown+check_lsb_release_installed-unknown+check_lsb_release_installed android unknown+check_lsb_release_installed-unknown+check_lsb_release_installed unknown+check_lsb_release_installed) in workspace /data/data/com.termux/files/home/workspace/Android First Job
[Android First Job] $ /bin/sh -xe /data/data/com.termux/files/usr/tmp/jenkins13760213506108463207.sh
+ uname -a        

Linux localhost 4.4.192-perf+ #1 SMP PREEMPT Fri Dec 10 13:53:37 WIB 2021 aarch64 Android

Finished: SUCCESS        

We now have a working Jenkins agent running on Android, thanks to Termux. Now what? Of course, we will be limited to the commands and packages that are?available?on Termux. For example, I can’t see?gcc?in the list of available packages, which could be troublesome.

pkg search gcc
Checking availability of current mirror:
[*] https://termux.astra.in.ua/apt/termux-main: ok
Sorting...
Done
Full Text Search...
Done        

No gcc? You’re right, there is no?gcc?in the official Termux repository. However, the Termux community comes to the rescue with some repositories that provide additional packages, like?gcc. After installing the repository, we can install?gcc.

pkg search gcc
Checking availability of current mirror:
[*] https://termux.astra.in.ua/apt/termux-main: ok
Sorting...
Done
Full Text Search...
Done
gcc-6/termux 6.5.0-2 aarch64
  GNU C compiler
gcc-7/termux 7.4.0-2 aarch64
  GNU C compiler
gcc-8/termux 8.3.0-3 aarch64
  GNU C compiler
libgccjit-8-dev/termux 8.3.0-3 aarch64
  GCC just-in-time compilation
libgomp-7/termux 7.4.0-2 aarch64
  openmp library for gcc
libgomp-8/termux 8.3.0-3 aarch64
  openmp library for gcc-8        

As you can see, we have a few?gcc?versions to try out.

What if we need?gcc?10, for example? We would have to?compile it ourselves?like in the good old days. This solves the problem for?gcc, but what about other packages? We are somewhat limited by the availability of packages on Termux.

What if we could work around that limitation though? What about running Docker on Termux? Docker has no limit on packages as long as we choose the right base image, right? So, we could run a Jenkins agent on Termux through a Docker image, based on another distribution that happens to supply all the packages we need. The?slight?problem that may arise, is that Docker is not easily installed on Termux, and once installed, it won’t work out of the box.

Android apps are running some kind of JVM, right? So why not use a Jenkins inbound agent?

Android apps are written in Java or Kotlin programming languages, and they run on one of two Java Virtual Machines (JVM):

• Android Runtime, known as?ART
• Dalvik Virtual Machine, known as?DVM.

It is possible to access the JVM from an?ADB?shell and run Java code using the?dalvikvm?command. This is a command-line tool that allows you to execute Java code on the DalvikVM.

Nevertheless, there are preliminary steps that you need to take before you can run Java code on an Android device: * Compile your Java code into a?.class?file * Transform it into the?DEX?format using the?d8?tool * Push the resulting?.dex?file to your Android device * Run the Java class using the?dalvikvm?command.

It’s possible to some extent to automate these steps, but it’s not trivial.

The?dalvikvm?command is a low-level tool that may not be suitable for running complex Java apps, which may need additional dependencies to function properly. Even if that would work, it would be a very roundabout solution (which is fine with me), but where would we go from there? I mean, we have a subset of the Linux commands available in the ADB shell, but we can’t install tools, packages, etc. For example, how would we install?gcc?

So what could our Jenkins agent do? Not so much I’m afraid… We could still use Termux, as we’ve seen earlier Termux uses the base shell that is available through ADB. If we could launch the inbound agent through Dalvik, we would be able to use the Dalvik VM while using Termux, to keep the best of both worlds (Android & Linux-like).

Another solution would be to create a library from the agent.jar file and integrate it into an Android app. That part could work but then the resulting agent would be even more limited. There wouldn’t be a shell available, as the app is sandboxed. We would have an agent able to do almost nothing…

I’d like to know more nonetheless, so I’ll write down my thoughts about that in another article, once I’ve done my homework.