Private Registries and Package Management on JuliaHub

JuliaHub is a platform for technical computing using the Julia language, but it is also a platform designed to support the development and deployment of production applications. One feature that is becoming increasingly important to developers in the Julia ecosystem is the ability to create private registries that can manage the installation and usage of proprietary Julia packages. When you use JuliaHub, you connect to a dedicated private package server that knows who you are, and serves you up all of the registries you have access to—both public and private. Recent work on the JuliaHub platform has added more features to create and manage private registries and lets you register private packages using the same workflow you use to register public packages through JuliaHub.

How does this work?

An admin can create one or more private registries to store sets of internal packages developed by your teams. These registries have access controls that allow the admin to designate which users and groups at the organization can access them. When a user updates their registries in the Julia client, whether locally or in the JuliaHub IDE, they are automatically served all the registries they have access to. From the user’s perspective, this is a zero hassle process: they magically get access to the packages they need—and the flip side, of course, is that they don’t see registries that they can’t access.

In JuliaHub, the current way to add a new private registry is by going to the Admin tab and clicking on the + button at the top to add a new one. You can see a quick example of how to find the private registries menu on JuliaHub and adding the URL for your repo where the packages are (note: currently we must activate this feature for your instance):

Registering a Package to a Private Registry on JuliaHub

To register a private package in a private registry, you simply provide the repo URL where your package source code is hosted on GitHub, GitLab, Bitbucket, or a self-hosted git server. Our private registry engine will then clone your repo, ingest registered package versions from the repo, securely back them up, and serve them to your users.

Users automatically have access to packages in all the registries that they’ve been granted access to. They can simply install and use them in the Julia Pkg client as they would with public packages. When a new version of a private package is registered, users will get the new version when they do a package update operation, just as they would with public packages.

The package manager interfaces in Julia UIs, in the JuliaHub IDE, Pluto notebooks, the terminal, or vertical products like Pumas, Cedar and JuliaSim, will all get new package versions automatically when they update their dependencies. Since the Julia REPL supports automatically installing dependencies that are in some registry that your package server advertises, getting a private package is as easy as typing in using DemoNewPackage as in the following example:

The package manager included in the REPL will search all registries public and private for that package and once found, install the latest version. Also, should you need to add new dependencies to the package, you can do so and then just type in “activate DemoNewPackage” and then add “DemoNewPackage” to your environment.

For more details about new package creation and registration, refer to:

https://julialang.org/contribute/developing_package/

Registry Management Best Practices

In general, having a strong foundation in package management best practices helps organizations increase development velocity by making sure all packages and versions are tracked, updated correctly, and are safe and compliant to the organization’s regulatory requirements. JuliaHub’s private package registries and the package management capabilities that go along with them, were built to help solve this need.

Capabilities of Private Registries on JuliaHub

• Users can create one or more private registries.
• Users can configure either HTTPS or SSH protocol to fetch the registry and registry resources.
• Users can configure a private API server for registry.
• Users can disable authorization for all users while registering a package.
• Users can use a private registry hosted on server which has a private certificate. User may have a root certificate configured on JuliaHub.
• Users can configure additional registries on which the registry packages will depend.
• Users can specify packages to include or block (and versions) in an allow/disallow list.

What is the main use case for the feature?

Imagine that your organization is a multinational conglomerate with various departments that all have their own software teams to help aid product development. It’s reasonable to believe that a large organization like this would have product teams that are in highly-regulated industries where safety and compliance are of utmost concern. Other departments under this multinational organization might not have such stringent regulations. This could mean that custom Julia packages being developed for different teams might have very different compliance needs. For example, the Aerospace division for a company probably has less freedom to use any open source package (and open source license) than the smart appliances division might have.

So in this scenario, a large organization could create a private registry for Julia packages created for their highly regulated software teams and a totally different private registry to manage packages and dependencies for software teams that work in less regulated environments. This level of control allows an organization to manage software standards for multiple departments under one roof on JuliaHub. There is even an allow/disallow list that can be used to create “gated/vetted registries” to keep non-compliant packages out of your development pipeline.

Matthjis Cox from ASML highlights the need in his article innersource ecosystem model of development.

“A multi-repository, multi-package setup. Similar to the public open-source ecosystem you observe on Github, including a separate registry. What is a package registry? A registry is merely a lookup table with links to all the packages in your organization. A package manager uses this registry to find and install packages for the users, including all the package dependencies. For example, see the Julia General Registry.”

Understanding the Julia Package Manager and Server:

The Julia Package Manager was built after understanding many of the situational limitations of package management in other programming languages. The package manager governs the way packages, versions, and dependencies are installed and the package server caches the installed packages (from the General Registry and all private registries) and only looks to the original source (usually GitHub, but can also be GitLab or BitBucket) as a fallback.?

Additionally, the package server works with Julia packages via two important files that store metadata that provide assurance that the specific package that you are hoping to install is actually the right one. These files also serve the need for provenance and can be used in a software bill of materials. For reference, these two files are:?

Project.toml

The project file describes the project on a high level, for example, the package/project dependencies and compatibility constraints are listed in the project file.

name = "HelloWorld"?

uuid = "b4cd1eb8-1e24-11e8-3319-93036a3eb9f3"?

version = "0.1.0"?

authors = ["Some One <[email protected]>"] [deps]        

Manifest.toml

The Manifest.toml file is generated and maintained by Pkg and, in general, this file should never be modified manually. It is used as an immutable record to secure the package/version information.

julia_version = "1.8.2"?

manifest_format = "2.0"?

project_hash = "4d9d5b552a1236d3c1171abf88d59da3aaac328a"        

To learn more about the package server itself, refer to the documentation here.

Coming Soon on Private Registries: Package Analytics

We are now in the process of adding advanced capabilities to our private registry tools. One of the upcoming features includes a package analytics dashboards. This will help Admins understand how packages in both the public and private registries are being used throughout the organization. Some of this package data will also be available via JuliaHub’s traceability dashboard to help track the provenance of a package and specific version.

Learn More at the Upcoming Webinar

As of today, Enterprise subscriptions on JuliaHub can create as many private registries as needed - but soon, we will be adding this capability for other JuliaHub subscriptions as well.?

To see these features in action and learn more about upcoming registry management capabilities? - join the upcoming webinar hosted by Deep Datta and Bill Burdick: Register today.

The original blog post can be found here on JuliaHub.com.