How to Create Your Own Kubernetes Custom Resources

This post was originally published here on Caylent.com.

As a platform for managing containerized services and workloads, Kubernetes is incredibly modular. Modularity is more than just a theme; even controllers and resources that are native to Kubernetes are now being built as custom resources and controllers, all for the sake of expanding the platform’s modularity to the next level.

Of course, Kubernetes custom resources are not new to the platform. Custom resources are handy for when you need to extend the basic Kubernetes API and add new, specific features to your cluster. Custom Resources are not the only options when it comes to adding new features, but creating your own custom resources is the perfect solution for many challenges.

Why You Should Use Custom Resources

There are many reasons why a custom resource is the way to go, the main one being its compatibility with kubectl. You can expect native, top-level support regardless of the resource you add, which means you can run commands like kubectl get my-resource resource-name right out of the box.

Custom resources are also compatible with the native CLI of Kubernetes, plus you can update your custom resources using commands. Since the command line is natively supported, custom resources are perfect for when you want to automate updates and maintenance.

It doesn’t stop there either. Custom resources are more flexible than configmaps, mainly because they can be made to run CRUD against other objects as if they are native components. As a consequence, you also get native support for Kubernetes security features and RBAC. Check out the Kubernetes documentation for whether your scenario needs a configmap or custom resource here.

Creating custom resources is incredibly easy, but that doesn’t mean you should convert every component you want to add to the cluster into custom resources. There are times when using custom maps or going beyond by adding custom controllers is the better option.

Creating a Custom Resource

Creating a custom resource is a process that begins with the creation of CustomResourceDefinition in the form of a YAML file. Your resourcedefinition.yaml file defines the API version you use, the kind of resource you are adding?—in this case, CustomResourceDefinition?—and the metadata for the CRD. A quick kubectl apply command completes this step.

Next, you can begin creating objects. By default, the same apiVersion – kind – metadata – spec combination is used to define objects that you want to integrate. This consistent format makes creating custom resources easier regardless of your experience level.

Using CRDs is not the only way you can add custom resources. If you want more control over API behaviors, you can also go the API aggregation route. Aggregated APIs provide features that CRDs do not, such as enabling protocol buffers, allowing endpoints to support PATCH HTTP verb, and going beyond CRUD, with logs or exec. These do take some programming, but you do get access to the aggregation layer as a result. You even have the option to add EnableAggregatedDiscoveryTimeout=false to completely eliminate timeout restrictions. More on Aggregated APIs later though.

Accessible Features and Commands

Regardless of the route you choose to take, you will be able to integrate your custom resources in no time. Whether you use CRDs or API aggregation, you can use certain features to make your custom resources functional, including:

• CRUD, which supports basic CRUD operations via HTTP as well as through kubectl. This feature alone lets you do a lot of things with a custom resource.
• Watch, which is very handy for adding monitoring for changes to an object in your cluster or a specific deployment.
• Labels and annotations, which are handy for keeping your cluster organized. Labels and annotations are also useful for maintenance routines.
• Client libraries generation, and access to tools for generating client libraries as needed.
• Admission webhooks, which?—as the name suggests?—runs validation tests curing CRUD runtimes.

Other features are also available. For example, you can use merge-patch to utilize application/merge-patch+json content type. HTTPS is a feature that enables new endpoints using encrypted connections. The possibilities are endless since you can also add objects or create your own.

As an added note, some advanced features are only available to the API Aggregator method and not the CRDs. The aggregation layer contrasts Custom Resources in its way to make the kube-apiserver recognise new kinds of object. API aggregation increases your flexibility one step further, such as allowing you to customize how data is stored, or how API versioning is handled. For example, you cannot use custom storage when deploying CRDs, but the feature is accessible when you use API Aggregator. Custom storage lets you define storage in isolation or with added security measures.

It’s the same with protocol buffers. This advanced feature is only supported by API Aggregator, along with extra commands such as logs and exec. These advanced features are not always needed but be sure to check before deciding to use CRDs, so you don’t run into issues later.

Deploying Custom Resources

The last part of the equation is deploying the newly created custom resources. Before you go ahead and add custom resources, there are a few steps you want to check first, starting with reviewing the dependencies of your custom resources.

Make sure all third-party packages and installation bundles are present before you add the custom resource. This will stop your CRDs or AAs from triggering errors. The errors won’t always be catastrophic but checks and tests are the safer way to go.

You also need to make sure that the CRDs and aggregated APIs have sufficient roles to run. They usually come with new role definitions, but keep in mind that the standard RBAC roles will not be enough to run custom resources.

Lastly, you can access the commands and resources added through kubectl and other Kubernetes clients. Yes, custom resources will even be available through the Kubernetes dynamic client, client-go. Still, we are not at the end of the story yet.

Your custom resources still need to be maintained. Deployment is far from the last thing in this process. You still need to update your custom resources regardless of the format or method you use. Kubernetes allows for the updating and deletion of custom resources on the fly.

Speaking of updates and deletion on the fly, remember that aggregated APIs always require new deployment, while CRDs are far more flexible in how they can be deployed and maintained. These are little factors that will dictate how you use custom resources in the long run.

As mentioned earlier in this article, even native Kubernetes resources are now built as custom resources for the sake of modularity. This goes to show just how well-integrated and capable custom resources are when used correctly.

Caylent provides a critical DevOps-as-a-Service function to high growth companies looking for expert support with Kubernetes, cloud security, cloud infrastructure, and CI/CD pipelines. Our managed and consulting services are a more cost-effective option than hiring in-house, and we scale as your team and company grow. Check out some of the use cases, learn how we work with clients, and read more about our DevOps-as-a-Service offering.