Building and sustaining a microservices ecosystem - Part 2

Introduction

This article is part 2 of a three-part series on?“Building and sustaining a microservices ecosystem”. In Part-1 we covered the various considerations w.r.t. Cloud, Infrastructure, Platform, etc.

In this article (Part-2), we will go through the various security and design considerations when designing microservices.

Design considerations

This reference architecture is focused on designing microservices-based applications and the recommended practices that will apply to workloads running on a Kubernetes platform.

1. Microservices

A microservice is a loosely coupled, independently deployable unit of code. Microservices typically communicate through well-defined APIs and are discoverable through some form of service discovery. The service should always be reachable, even when the pods move around.

Some of the salient features of a microservice are:

• The microservices ecosystem is a platform of services, each encapsulating a business capability.
• A business capability represents what a business does in a particular domain to fulfill its objectives and responsibilities.
• Each microservice exposes an API that developers can discover and use in a self-serve manner.
• Microservices have an independent lifecycle. Developers can build, test, and release each microservice independently.
• The microservices ecosystem enforces an organizational structure of autonomous longstanding teams, each responsible for one or multiple services.
• Contrary to general perception and the “micro” in microservices, the size of each service matters least and may vary depending on the operational maturity of the organization.?

The Kubernetes Service object is a natural way to model microservices in the Kubernetes platform.

2. Cross-cutting concerns

Cross-cutting concerns are technical requirements that are commonly applicable for all the features of a given software, for instance - logging.

If there’s one area where Monolithic applications have an advantage of, it is the code reuse for cross-cutting concerns. They can be shared centrally in Monolithic through single instances of libraries or frameworks.

In Microservices architecture however, these cross-cutting concerns may be addressed differently in each microservice or the same may be duplicated across all microservices. This creates manageability and maintainability overhead. In some other Microservices-based applications folks try to create a separate service altogether to address the cross-cutting Concerns like logging, which is then called over the network via an API call. The result is unpleasant, crude implementation, latency overhead, and it introduces new forms of challenges. It is not always recommended to create separate standalone Microservice/s to address cross cutting concerns.

Solutions that address the cross-cutting concerns in fact doesn’t really apply to the Microservices Architecture as such; but that doesn’t mean we should ignore it in the Microservices Architecture & Design. Instead the cross-cutting concerns should be addressed uniformly across all Microservices. For disparate development Agile team setup, sets of standards should be followed to avoid different ways of implementations by different teams for the same generic concerns. During sprints, cross cutting concerns should be default task cards for every service.

Below are some primary cross-cutting concerns and guidelines to address them:

• Externalized configuration: Includes credentials and network locations of external services, such as databases and message brokers. These are shared configurations, placeholders, and lookups — these have different contexts on various technical angles. Standards needs to be set on how this should be implemented. Example concerns for this are: file and directory separation for credential values, dynamic values, communication session-related parameters, whether to localize a parameter or not, etc. Getting your configuration management right is the difference between pleasant, scalable microservice architecture and a configuration mess like you have not seen before.
• Logging: Configuring of a logging framework, such as log4j. For investigation purposes, implementation of logging should ideally have conventions to help with tracing and transaction querying.
• Exception handling: Strategies and standards that should be implemented before terminating the session/message in an event of an exception. In native object-oriented programming exception handling has different types: checked, unchecked, irrecoverable error, and guidelines should be set to address each type of exception across microservices for consistency.
• Monitoring and alerting: Proactive health-check frameworks to be defined, and in some cases benchmarking performance utilities to provide throughput metrics and sending notification and/or alert logics (e.g., send email or SMS) based on alert rules configured. Metric measurements that provide insight into what the application is doing and how it is performing
• Audit tracing and correlation. For end-to-end tracing of transactions with an external vertical system as part of the consideration. A strategy is needed to have a traceable form of “reference number,” an ID that maps the entire lifecycle of the payload message including external interactions with another microservice.
• Authentication and authorization: Repeatable security-related standards that need to be abstracted due to a number of services will be implementing similar authentication and/or authorization mechanism.

3. Event-driven microservices

An excellent way to add visibility to what is happening to your service is through event sourcing or event logging. The fundamental concept behind this pattern is that every change to the state of an application should be encapsulated in an event object and stored sequentially.

Event-driven architectures are ideal for improving agility and moving quickly. They’re commonly found in modern applications that use microservices, or any application that has decoupled components. When adopting an event-driven architecture, you may need to rethink the way you view your application design.

Benefits of event-driven microservices

• Scale and fail independently.
• Develop with agility.
• Audit with ease.
• Optimized resource utilization.

4. Data storage

In a microservices architecture, services should not share data storage. Each service should own its own private data in a separate logical storage, to avoid hidden dependencies among services. The reason is to avoid unintentional coupling between services, which can happen when services share the same underlying data schemas. Also, when services manage their own data stores, they can use the right data store for their particular requirements.

Avoid storing persistent data in local cluster storage, because that creates host affinity and ties the data to the node. Instead:

• Use a mounted persistent volume, or
• Use Database-as-a-Service, or
• Use an external database service

5. I18N & L10N

The microservices should support internationalization and localization.

The art of making software independent of the underlying system defaults is called internationalization (L18N). Every system has defaults, such as encoding and locale. Any piece of software that unknowingly relies on these defaults may not work correctly when ported from one system to another because system defaults vary from one system to another; and thus, are not internationalized.

The art of making software adapt to the underlying locale is called localization (L10N). Software must rely on these defaults, to display the text messages, error messages, and other messages in the locale it is running on so that it makes sense for the person who is using that software.

A locale represents a language. Therefore, adapting to a locale means the software should be able to display or take inputs in a language specific manner. For example, the French language is a locale. But the French spoken in France differs from the French spoken in Canada, so we augment our definition of locale to say that a locale represents a specific language of a specific country. Again, a language spoken in a specific country can have variations. For example, the ancient traditional Chinese spoken in China vs the simplified Chinese spoken in China, so we augment our definition of locale again to say that a locale represents a specific language of a specific country with variants.

When we write software that adapts to the underlying locale this actually means addressing the community that will use this software. Therefore, locale-specific things such as text messages, exception messages, the date and time, currency, and other messages should be displayed in a way that allows the user of the software to understand what is displayed (the output of the software). There is a community of users that the localization tries to address, because we don’t want to re-write the entire software for every language, and every dialect of that language.

Resource constraints

Resource contention can affect the availability of a service. Define resource constraints for containers, so that a single container cannot overwhelm the cluster resources (memory and CPU). For non-container resources, such as threads or network connections, consider using the bulkhead pattern to isolate resources.

Use resource quotas to limit the total resources allowed for a namespace. That way, the front end can’t starve the backend services for resources or vice-versa.

When you specify a pod, you can specify how much of each resource a container needs. The most common resources to specify are:

• CPU
• Memory (RAM)

Securing microservices

1. Role-based access control (RBAC)

Kubernetes has mechanisms for role-based access control (RBAC). Kubernetes RBAC controls permissions to the Kubernetes API. For example, creating pods and listing pods are actions that can be authorized (or denied) to a user through RBAC.

It’s good practice to scope Kubernetes RBAC permissions by namespace, using Roles and RoleBindings, rather than ClusterRoles and ClusterRoleBindings.

2. Managing secrets

Applications and services often need credentials that allow them to connect to external services such as a SQL database. The challenge is to keep these credentials safe and not leak them.

Kubernetes Secrets let you store and manage sensitive information, such as passwords, OAuth tokens, and SSH keys. Storing confidential information in a secret is safer and more flexible than putting it verbatim in a pod definition or in a container image.

3. Pod and container security

This list is not exhaustive, but here are some recommended practices for securing your pods and containers:

• Don’t run containers in privileged mode. Privileged mode gives a container access to all devices on the host.
• When possible, avoid running processes as root/superuser inside containers. Containers do not provide complete isolation from a security standpoint, so it’s better to run a container process as a non-privileged user.
• Automate image patching. A container image is built up from layers. The base layers include the OS image and application framework images, such as Node.js. The base images are typically created upstream from the application developers and are maintained by other project maintainers. When these images are patched upstream, it’s important to update, test, and redeploy your own images, so that you don’t leave any known security vulnerabilities.

4. TLS/SSL encryption

In common implementations, the Ingress controller is used for SSL termination. So, as part of deploying the Ingress controller, you need to create a TLS certificate. Only use self-signed certificates for dev/test purposes.

For production workloads, get signed certificates from trusted certificate authorities (CA). You may also need to rotate your certificates as per the organization’s policies.?

Namespaces – organizing services

Use namespaces to organize services within the cluster. Every object in a Kubernetes cluster belongs to a namespace. By default, when you create a new object, it goes into the default namespace. But it’s a good practice to create namespaces that are more descriptive to help organize the resources in the cluster.

For a microservices architecture, consider organizing the microservices into bounded contexts, and creating namespaces for each bounded context. For example, all microservices related to the “Order Fulfillment” bounded context could go into the same namespace. Alternatively, create a namespace for each development team.

Place monitoring and utility services into their own separate namespace. For example, you might deploy ELK or Prometheus and Grafana for cluster monitoring, or Tiller for Helm, etc.

To be concluded in part 3 where we will talk about monitoring, upgrades and deployment considerations and briefly on the DevOps considerations for operating the microservices.

Stay Tuned!