Crossplane, A Unified Approach to Automation

The end-to-end control plane for infrastructure, services, and application DevOps

As a cloud infrastructure and services consumer organization, automation is all about provisioning and life cycle management of resources after applying the in-house policies. We use infrastructure automation tools to build an abstraction over the cloud and services APIs. These abstractions will encode all the in-house policies to build reusable recipes. Many organizations prefer to execute the automation efforts with a centralized platform team to reduce the cognitive load on every product team and manage policies centrally.

Cognitive load is a key aspect as there will be thousands of infrastructure and service components from multiple cloud providers with hundreds of parameters to configure.

Configuring them securely according to the product team’s requirements and organization policy requires a lot of skill. It will not be optimum to compose this cognitive load inside every product team. But the centralized team model will create delivery velocity challenges for the modern agile product teams. They cannot wait for the infrastructure after placing a request to a central team. The central team will have challenges in prioritizing requests from multiple teams.

The centralized platform team may choose to build libraries to solve these issues to an extent. Still, most of the template-based infrastructure automation tools does not scale well from multiple points of view. Let’s look at these limitations from the perspective of Terraform, one of the most popular template-based infrastructure automation tools.

It does not mean that Terraform is not best for automation with all due respect. It has transformed software engineering positively for more than a decade. We will attempt to explain in this blow, how evolving situations expose the weakness of template-based IaC tools and how control plane-based tools can be an alternative and the next evolutionary step.

Terraform modules/libraries would have the following limitations.

The abstraction leaking

There is a leak in cognitive load abstraction by the platform team, as using Terraform modules/libraries by the product team means learning Terraform fundamentals at least. Terraform plan and apply action demands a high cognitive load going through all the changes to infrastructure. It's not ideal for protect team to perform this action. Additionally, templates erode with time exposing new parameters every day to meet different needs of different products. At some point, we will expose every parameter the cloud provider has to offer. This will create a leak in cognitive load abstraction by increasing the complexity to consume the library. Alternative methods like template forking are costly and error-prone to maintain.

Collaborative mode of interaction

Modules/libraries will demand a collaborative mode of interaction between teams, especially when we have to upgrade the library or provide a security patch or deprecate a component. The collaborative model of interaction will create a lot of friction in delivery velocity.

The access control and governance

Exposing the terraform modules to the product team will create authorization and security challenges. For example, a product team that requires a database provisioning access does not need to have access to the dependent network components. But the product team running the libraries had to configure the library with full access required by all dependent components. Governance of centralized policy and security controls will be very challenging with leaked authorization and de-centralized infrastructure management.

Synchronous provisioning

The resources are provisioned in a sequence with Terraform and other similar template-based tools. If infrastructure A depends on infrastructure B, we must define the order of execution. If one of the executions fails the whole automation fails. The monolithic representation of infrastructure and services is a key concern. Scenarios like shared infrastructure between multiple teams (CI/CD components) and private infrastructure ownership (network layer) will require us to model the infrastructure management with different team constructs. Modeling team boundaries requires an artificial effort with synchronous monolithic state files.

The multi-cloud challenge

Any organizations of a significant size run their workloads in more than one cloud provider. Embedding policies into the automation templates of every cloud provider requires a significant effort. Keeping these policies in sync across the automation scripts involves friction and is error-prone.

To balance the cognitive load, agile infrastructure provisioning, and policy governance we have to enable an API-based self-service model in infrastructure and services automation.

1. An API model for infrastructure and services provisioning would run the implementation remotely solving the abstraction leaking.
2. APIs can be modeled for any team construct easily with a precise authorization model.
3. APIs are the best construct for self-service collaboration between teams.
4. APIs provide a better construct for scenarios like security patching, version management, depreciation, policy encoding, etc.
5. Building these APIs with Kubernetes will additionally provides drift management and will enable us to model different team boundaries with asynchronous provisioning.

Crossplane

Crossplane, a modern API plane-based infrastructure automation platform built on Kubernetes, matches all the attributes required for the next evolution of infrastructure engineering. It solves all the problems that we discussed by taking advantage of the Kubernetes operating model. Crosplane uses CRDs (Custom Resource Definition) a Kubernetes extension point to extend Kubernetes for the next big use case.

With Crossplane, we can assemble infrastructure and services from any cloud/service provider to expose them as a high-level Kubernetes API.

These APIs can provide a universal experience for the consuming product teams, irrespective of the underlying infrastructure and services vendor. While composing the APIs, the platform team can model different granularity to suit the organization’s structure. Crossplane allows us to compose these APIs in a no-code way with configuration YAML. It allows an organization to build its own multi-cloud control plane with ease. Want to learn more about Crossplane? I am happy to announce the launch of the first book on Crossplane.

End-to-End Automation with Kubernetes and Crossplane: Develop a control plane-based platform for…

The book covers a wide range of topics like

1. Challenges in current infrastructure and services automation tools
2. How Crossplane can solve these challenges and bring new capabilities to the table.
3. Deep dive into all Crossplane concepts with hands-on examples
4. Architecture patterns to adopt Crossplane and Kubernetes in a best possible way
5. Ways to unify infrastructure, services, and application DevOps
6. Engineering practices required to manage the Crossplane platform team in an organization efficiently

Are you a Cloud-Native Architect or Platform Engineer or an Infrastructure/Application Operator interested in simplifying the infrastructure, services, and application automation? The book will guide building a fully declarative self-service infrastructure automation platform with Crossplane and Kubernetes. It will be a complete journey to automate the deployment of the cloud-native application, infrastructure, and services with Kubernetes APIs.

See you in an upcoming article to discuss more on Crossplane 🏄