The perfect secure Azure Kubernetes Deployment (part 1)
(EN)
ccording to Gartner, in 2022 75% of all organisations will run containerised applications. At he moment Kubernetes is the most populair container orchestrator available. Azure Kubernetes Service (AKS) is Microsofts “cloud implementation” of Kubernetes.
As Kubernetes is quite a complex orchestrator, security isn’t simple either. In this story I will give some insights of the way how you can deploy Azure Kubernetes with security in mind. In this story I will mainly be focussing on the Azure part of AKS. In upcoming stories I will get into detail on security of the cluster itself.
Azure Kubernetes Service is represented as an Azure resource that lives in a resource group. Aside from that, each Azure Kubernetes Deployment has a resource group that starts with “MC_”. This resource group will hold al AKS related resources (think of the virtual machines, scale-sets, load balancers etc.)
Your initial deployment is important
The initial deployment of Azure Kubernetes Service is important. A lot of settings that can be configured during the initial deployment cannot be changed afterwards. You should ask yourself the following questions before deploying Azure Kubernetes Service:
-
Are planning to consume other Azure services (such as databases, virtual machines, etc.) and are they required to have a private connection to the K8S cluster? During the initial deployment you can configure Azure Kubernetes Service to use an existing Azure virtual network.
- Are there any requirements to disk encryption? By default, data is encrypted with Microsoft-managed keys. During the initial deployment it is possible to configure Azure Kubernetes Service to use your own managed keys.
- Do you want the Azure Active Directory to be responsible for authentication in the Azure Kubernetes Service? With Azure AD-integrated AKS clusters, you can grant users or groups access to Kubernetes resources within a namespace or across the cluster.
Pro tip: when deploying Azure Kubernetes Services don’t limit yourself in using the Azure portal for your deployment. Azure Kubernetes Service can be deployed using the Azure CLI which is reliable than te portal.
Networking
When deploying AKS, a big decision regarding to network need to be made. Are you going to use your existing VNET or are you going to use a out-of-the box VNET that can come with AKS? Please see the following pros and cons of the network strategies
Azure Kubernetes Basic Networking (Using the Kubenet plugin)
When deploying Azure Kubernetes Service with a basic network setup. After deployment of the cluster (with basic networking) you will find a VNET in the “MC_” resource group that corresponds with your AKS cluster.
Pros
- This is quite a simple deployment that does not require any maintenance on the network part of AKS.
- This deployment is secure by default. The network is dedicated to the AKS cluster. There should be no other resources connected to this network.
Cons
- When removing the AKS cluster, the corresponding “MC_” resource group will be deleted. This will include the virtual network that did come with the AKS deployment.
- It is not recommended to change the network as it is part of a AKS managed deployment. This means that it is not recommended to add VNET peering, VPNs etc. Altering the VNET will however work, but you will receive no support when changing the VNET.
- An additional hop is required in the design of kubenet, which adds minor latency to pod communication.
- Windows Node Pools are not supported
Azure Kubernetes Advanced Networking (using the Azure-CNI plugin)
When deploying Azure Kubernetes Service, there is an option to deploy Azure Kubernetes in an existing VNET. When looking in the “MC_” resource group you will find no VNETs here.
Pros
- Your can deploy kubernetes in an existing VNET that already holds other resources. Traffic between the AKS cluster and your other resources will remain within the same VNET.
- It is supporter to apply changes to your VNET an thus add service-endpoints, UDRs (route tables), etc.
- Enables the cluster to use a network policy. With network policies you can define to which pods (and networks) your pod can communicate.
Cons
- Aside from the AKS cluster, the VNET also requires maintenance.
- If not configured well; you can easily access all resources from within your VNET.
Pro Tip: Always deploy an AKS cluster using the Azure CNI plugin (Advanced Networking). This will allow you to change the virtual network and will enable you to use features such as Windows Nodes and Network Policies. You can deploy an Azure Virtual Network that is only used by the AKS cluster to keep things simple and maintainable.
Disk Encryption
According to Microsoft, all data at rest is encrypted at rest:
“Azure Storage encrypts all data in a storage account at rest. By default, data is encrypted with Microsoft-managed keys. For additional control over encryption keys, you can supply customer-managed keys to use for encryption at rest for both the OS and data disks for your AKS clusters” — Microsoft docs
In normal circumstances, Microsoft should not be able to access/touch your data. If you do not use customer managed keys, it is however technically possible for them to access your data. To completely rule out that the cloud-vendor is able to access your data, you should use your own encryption keys.
If the AKS cluster holds valuable data, it is important to make use of customer managed keys in AKS. This allows you to take control over the data. If you are afraid that data is still usable after deletion, you can delete the customer managed key (which is represented as a disk-encryption set in Azure) and make sure all data is destroyed (because it cannot be encrypted anymore).
Pro Tip: The only costs that come with customer managed keys are the costs of Azure Key Vault. Have a look for the costs of Key Vault over here: https://azure.microsoft.com/en-us/pricing/details/key-vault/
Azure Active Directory Integration for Azure Kubernetes Service
Using Kubernetes role-based access control (RBAC), you can grant users, groups, and service accounts access to only the resources they need. There are two levels of access needed to fully operate an AKS cluster:
-
Access the AKS resource in your Azure subscription. This access level allows you to execute actions outside of the Kubernetes cluster; such as scaling the AKS cluster or upgrading the AKS cluster to a newer version.
- Access to the Kubernetes API. With this access level you can execute things inside of the AKS cluster; such as creating pods, services, replica sets, deployments etc. This access is controlled either by Kubernetes RBAC (traditionally) or by integrating Azure RBAC with AKS for Kubernetes authorization.
By default, all users that have access to the AKS resource in the Azure subscription, are able to get API access by using the “az aks get-credentials” command.
Integrating with the Azure Active Directory
You can configure Azure Kubernetes Service to make use of the Azure Active Directory for authentication. Using this configuration you sign-in to the AKS cluster using an Azure Active Directory identity. It is also possible to limit access to resources (the RBAC model in AKS) by using Azure Active Directoty identities and group membership.
In order to authenticate using the Azure Active Directory, the AKS cluster uses OpenID Connect. OpenID Connect is an identity layer build on top of the OAuth 2.0 protocol which is used by the Azure Active Directory.
By using this authentication method, security measures like multi-factor-authentication are working from within the cluster.
Using this kind of authentication will have some downsides at the moment of writing. Non-interactive sign-ins, such as Azure DevOps pipelines will not be supported by Kubectl. There is a workaround available where you could use kubelogin to have a non-interactive sign-in.
You can upgrade your AKS cluster to make use of AKS-Managed Azure Active Directory by executing the following command:
az aks update -g myResourceGroup -n myManagedCluster --enable-aad --aad-admin-group-object-ids <id> [--aad-tenant-id <id>]
Using managed identities
Azure Kubernetes Clusters make use of an identity. With this identity, the AKS Azure resource has permissions on the “MC_*” resource group which holds the virtual machines, loadbalancers, networkinterfaces etc. that are used by AKS. This identity can be either a managed identity or a service principal. If you chose to use a service principal, this principal can either be manually created, or the AKS cluster will create a service principal by itself.
Managed Identities are in essence a wrapper around service principals which will make sure credential rotation is happening automatically. There are however some limitations to the use of managed identities:\
Limitations:
- Managed Identities can only be enabled during deployment time. There is no way of upgrading a cluster to use managed identities.
- Tenant move of managed identities is not supported.
- Azure CLI version 2.8.0 or higher is required.
Pro Tip: I strongly recommend to use managed identities. Keys will automatically rotate every 46 days. If you chose to use a service principal it is probably less secure as most administrators will not rotate the service principal key that often.
The all mighty deployment script
Most of the time when I deploy an AKS-cluster, I make use of bash as the deployment is highly configurable, not limited to the AKS cluster and it is more solid than the Azure Portal.
Enjoy you Kubernetes Deployment!
