This article provides insights into Virtual Nodes feature (in preview) of Azure Kubernetes Service (AKS). Enabling virtual nodes for AKS cluster allows to deploy or burst out containers to nodes backed by serverless Azure Container Instances beyond the defined cluster size. Also, along with fast provisioning of Pods, you only pay per second for their execution time. AKS virtual nodes currently support Linux container instances only.

Kubernetes Virtual Kubelet with ACI

Virtual Kubelet is an open source Kubernetes kubelet implementation that masquerades as a kubelet for the purposes of connecting Kubernetes to other APIs. 

Image source: Kubelet

Azure Container Instances (ACI) provide a hosted environment for running containers in Azure. When using ACI, there is no need to manage the underlying compute infrastructure, Azure handles this management for you. When running containers in ACI, you are charged by the second for each running container.

The Azure Container Instances provider for the Virtual Kubelet configures an ACI instance as a node in any Kubernetes cluster. When using the Virtual Kubelet ACI provider, pods can be scheduled on an ACI instance as if the ACI instance is a standard Kubernetes node. This configuration allows you to take advantage of both the capabilities of Kubernetes and the management value and cost benefit of ACI.

Create AKS cluster – Enable Virtual Nodes

You can enable Virtual Nodes on creating AKS cluster either in Azure Portal or Azure CLI. The steps needed to enable virtual nodes in Azure Portal are

• Set ‘Virtual nodes (preview)’ setting to ‘Enabled’ as displayed below

• Only Advanced Networking option is available for Network Configuration when Virtual Nodes feature is enabled since Virtual Nodes subnet needs to be specified.

In order to enable Virtual Nodes feature using Azure CLI, please refer use virtual nodes using Azure CLI.

Tip: If RBAC was enabled, after AKS cluster is created you will need to run command kubectl create clusterrolebinding kubernetes-dashboard -n kube-system --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard to open the Kubernetes Dashboard.

Now that AKS cluster with Virtual Nodes is created, let’s understand details of Virtual Nodes in subsequent sections.

Insights into Virtual Nodes 

Run command kubectl get nodes to view nodes created for AKS cluster and you will find that ‘virtual-node-aci-linux’ node has been provisioned.

Run command kubectl describe nodes virtual-node-aci-linux to view details of ‘virtual-node-aci-linux’ node.

The main points to notice in ‘virtual-node-aci-linux’ node details are

• Labels are the mechanism you use to organize Kubernetes objects. A label is a key-value pair with certain restrictions concerning length and allowed values but without any pre-defined meaning. Out of many labels defined for this node, the important labels which will be specified in PodSpec’s nodeSelector field are kubernetes.io/role: agent, beta.kubernetes.io/os: linux and type: virtual-kubelet. nodeSelector is the simplest form of constraint and it specifies a map of key-value pairs. For the pod to be eligible to run on a node, the node must have each of the indicated key-value pairs as labels (it can have additional labels as well).
• You can constrain a pod to only be able to run on particular nodes or to prefer to run on particular nodes. There are several ways to do this, and they all use label selectors to make the selection. Generally such constraints are unnecessary, as the scheduler will automatically do a reasonable placement (e.g. spread your pods across nodes, not place the pod on a node with insufficient free resources, etc.) however in this case you want more control on virtual node where a pod lands.
• Node affinity, described here, is a property of pods that attracts them to a set of nodes (either as a preference or a hard requirement). Taints are the opposite – they allow a node to repel a set of pods. A taint consists of a key, a value for the key, and an effect. While key and value can be anything, effect can be one of PreferNoSchedule, NoSchedule or NoExecute. Taints defined for ‘virtual-node-aci-linux’ node is virtual-kubelet.io/provider=azure: NoSchedule which means that tolerations need to be specified in PodSpec for the pod to be scheduled on virtual node. The pods which do not specify this toleration in PodSpec will not be scheduled on virtual node.
• Taints and tolerations are a flexible way to steer pods away from nodes or evict pods that shouldn’t be running. Taints and tolerations work together to ensure that pods are not scheduled onto inappropriate nodes. One or more taints are applied to a node; this marks that the node should not accept any pods that do not tolerate the taints. Tolerations are applied to pods, and allow (but do not require) the pods to schedule onto nodes with matching taints. Once a node is tainted pods that don’t declare a toleration for the taint won’t be scheduled to that node.
• Virtual Nodes have defaults for capacity
  • Virtual CPU limit is 800
  • Memory limit is 4 Ti
  • Number of PODS which can be scheduled is 800
• OS is Linux

Create a Kubernetes Deployment

Let’s create a Kubernetes deployment where pods will be scheduled on virtual node. The sample yaml(source: msdn) will schedule 3 instances of the pod and each of these instances is going to get scheduled on virtual node.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: aci-sample
spec:
  replicas: 3
  selector:
    matchLabels:
      app: aci-sample
  template:
    metadata:
      labels:
        app: aci-sample
    spec:
      containers:
      - name: aci-sample
        image: microsoft/aci-helloworld
        ports:
        - containerPort: 80
      nodeSelector:
        kubernetes.io/role: agent
        beta.kubernetes.io/os: linux
        type: virtual-kubelet
      tolerations:
      - key: virtual-kubelet.io/provider
        operator: Exists

This is possible because yaml file has couple of specifications (as explained in previous section)

• nodeSelector matches the labels defined for the virtual node.
• tolerations match the Taints defined for the virtual node and operator is Exists.
  • operator can be either Equal or Exists.
    • Equal: In this case you need to specify value and it must match the key value on the taint.
    • Exists: In this case toleration will match any taint with the specified key name.
  • Since effect is not specified, toleration will match any taint with any effect as long as the key and value match. 

Run command kubectl get pods -o wide to view pods and you can see that each of the pods were deployed to virtual nodes. Also the IP address of the pods is assigned from virtual nodes subnet.

Tip: If POD Status is ‘ProviderFailed’ then you may need to run command az provider register --namespace Microsoft.ContainerInstance to register namespace.

Tip: Run command az aks disable-addons --resource-group AKSDevGroup --name DevCluster --addons virtual-node to disable virtual nodes however it doesn’t delete virtual network resources.

Tip: Run commandaz aks enable-addons --resource-group AKSDevGroup --name DevCluster --addons virtual-node --subnet-name virtual-node-aci to re-enable Virtual Nodes assuming virtual network resources weren’t deleted.

Summary

This completes this article where I have shared steps on enabling Virtual Nodes feature of Azure Kubernetes Service (AKS) to scale the cluster.