Summary:In the newly released version 1.3, Karmada has redesigned the cross-cluster failover function of applications, implemented a taint-based fault eviction mechanism, and provided a smooth failover process, which can effectively ensure the continuity of the service migration process (continuous service).

This article is shared from the HUAWEI CLOUD community “Karmada v1.3: More Elegant, Accurate and Efficient”, author: Cloud Container Future.

Karmada is an open multi-cloud and multi-cluster container orchestration engine designed to help users deploy and operate business applications in a multi-cloud environment. With the ability to be compatible with Kubernetes’ native API, Karmada can smoothly migrate single-cluster workloads, and still maintain collaboration with the ecological tool chain around Kubernetes.

In the newly released version 1.3, Karmada has redesigned the cross-cluster failover function of applications, implemented a taint-based failover mechanism, and provided a smooth failover process, which can effectively ensure the continuity of the service migration process (continuous service).

New features added in this version:

• A new feature of multi-cluster-oriented resource proxy has been added. Through this proxy platform, business parties can directly manipulate workloads deployed in multi-clusters in a single-cluster access posture without being aware of multi-clusters;
• Provides the ability to model cluster resources. Through the customized cluster resource model, the scheduler can schedule resources more accurately;
• Provides the ability to register Pull mode clusters based on Bootstrap tokens, which not only simplifies the cluster registration process, but also facilitates access control;

In addition, based on user feedback in the production environment, this version has also carried out many performance optimizations, which greatly reduce the CPU and memory resource requirements during system operation. The detailed performance test report will be released later.

Like the previous version, v1.3 remains compatible with the previous version, and users of the previous version can still upgrade smoothly.

Overview of new features

Stain-based graceful eviction

When the cluster is judged to fail, and the failure time exceeds the grace period (default 5 minutes), Karmada will add a “NoExecute” taint to the failed cluster, and then the newly introduced taint-manager controller will start to evict the workload on the failed cluster , and then the scheduler reschedules the evicted workload to a new available cluster. If the user enables the GracefulEviction feature, the evicted workload will not be deleted immediately, but will be delayed until the new workload runs to ensure the eviction. The business is not interrupted during the process.

The overall failover process can be expressed as: “cluster failure determination” → “load pre-eviction” → “rescheduling” → “cleaning up redundant loads”.

Here, regardless of fault determination or expulsion, users can control the parameters by referring to:

• –failover-eviction-timeoutspecifies the grace period for deleting the faulty cluster from the scheduling result, the default is 5 minutes
• –default-not-ready-toleration-secondsspecifying the tolerance time added by default to propagation policies that do not already have a notReady:NoExecute tolerance, default 600 seconds
• –default-unreachable-toleration-secondsspecifying the tolerance time added by default to propagation policies that do not yet have the unreachable:NoExecute tolerance, default 600 seconds
• –graceful-eviction-timeoutspecifies the timeout to wait for the graceful eviction controller to perform final deletion since the workload has moved to the graceful eviction task, the default is 10 minutes

Global proxy across multi-cluster resources

Karmada has added a “karmada-search” component in version 1.2, which is an optional component used to cache resource objects and events deployed in the cluster and provide external retrieval services through the search API.

In version 1.3, we introduced a new proxy function in this component, which allows users to access resources in multiple clusters by accessing a single cluster, no matter whether the resources are managed by Karmada or not, using the proxy function, users can unified through the Karmada control plane to operate the resources in the member cluster.

Users can use the ResourceRegistry API to specify the cached resource type and data source (target cluster). For example, the following configuration indicates that the Pod and Node resources are cached from the member1 and member2 clusters:

apiVersion: search.karmada.io/v1alpha1
kind: ResourceRegistry
metadata:
     name: proxy-sample
     spec:
         targetCluster:
        clusterNames:
        - member1
        - member2
        resourceSelectors:
        - apiVersion: v1
         kind: Pod
        - apiVersion: v1
         kind: Node

After submitting this configuration to karmada-apiserver, the URL: /apis/search.karmada.io/v1alpha1/proxying/karmada/proxy/api/v1/namespaces/default/pods can be used for cluster resource access. In this URL, /apis/search.karmada.io/v1alpha1/proxying/karmada/proxy is a fixed prefix, and the latter part is exactly the same as the Kubernetes native API path.

More information about this feature can be found at:https://karmada.io/docs/userguide/globalview/proxy-global-resource/

Scheduling based on custom cluster resource model

In the process of cluster scheduling, karmada-scheduler will make scheduling decisions based on a series of factors, one of which is the available resources of the cluster. In previous versions, Karmada adopted a common resource model, ResourceSummary, to abstract the availability of clusters, as follows:

resourceSummary:
   allocatable:
      cpu: "1024"
      memory: 4096Mi
      pods: "110"
   allocated:
      cpu: "512"
      memory: 2048Mi
      pods: "64"

However, ResourceSummary mechanically accumulates the resources of all nodes in the cluster, ignoring the fragmented resources on the nodes, which will cause Pods with large resource requirements to be unable to accurately schedule to the appropriate cluster. At the same time, it also ignores the fact that the resources that can be allocated by nodes in the clusters of different users are not exactly the same.

In version 1.3, Karmada introduced a new way – a custom cluster resource model to abstract the available resources of the cluster, aiming to make the scheduler’s result of scheduling the cluster more accurate. Users can enable the feature switch –CustomizedClusterResourceModeling to enable this feature. After enabling, after the cluster is managed by Karmada, Karmada will automatically set the default resource model for the cluster. This resource model divides each node in the cluster. For models with different levels, the default resource model will divide nodes into 9 different levels according to the two resource indicators of CPU and memory, as follows:

resourceModels:
- grade: 0
ranges:
- max: "1"
min: "0"
name: cpu
- max: 4Gi
min: "0"
name: memory
- grade: 1
ranges:
- max: "2"
min: "1"
name: cpu
- max: 16Gi
min: 4Gi
name: memory
.....
- grade: 8
ranges:
- max: "9223372036854775807"
min: "128"
name: cpu
- max: "9223372036854775807"
min: 1Ti
name: memory

Cluster-status-controller will collect node and Pod information in the cluster to calculate the number of corresponding model nodes. At the same time, karmada-scheduler compares the number of nodes that meet the requirements in different clusters according to the instance resource request to be scheduled, and schedules the instance. to a cluster with more nodes that meet the requirements.

At the same time, in some scenarios, the default cluster resource model cannot meet the needs of users’ specific clusters. Now users can set the resource model of the cluster by using the kubectl edit cluster command, so that the resource model can better fit the resource topology of the cluster. .

Bootstrap token based cluster registration

In version 1.3, for clusters in Pull mode, we provided a way to register with the Karmada control plane via the command line. Now through the karmadactl token command we can easily create a Bootstrap startup token token, the default valid duration of the token is 24 hours.

$ karmadactl token create --print-register-command --kubeconfig /etc/karmada/karmada-apiserver.config

# The example output is shown below
karmadactl register 10.10.x.x:32443 --token t2jgtm.9nybj0526mjw1jbf --discovery-token-ca-cert-hash sha256:f5a5a43869bb44577dba582e794c3e3750f2050d62f1b1dc80fd3d6a371b6ed4

Through the karmadactl register command, the registration process including deploying karmada-agent can be easily completed without duplicating the kubeconfig of the member cluster, which enhances the usability and security of the control plane to manage the member cluster in Pull mode.

$ karmadactl register 10.10.x.x:32443 --token t2jgtm.9nybj0526mjw1jbf --discovery-token-ca-cert-hash sha256:f5a5a43869bb44577dba582e794c3e3750f2050d62f1b1dc80fd3d6a371b6ed4

# The example output is shown below
[preflight] Running pre-flight checks
[prefligt] All pre-flight checks were passed
[karmada-agent-start] Waiting to perform the TLS Bootstrap
[karmada-agent-start] Waiting to construct karmada-agent kubeconfig
[karmada-agent-start] Waiting the necessary secret and RBAC
[karmada-agent-start] Waiting karmada-agent Deployment
W0825 11:03:12.167027 29336 check.go:52] pod: karmada-agent-5d659b4746-wn754 not ready. status: ContainerCreating
......
I0825 11:04:06.174110 29336 check.go:49] pod: karmada-agent-5d659b4746-wn754 is ready. status: Running

cluster(member3) is joined successfully

version upgrade

We verified the upgrade path from karmada version 1.2 to version 1.3, the upgrade process is smooth,Please refer to the upgrade documentation: https://karmada.io/docs/administrator/upgrading/v1.2-v1.3

Thanks to contributors

Karmada v1.3 includes hundreds of code commits from 51 contributors, and I would like to express my sincere thanks to each contributor:

Contributor GitHub ID:

@AllenZMC

@calvin0327

@carlory

@CharlesQQ

@Charlie17Li

@chaunceyjiang

@cutezhangq

@dapengJacky

@dddddai

@duanmengkk

@Fish-pro

@Garrybest

@gy95

@halfrost

@hanweisen

@huntsman-li

@ikaven1024

@joengjyu

@JoshuaAndrew

@kerthcet

@kevin-wangzefeng

@kinzhi

@likakuli

@lonelyCZ

@luoMonkeyKing

@maoyangLiu

@mathlsj

@mikeshng

@Momeaking

@mrlihanbo

@my-git9

@nuclearwu

@Poor12

@prodanlabs

@RainbowMango

@suwliang3

@TheStylite

@wawa0210

@weilaaa

@windsonsea

@wlp1153468871

@wuyingjun-lucky

@XiShanYongYe-Chang

@xuqianjins

@xyz2277

@yusank

@yy158775

@zgfh

@zhixian82

@zhuwint

@zirain

Reference link

● Release Notes:https://github.com/karmada-io/karmada/releases/tag/v1.3.0

● Cluster failover usage guide: https://karmada.io/docs/userguide/failover/#concept

● Guidelines for using the global agent for multi-cluster resources:https://karmada.io/docs/userguide/globalview/proxy-global-resource/

● Instructions for using the cluster resource model:https://karmada.io/docs/userguide/scheduling/cluster-resources

● Bootstrap token-based cluster registration usage guide: https://karmada.io/docs/userguide/clustermanager/cluster-registration