Summary:In September of this year, the Istio community announced the open source of Ambient Mesh, which sparked heated discussions among many developers at home and abroad.

This article is shared from HUAWEI CLOUD Community “Deep Analysis! Ambient Mesh, a New Mode of Istio Shared Proxy”, by HUAWEI CLOUD Cloud Native Team.

In September of this year, the Istio community announced the open source of Ambient Mesh, which sparked heated discussions among many developers at home and abroad. In fact, by collaborating with Istio TOC member linsun(https://github.com/linsun) exchanges, we learned as early as 2021,http://Solo.ioalready startedThe research and design of shared agents are also exploring the shared agent model within Google in 2021. Therefore, the two companies hit it off and started to accelerate the development of the shared agency model through collaborative development from April to May this year.

At present, Ambient Mesh has released a preview version, and interested readers can experience it on demand. Due to space limitations, this article mainly conducts an in-depth analysis of the Ambient Mesh architecture and the four-layer traffic management process. For the detailed explanation of the seven-layer traffic management, please pay attention to the follow-up articles.

1. What is Ambient Mesh

Simply put, Ambient Mesh is a new mode of shared proxy for Istio service mesh.It is made by Google and http://Solo.io joint developmentA new data plane model for , designed to simplify operations, improve application compatibility, and reduce infrastructure costs. Ambient mode maintains Istio’s core features of zero-trust security, traffic management, and telemetry, without the introduction of Sidecar.

2. Ambient Mesh Architecture Analysis

Before starting Ambient’s architecture, let’s briefly review Istio’s architecture. It mainly consists of two parts, namely the control plane and the data plane. The control plane Istiod performs basic configuration generation and push, and manages all data planes; the sidecar proxy is introduced into the data plane to take over the ingress and egress traffic of the application.

Figure 1 Istio architecture

Compared to Sidecar, Ambient Mesh offers a less intrusive option with simpler upgrade management. Ambient divides Istio’s functionality into two distinct layers, a security overlay (four layers of governance) and a seven-layer processing layer (seven layers of governance):

Figure 2 Ambient mesh is divided into a layer

• Safety Overlay:Handling TCP routing, monitoring metrics, access logs, mTLS tunneling, simple authorization
• Seven processing layers:In addition to the functions of the security overlay, it provides HTTP protocol routing, monitoring, access logs, call chains, load balancing, circuit breakers, current limiting, retry and other traffic management functions, as well as rich seven-layer authorization policies

Loads under Ambient Mesh can seamlessly interoperate with loads under Sidecar mode, allowing users to mix and match the two modes according to their needs.

• Four-tier governance structure:In Sidecar mode, Istio implements traffic interception through InitContainer or Istio-CNI. Istio-CNI is a required component under Ambient Mesh. The following figure shows the basic four-layer governance structure of Ambient Mesh:

Figure 3 Ambient mesh four-layer governance architecture

In Istio-CNI, add the processing of Ambientmodule this module will monitor the changes of Namespace and Pod, and set routing and iptables rules for the application of the node:

• Routing:Set the routing table to route the traffic sent by the application of this node to ztunnel, and route the traffic received by this node to ztunnel.

• iptables:Set iptables rules in the ztunnel container to transparently intercept traffic to the port corresponding to ztunnel.

ztunnel is a newly introduced component of Ambient, which is deployed on each node in the form of Daemonset. ztunnel provides mTLS, telemetry, authentication, and L4 authorization for application communication in the mesh, and does not perform any Layer 7 protocol related processing. The control plane will only issue a workload certificate to a ztunnel if it is running on a node with the same workload. Therefore, when ztunnel is attacked, only the certificate of the load running on the node may be stolen, and the security risk is relatively controllable, which is similar to other well-implemented node sharing infrastructure.

• Seven-layer traffic governance architecture:Currently, Ambient Mesh needs to explicitly enable Layer 7 processing for a service by defining a Gateway API resource. The following figure shows the architecture of Ambient’s seven-layer governance:

Figure 4 Ambient mesh seven-layer governance architecture

Compared with Ambient’s four-layer governance, a waypoint component has been added to the seven-layer governance architecture. Added waypoint handling in Pilotmodule it listens for changes in ServiceAccount, Deployment, Gateway API objects, and then coordinates the related waypoint objects:

• When ServiceAccount changes, Pilot will try to update all waypoints in the current namespace
• When the Deployment changes, trigger the maintenance of the waypoint through the Gateway object associated with its OwnerReference
• Update the associated waypoint proxy when the Gateway changes

Currently Ambient needs to rely on Gateway API resources like the following to create a waypoint proxy:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
  name: productpage
  annotations:
    istio.io/service-account: bookinfo-productpage
spec:
 gatewayClassName: istio-mesh

The gatewayClassName value must be set to istio-mesh, otherwise it may be ignored. Each ServiceAccount has its own dedicated waypoint proxy, much like the Sidecar model. It is recommended that each service use its own separate identity to avoid additional security risks.
Pilot will update the Layer 7 and Layer 7 traffic rules to the waypoint proxy through xDS to implement Layer 7 related traffic governance capabilities. The waypoint proxy is not necessarily guaranteed to be on the same node as the workload it is serving, which seems to introduce certain performance issues. But for Istio, the delay is more from the complex seven-layer processing, and it is expected that the seven-layer governance delay of the final Ambient mode is close to that of the Sidecar mode. The waypoint agent is deployed through a separate Deployment, so it can be individually configured with the required CPU and memory, set the relevant HPA elastic scaling strategy, no longer coupled with the application, provide more flexible scalability and can improve the use of resources to a certain extent Rate.

3. Ambient Mesh four-layer traffic management

We know that ztunnel can only perform four-layer traffic management, four-layer load balancing, TLS traffic encryption, basic authentication and authentication, etc., but cannot perform more advanced seven-layer routing and authentication. Here we use the example of accessing bookinfo through the sleep application to gain an in-depth understanding of how the four-layer traffic of Ambient Mesh is routed. The actual environment background of this example is as follows. The sleep and productpage applications are running on two different nodes.

Figure 5 Ambient mesh four-layer traffic proxy process

• To access the productpage service in the sleep container, the request is first intercepted to the ztunnel of the same node, and the ztunnel performs basic four-layer load balancing and TLS encryption and decryption, and finally selects a target instance (the IP of the productpage container) to forward the request. .

• When this request enters the node where the productpage container is located, it is first intercepted by ztunnel, which is responsible for decrypting TLS traffic, then executes the authentication policy specified by the user, and finally sends the request to the productpage container.

The above is a basic process of Ambient mesh traffic forwarding. Let’s take a deep understanding of the complete communication process in combination with the specific xDS configuration.

3.1 Traffic processing on the sending side of sleep

(1) The traffic from sleep to access productpage is intercepted and forwarded to ztunnel (monitoring 127.0.0.1:15001) by the tunnel of the same node in TPROXY (transparent proxy) mode. The advantage of using TPROXY is to retain the original destination address, and ztunnel must rely on it for forwarding. Original destination address.

-A PREROUTING -i pistioout -p tcp -j TPROXY --on-port 15001 --on-ip 127.0.0.1 --tproxy-mark 0x400/0xfff

(2) ztunnel listens on port 15001 through the “ztunnel_outbound” listener. The ztunnel_outbound listener is completely different from the listener in Istio sidecar mode, it contains a filter chain from all services on this node to other services in the entire mesh.

Figure 6 ztunnel_outbound listener

It can be seen that no matching conditions are set for all filter chains (all matching by default), so how does ztunnel select the target filter chain according to the traffic characteristics? It turns out that there is also a way to set the filter matching conditions on the listener root. Through the following matching, the source address is 10.244.1.4, the destination address is 10.96.179.71, and the destination port is 9080. The traffic is handed over to “spiffe://cluster. local/ns/default/sa/sleep_to_http_productpage.default.svc.cluster.local_outbound_internal” filter processing,

Figure 7 ztunnel_outbound filter chain matching

(3) The “spiffe://cluster.local/ns/default/sa/sleep_to_http_productpage.default.svc.cluster.local_outbound_internal” filter is associated with the Cluster of the same name. The Cluster contains a total of two Endpoint instances. An Endpoint is selected according to the load balancing algorithm, and the most important thing is to pass the metadata (destination and address of the tunnel) to the name “outbound_tunnel_lis_spiffe://cluster.local/ns/default/sa” Listener handling for /bookinfo-productpage”.

Figure 8 outbound_internal internal Cluster configuration

Figure 9 outbound_internal internal Cluster Endpoint

(4) The “outbound_tunnel_lis_spiffe://cluster.local/ns/default/sa/sleep” listener uses the “set_dst_address” filter to set the destination address of the data according to the metadata of the endpoint selected in the previous step. If the previous outbound_internal Cluster selected the endpoint 10.244.2.8:9080, then the tunnel listener here will set 10.244.2.8:15008 as the destination address. In addition, the listener has only one TcpProxy, which is associated with the Cluster named “outbound_tunnel_clus_spiffe://cluster.local/ns/default/sa/sleep”, then the traffic is naturally handed over to the Cluster. An HTTP Connect tunnel (carrying traffic sent to 10.244.2.8:9080) is also set on the TCP filter for use by the ztunnel of the node where the productpage is located. HTTP tunneling is the bearer protocol for secure communication between Ambient Mesh components.

Figure 10 outbound_tunnel listener configuration

The type of outbound_tunnel Cluster is “ORIGINAL_DST”, and it is configured with UpstreamTlsContext, so it is responsible for TLS encryption of traffic, and then sent directly to the destination address, which is 10.244.2.8:15008.

Figure 11 outbound_tunnel cluster configuration

3.2 Productpage receiving side traffic processing

(1) The traffic of sleep accessing productpage (destination address is “10.244.2.8:15008”) reaches the node where productpage is located, and is intercepted by TPROXY (transparent proxy) to ztunnel (monitoring 127.0.0.1:15008), the benefits of using TPROXY It keeps the original destination address, and ztunnel must rely on the original destination address when forwarding.

10.244.2.8 via 192.168.126.2 dev istioin table 100 src 10.244.2.1
-A PREROUTING -i pistioin -p tcp -m tcp --dport 15008 -j TPROXY --on-port 15008 --on-ip 127.0.0.1 --tproxy-mark 0x400/0xfff

(2) The “ztunnel_inbound” listener on ztunnel listens on port 15008, so the traffic is first processed by the ztunnel_inbound listener. TLS is set on the ztunnel_inbound listener, and TLS handshake is performed with the downstream according to its configuration, so that all ztunnels communicate based on mutual TLS encryption. In addition, as you can see from the configuration below, the CONNECT upgrade has been set, so Envoy will proxy HTTP Connect requests. In addition, the connectMatcher is set in RouteMatch, which means that the HTTP Connect request is handed over to the “virtual_inbound” Cluster for processing.

Figure 12 ztunnel_inbound listener configuration

(3) The virtual_inbound Cluster type is ORIGINAL_DST, and it is set to usex-envoy-original-dst-host The HTTP Header rewrites the original destination address, and this Header is set by the “outbound_tunnel_lis_spiffe://cluster.local/ns/default/sa/sleep” listener on the sending side, with a value of 10.244.2.8:9080. Therefore, this request is finally successfully sent to the productpage container through virtual_inbound.

Figure 13 virtual_inbound cluster configuration

3.3 Ambient Mesh four-layer traffic management summary

Figure 14 Complete service access four-layer proxy

In the instance of sleep accessing the productpage, although we use the HTTP protocol, from the perspective of all Ambient components, the proxy is TCP traffic. Earlier, we deeply analyzed the working principle of each listener and each Cluster in ztunnel, which may seem complicated. Therefore, here is a brief summary through Figure 14. We find that in the process of communication, there are not many modules that actually participate in the work:

1. Routing and iptables on the sending side: intercept traffic to port 15001 of ztunnel
2. Sending side ztunnel: two listeners and two corresponding clusters
3. Routing and iptables on the receiving side: intercept traffic to port 15008 of ztunnel
4. Receive ztunnel: virtual_inbound listener and associated cluster

4. Future Outlook

Sidecar is a feature of Istio. Using Sidecar, you can enjoy the benefits of service mesh and reduce the burden of operation and maintenance by making very small modifications to the application. However, the Sidecar mode also has some limitations:

1. Intrusive: The sidecar container is injected in the way of Admission Webhook, and belongs to the same Pod as the application container. Therefore, the upgrade of the sidecar must be accompanied by the reconstruction of the business container. Can be disruptive to application load (eg, rolling upgrades can cause surges in long connection scenarios).
2. Low resource utilization: Sidecars correspond to applications one-to-one, and sufficient CPU and memory must be reserved, which may lead to low resource utilization of the entire cluster; elastic scaling can only be performed for the entire workload, and cannot be performed on sidecars alone.
3. Traffic disruption: Traffic capture and HTTP processing is done by Sidecar, which is expensive and may break some incompatible HTTP implementations.

At present, Ambient Mesh has well solved the deployment dependency problem of applications and sidecars in sidecar mode, and no longer needs to inject sidecars; the capabilities of service mesh are provided through ztunnel and waypoint proxy, and the deployment and upgrade of applications and mesh components is not easy. interdependent again.

In addition, the Ambient sharing mode can greatly reduce the resource overhead of the grid component itself, which is a huge boon for resource-sensitive users.

Ambient is still in preview, many features are still in development, in the officialDocumentationMany restrictions have been listed in , in addition, community users also have new discoveries in the process:

• Does not support IPV6

• Not compatible with Calico CNI, because iptables created by Ambient conflict with Calico.

At the same time, the current envoy-based ztunnel may have performance problems in xDS efficiency, multi-tenancy, and telemetry. In the future, a more lightweight and high-performance ztunnel may be rewritten based on rust.

In the long run, the Sidecar model will remain the mainstream model for Istio. The Ambient sharing model has brought enough stimulation to the Istio community or the service mesh industry. It is believed that based on the joint efforts of all developers in the community, the Ambient sharing model will become the second choice of Istio.

Click Follow to learn about HUAWEI CLOUD’s new technologies for the first time~