Latency Benchmarks: Istio, Linkerd, Consul

When choosing a service mesh for Kubernetes, latency and resource consumption are critical factors. Here's a quick summary of how Istio, Linkerd, and Consul compare:

• Linkerd: Lowest latency and resource usage. Adds 6ms median latency at 2,000 RPS and uses around 26MB memory per proxy. Ideal for performance-focused environments.
• Istio: Rich feature set but higher resource demands. Adds 15ms median latency at 2,000 RPS, with proxies consuming up to 150MB memory. Suited for complex, multi-platform deployments.
• Consul: Balanced option with moderate resource usage (30MB per proxy). Performs well under light loads but struggles with higher traffic.

Quick Comparison:

Key Takeaway:
Choose Linkerd for low-latency workloads, Istio for feature-rich environments, and Consul if you rely on HashiCorp tools or hybrid setups.

::: @figure {Service Mesh Performance Comparison: Istio vs Linkerd vs Consul Latency and Resource Benchmarks} :::

1. Istio

Latency Performance

Istio's Envoy proxy architecture offers robust traffic management, but it introduces some latency. At a load of 2,000 RPS, Istio adds about 15ms of median latency compared to a Kubernetes setup without a service mesh. The maximum latency under the same load can reach 253ms [2].

In larger deployments - such as a mesh with 1,000 services and 2,000 sidecars - the Envoy proxy adds approximately 2.65ms to the 90th percentile latency [1]. While this increase may seem minor, enabling advanced features like fault injection, traffic mirroring, and header-based routing adds more processing time. Additionally, using external authorisation providers like Open Policy Agent (OPA) introduces extra network hops, further increasing response times [3].

These latency figures highlight the trade-offs involved in using Istio, especially when combined with its resource demands under different workloads.

Resource Consumption

Istio's resource usage scales alongside traffic. Each Envoy proxy consumes around 0.35 vCPU and 40MB of memory per 1,000 RPS [1]. Meanwhile, the Istiod control plane component requires roughly 1 vCPU and 1.5GB of memory to manage large-scale deployments [1]. Under high load, individual Envoy proxies can consume up to 154.6MB of memory, while the control plane averages 837MB [2].

Istio is often perceived as 'heavy' primarily because it supports all kinds of workloads - K8s, public cloud, and on-prem VMs. - Zachary Butcher and Debasree Panda, Tetrate [1]

Istio's CPU usage is notably higher than other service meshes, consuming 2–3 times more [3]. Under certain configurations and loads of around 600 RPS, Istio may experience extreme tail latencies lasting several minutes. During these conditions, socket and HTTP error rates can range between 1% and 5.2%, and control plane components like Pilot have been observed to restart, which can impact overall stability [5].

Scalability

The resource-heavy design of Istio's Envoy and control plane components directly affects scalability. Default configurations often hit saturation at loads beyond 600 RPS [5]. These out-of-the-box settings aren't optimised for maximum performance. Disabling unused features - such as Mixer Policy, high levels of tracing, or specific telemetry components - can improve throughput and reduce control plane resource demands [5].

However, benchmarks show that performance degradation becomes significant as loads approach 600 RPS in certain setups. This includes latency spikes and increased error rates, demonstrating the importance of tuning Istio for specific environments [5].

2. Linkerd

Latency Performance

Linkerd stands out with its minimal latency overhead. At a traffic rate of 2,000 requests per second (RPS), it adds just 6ms of median latency to the baseline, compared to Istio's 17ms [6]. The difference is even more striking at the higher end of latency - Linkerd introduces only 42ms of additional maximum latency, while Istio adds a hefty 350ms under the same conditions [6].

This efficiency is largely due to Linkerd's specialised micro-proxy, Linkerd2-proxy, which is built with Rust and tailored specifically for service mesh sidecars rather than general-purpose proxying [2][1]. In more typical scenarios, such as workloads under 500 RPS, Linkerd adds between 0.5ms and 2ms to median latency. Even with mutual TLS (mTLS) enabled, there’s no noticeable latency penalty [3].

Linkerd not only remains dramatically faster than Istio, but now also consumes an order of magnitude less data plane memory and CPU while doing so. – William Morgan, CEO of Buoyant [2]

This low latency pairs well with its efficient use of resources, as explored further in the Resource Consumption section.

Resource Consumption

Linkerd’s resource efficiency is equally impressive. At 2,000 RPS, each Linkerd proxy uses just 26.3MB of memory - about one-sixth of Istio’s 156.2MB [6]. CPU usage follows a similar trend, with Linkerd consuming 36ms compared to Istio’s 67ms [6]. The control plane is also lighter, requiring 365MB of memory versus Istio’s 597MB [6].

The release of Linkerd v2.11.1 brought a slight increase in proxy memory usage (from 18MB in v2.10) due to the adoption of jemalloc for handling spiky loads [6]. Despite this, the typical resource overhead per sidecar remains modest, averaging around 10MB [4].

Scalability

The benchmarks for latency and resource use highlight Linkerd’s ability to maintain steady performance as traffic scales. Unlike Envoy-based service meshes, which can experience significant latency spikes under heavy loads, Linkerd keeps tail latency increases minimal, even at the 99th percentile [6][2]. For instance, at 600 RPS, Linkerd’s p99 latency was just 7ms, whereas Istio’s p99 latency stretched to several minutes in a similar environment [7].

By solving the very specific problem of being a 'service mesh sidecar proxy' only, we can be extremely efficient at the data plane level. – William Morgan, CEO of Buoyant [6]

Even at 2,000 RPS, Linkerd’s control plane remains efficient, consuming only 212ms of CPU compared to Istio’s 5 seconds [6]. This makes Linkerd an excellent choice for latency-sensitive applications where reducing proxy overhead is a priority [4].

3. Consul

Latency Performance

Consul uses Envoy for its data plane, which helps it perform well under light loads [8][4]. At a moderate 200 RPS, it introduces a median latency of 0.7–1.5ms. However, as the load increases to 700–1,200 RPS, latency grows significantly, reaching 5–50ms. This happens because the Envoy sidecar intercepts traffic multiple times [3]. Compared to Linkerd and Istio, Consul's latency grows more noticeably under higher loads. While Istio experiences more pronounced latency spikes and Linkerd keeps overhead minimal, Consul offers a middle ground, with moderate latency increases under pressure.

Resource Consumption

Each Consul sidecar proxy consumes around 30MB of memory. This is higher than Linkerd's lightweight 10MB but less than Istio's more demanding 50MB [4]. Its decentralised architecture, relying on distributed agents and a gossip protocol for service discovery, ensures that the workload is spread across nodes effectively [8].

Scalability

Consul's decentralised design enables horizontal scaling across a large number of services and nodes [8]. Its distributed agent model adjusts well to dynamic cluster conditions, making it ideal for organisations that need to mesh services across both Kubernetes and non-Kubernetes setups, including virtual machines or bare metal servers [4][1].

However, when it comes to latency, Consul's scaling isn't as predictable as Linkerd's. While enabling mTLS does not add noticeable latency beyond Envoy's inherent overhead [3], the performance gap becomes more evident under heavy, sustained loads. For applications that are highly sensitive to latency and operate above 200 RPS, this trade-off can be a concern. That said, Consul integrates seamlessly with HashiCorp tools like Vault and Nomad, making it a strong choice for HashiCorp-focused environments [4]. These characteristics provide a well-rounded view for evaluating its strengths and limitations in the upcoming analysis.

Benchmarking Journey Through Service Meshes - Dominik Táskai, adesso Hungary

Pros and Cons

Analysing the benchmarks reveals the key strengths and weaknesses of the three service meshes. Here's a breakdown of the trade-offs based on performance data.

Linkerd stands out for its efficiency, requiring just a fraction of the memory compared to Istio - about one-ninth per proxy at 2,000 RPS. Its lightweight Rust-based microproxy uses only 10ms of CPU time, a stark contrast to Istio's Envoy proxy, which needs 88ms [2].

Istio boasts the most extensive feature set, supporting Kubernetes, virtual machines, and multi-cloud setups. It includes advanced features like fault injection and header-based routing. However, this versatility comes with hefty resource demands - its control plane (Istiod) can consume up to 1.5GB of memory and 1 vCPU to manage larger deployments [1]. As noted by Zachary Butcher and Debasree Panda from Tetrate:

Istio is the most capable, most flexible, and most widely used service mesh... Consul and Linkerd are designed to be lighter-weight and are not as widely supported by the community [1].

Consul strikes a middle ground, using about 30MB per sidecar and integrating seamlessly with HashiCorp tools like Vault and Nomad [4]. However, its performance struggles under heavier loads. At 1,200 RPS, its P50 latency jumps to 55.5ms, and its P99 latency reaches 125.5ms. In comparison, Linkerd maintains a P50 latency of 16.5ms and a P99 of 38.5ms, while Istio delivers 18.5ms (P50) and 45.5ms (P99) [3].

For enterprise-level features and broad platform support, Istio is a strong choice, though it demands more resources. Linkerd excels in environments where low resource usage and latency are critical. Consul, while offering good integration with HashiCorp tools, may struggle in high-traffic scenarios beyond 200 RPS. These trade-offs are crucial when deciding which service mesh best suits your needs.

Conclusion

The benchmarks in this analysis highlight the strengths and trade-offs of the three service meshes, each tailored to different deployment needs.

Linkerd stands out for organisations that value low latency and efficient resource usage. Thanks to its lightweight Rust-based microproxy, it maintains minimal overhead even under varying loads. This makes it a strong choice for applications where speed and resource efficiency are critical.

Istio shines when advanced features and multi-platform compatibility are key. While it demands more resources, it compensates with a rich feature set and strong community backing. It’s particularly well-suited for enterprises needing sophisticated traffic management and support across diverse platforms.

Consul is ideal for teams heavily invested in HashiCorp tools or managing hybrid environments. It handles moderate loads effectively and excels in bridging Kubernetes with legacy VM infrastructures. However, its performance declines more noticeably beyond 200 RPS.

Choosing the right service mesh depends on your priorities. Linkerd offers top-notch performance and efficiency, Istio provides unmatched functionality and flexibility, and Consul delivers seamless integration within the HashiCorp ecosystem. Weighing these factors alongside the benchmarks shared here will help you align your choice with your specific infrastructure and operational goals.

FAQs