By the end of this post, you'll know:

• What problem Kmesh is solving (and why that problem matters)

• How the current solutions work (and why they're slow)

• What Kmesh does differently (and why it's faster)

• How all the pieces fit together

What is a Service Mesh and Why Does It Exist?​

You must have deployed apps on Kubernetes. You know that your application runs inside a Pod, and that Pods talk to each other through Services. You write a YAML file, run kubectl apply, and your app is running.

Now imagine you're not deploying one app. You're deploying fifty. Imagine you are an engineer at a fast-growing e-commerce company. Your team started with a monolithic application—a single, large codebase handling user requests, inventory, and payments. As the company grew, you migrated to a microservices architecture with separate services for:

• User Service (handles user accounts)
• Order Service (processes orders)
• Inventory Service (tracks stock)
• Payment Service (handles payments)

At first, everything runs smoothly. But as complexity increases, several massive problems emerge:

• Authentication & Authorization is duplicated. Every microservice has to handle logins and permissions separately. Your developers are writing the same security logic over and over again for the Order Service, the Inventory Service, and the Payment Service.
• Failures become tougher to debug. Logs indicate errors in the Payment Service, but it's unclear whether these issues are from network glitches, load spikes, or faulty service updates. There is no central visibility.
• Unencrypted Security Risks. Sensitive data, like payment details, is being transmitted between your services unencrypted. Without proper encryption (like mTLS), any breach in your cluster becomes a major risk.
• Dumb Load Balancing. You have three replicas of your Payment Service. The Order Service is overwhelmed while the Inventory Service remains underutilized. Standard Kubernetes traffic distribution isn't smart enough to handle complex routing.
• Complex Canary Deployments. Let's say you want to release a new version of the Payment Service. Ideally, you want to route exactly 10% of your live traffic to the new version to test it out, but doing so without downtime is significantly challenging.

Every production team eventually faces:

"Who is making sure all of this communication works properly?"

This growing complexity makes managing your microservices painful. Several solutions have been proposed to address this.

How the Industry Tried to Fix It​

These problems weren't unique to your company. Every engineering team building microservices ran into the same wall. And the industry didn't solve it overnight—it went through three distinct eras of trying.

Think of it like this: the networking logic that keeps your services talking to each other had to live somewhere. The question was always—where?

Era 1: The Big Box in the Middle (Hardware Load Balancers)​

The earliest answer was straightforward: put a big, expensive box between your clients and your servers.

If you've been around infrastructure long enough, you've heard names like F5 BIG-IP, or maybe you've set up NGINX or HAProxy as a reverse proxy. The idea was simple—all traffic enters through one central point, and that point decides which backend server gets the request.

This worked fine when you had a handful of backend servers. Your company had maybe 3-5 services, and a load balancer in front of them was perfectly manageable.

But here's where it breaks down with microservices:

• It only handles North-South traffic. A load balancer sits at the edge. It distributes traffic coming into your cluster. But what about Service A calling Service B calling Service C inside the cluster? That's East-West traffic, and the load balancer doesn't even see it.
• It becomes a single point of failure. Every request flows through one box. If that box goes down, everything goes down.
• It doesn't understand your application. A load balancer can do round-robin or weighted routing, sure. But it doesn't know how to retry a failed request, or detect that one of your services is unhealthy and should be pulled from rotation automatically.

As the number of microservices grew from 5 to 50 to 500, pointing everything at one big box in the middle simply stopped making sense. The networking logic needed to move closer to the services themselves.

Era 2: Bake It Into the Code (Smart Libraries)​

Netflix was one of the first companies to really feel the pain of running hundreds of microservices. And their answer was pretty direct: if the infrastructure can't handle this, let's handle it in the application code itself.

They built a set of open-source Java libraries — collectively called Netflix OSS — that developers would import directly into their services. Think of it like adding an extra package to your project, except this package handled all the networking headaches for you:

• How do I find another service? → Eureka (service discovery)
• What if that service is down? → Hystrix (stop calling a broken service, don't let it drag you down too)
• Which instance should I send my request to? → Ribbon (pick the healthiest one)
• How should I route incoming traffic? → Zuul (smart routing)

Every microservice now carried its own networking smarts built right into the application code.

Netflix scaled to hundreds of microservices this way. Other companies followed — Twitter built Finagle, Google had internal equivalents.

But as teams adopted this approach, cracks started showing:

• Language lock-in. Netflix OSS was Java. If your ML team writes Python and your frontend team uses Node.js, you'd have to rewrite all that networking logic from scratch for each language. Not realistic.
• Upgrade nightmare. Found a bug in one of these libraries? You now need to update it in every single microservice, rebuild, test, and redeploy each one. With 200 services, that's not a quick patch — that's a week-long campaign.
• Your developers didn't sign up for this. They wanted to build a payment system, not debug why the load balancing library is sending traffic to a dead instance. Business logic and networking plumbing were tangled together in the same codebase.

The big takeaway from this era was clear: the networking logic needs to live outside the application code. Developers should focus on business logic. Something else should handle the networking.

But that "something else" shouldn't be one big box in the middle either. What if you could pull the networking logic out of the code and run it right next to the application — but as its own separate process?

Era 3: The Sidecar Proxy (The Birth of the Service Mesh)​

Instead of embedding networking logic inside your application, you deploy a small, lightweight proxy alongside every instance of your service. In Kubernetes terms, this proxy runs as a second container inside the same Pod. The most popular one is Envoy, originally built at Lyft.

Here's how it works. Back to your e-commerce example:

1. Your Order Service wants to call the Payment Service. It doesn't call it directly — the request gets transparently intercepted (via iptables rules) and redirected to the Envoy sidecar running in the same Pod.
2. The Envoy sidecar applies all the networking logic (load balancing, retries, mTLS encryption, timeouts) and forwards the request to the Payment Service's Pod.
3. There, the request hits the Envoy sidecar inside the Payment Service Pod first (rate limiting, auth checks), and only then reaches your actual Payment Service code.

Here's Istio's own architecture diagram showing this in action — the Data Plane (proxies) at the top and the Control Plane (Istiod) at the bottom:

And here's a more detailed view of the Service Mesh architecture with sidecar proxies managing all mesh traffic between pods:

Why was this a breakthrough?

• Language independence: Order Service in Java, Payment Service in Go? Doesn't matter — the same Envoy proxy handles networking for all of them.
• No code changes: Developers write plain business logic. No networking libraries to import. The proxy handles everything transparently.
• Centralized control: All sidecars get their config from a Control Plane (like Istio's Istiod). Want mTLS everywhere? One policy change, pushed to every sidecar. No redeployments.
• Uniform observability: Every request flows through a sidecar, so you get metrics, traces, and logs for all service-to-service traffic automatically.

This is what a Service Mesh is:

A service mesh is an infrastructure layer that manages service-to-service communication. It consists of a Data Plane (the sidecar proxies handling the actual traffic) and a Control Plane (the central brain configuring all the proxies).

Projects like Istio, Linkerd, and Consul Connect implement this pattern. Authentication? The mesh handles it. Encryption? mTLS everywhere. Canary/test deployments? Route 10% of traffic to the new version with just one config change.

So If the Service Mesh Solved Everything, Why Are We Still Talking?​

Because it introduced a new problem. A performance problem.

Instead of Service A talking directly to Service B, the request bounces through four extra network hops: out of Service A's container → into Service A's sidecar → across the network → into Service B's sidecar → into Service B's container.

All that bouncing means data gets copied back and forth between user space and kernel space multiple times. It means extra TCP connections get established. It means more CPU cycles, more memory, and more latency.

According to Istio's own benchmarks, a single Envoy proxy adds approximately 2.65 milliseconds of latency at the 90th percentile, and this overhead compounds as requests traverse multiple proxies/hops.

But what if the networking logic didn't have to run in user space at all?

That question is what leads us to eBPF and Kmesh — which we'll cover in the next section.

Why Traditional Service Meshes Are Slow​

To understand why, we need to understand how your operating system handles memory.

The Wall Between User Space and Kernel Space​

Linux splits its memory into two strictly isolated zones:

Think of User Space as the safe playground where your application, your database, and your Envoy proxy live. If an app crashes here, it's isolated. No big deal.

Kernel Space, on the other hand, is the engine room. It has direct access to the hardware (network cards, disks, CPU). It's incredibly fast, but if something crashes here, the whole server goes down. Because of this risk, programs in User Space are completely walled off from the hardware. If your app wants to send a network packet, it has to politely ask the Kernel to do it via a "system call."

Here is the golden rule of OS performance: Crossing that boundary between User Space and Kernel Space is expensive. It costs CPU cycles, memory copies, and time.

The Bouncing Problem​

Without a service mesh, two microservices talking to each other cross that boundary exactly twice. App A sends a request down to the kernel, the kernel ships it across the network, and the receiving kernel passes it up to App B. Simple.

But watch what happens to that exact same request when we drop an Envoy sidecar proxy into the mix:

Notice how the request is suddenly playing ping-pong across the system call boundary?

Instead of a clean, direct path, the traffic is intercepted by iptables, forced up into the Envoy sidecar, processed, and sent back down into the kernel. Then, the exact same steps happen on the receiving end.

What used to be 2 boundary crossings has exploded into 6 boundary crossings for a single request.

Where is the Time Actually Going?​

You might be thinking, "Sure, but Envoy is doing a lot of valuable work like mTLS and intelligent routing!"

You'd be right, but take a look at this breakdown of where the sidecar latency actually comes from:

This is the harsh truth of traditional service meshes: 90% of the overhead is just plumbing. Only a tiny sliver of the time (10%) is actually spent doing the traffic governance we bought the service mesh for in the first place! The rest is just the tax we pay for bouncing data back and forth between the kernel and our user-space proxy.

And the toll adds up. According to Istio's official benchmarks, a single Envoy proxy adds 2.65 milliseconds of latency to the 90th percentile:

But, what if we could just run our networking logic directly inside the kernel, right where the traffic is already flowing?

That is done via eBPF.

What is eBPF?​

eBPF (extended Berkeley Packet Filter) allows you to run your own custom programs safely inside the Linux kernel, without having to change the kernel source code or reboot the machine.

How Does That Even Work?​

Imagine the Linux kernel is a highly secure bank vault. Before eBPF, if you wanted to change how the vault operated, you had to rewrite the blueprint and rebuild the vault (writing a kernel module). It was slow, dangerous, and one mistake could blow the whole thing up.

eBPF changes the game. It gives you a way to slide instructions under the vault door.

As you can see in the diagram, you write your eBPF program, and a built-in Verifier checks it to ensure it won't crash or get stuck in an infinite loop. Once it passes the security check, the kernel compiles it and attaches it to specific "hooks" on the network path.

Your custom code is now running safely inside the vault, at native wire speed.

Why This Changes Everything for Service Meshes​

A hook is just a checkpoint in the kernel's code where eBPF programs can be attached. Each one works at a specific stage of the packet's journey, letting you inspect, modify, or redirect traffic without touching the kernel source.

With eBPF, we can short-circuit that entire journey. Look at the bottom hook in the diagram above — SK_MSG. It lets an eBPF program grab traffic right at the source and redirect it straight to its destination, without ever leaving the kernel. No detour through user space. No bouncing.

Think of it like this: instead of your package going through six different post offices (the bouncing problem), eBPF lets a worker inside the sorting facility hand it directly to the right truck. Same building. No unnecessary trips.

This is the missing piece. If we can run our routing and load balancing logic as eBPF programs inside the kernel, we don't need sidecar proxies sitting in user space anymore.

And that is exactly what Kmesh does. Let's dive into it.

How Kmesh Works​

Now that we understand eBPF, the idea behind Kmesh becomes almost obvious: move the traffic governance logic from user-space sidecar proxies into the kernel using eBPF.

From 3 Hops to 1 Hop​

In a traditional service mesh, every request between two services takes a route through three separate connections:

App A → Envoy Sidecar A → Envoy Sidecar B → App B

Three hops. Three TCP connections. Six boundary crossings between user space and kernel space. We've seen why that's expensive.

Kmesh removes the sidecars entirely:

On the left, you can see the traditional approach where traffic bounces up to sidecar proxies for governance, then back down through the OS. On the right, Kmesh moves that governance logic directly into the OS layer using eBPF. The result is a single, clean hop:

App A → Kernel (Kmesh eBPF) → App B

One connection. Two boundary crossings. The routing decision, the load balancing, the traffic policy — it all happens inside the kernel, right on the natural path the packet was already taking.

How Does Kmesh Know Where to Route Traffic?​

Kmesh doesn't reinvent any of the control plane logic. If you're already using Istio, defining your traffic rules works exactly the same way. Same YAML files, same routing policies, same canary deployments.

As the diagram shows, a lightweight component called kmesh-daemon runs on each node in your cluster. Its job is:

1. Listening to Istio's control plane for routing rules via xDS (a standard protocol that Istio uses to push configuration to its data plane) — both Envoy and Kmesh use xDS.
2. Translate those rules into eBPF programs.
3. Load them into the kernel.

Once those eBPF programs are in the kernel, traffic between Pod A and Pod B is governed at wire speed — no proxy containers, no extra pods, no resource overhead per service.

What About Complex HTTP Routing? (L4 vs L7)​

Not all traffic is equal, and this is where Kmesh gets smart about it.

Layer 4 (L4) traffic — basic TCP connections, load balancing, connection management — is perfect for eBPF. It's straightforward enough that eBPF handles it entirely inside the kernel with zero overhead.

But Layer 7 (L7) traffic is a different beast. Think HTTP header-based routing ("send requests with header X-Version: v2 to the canary deployment"), complex retry policies, or request-level authentication. These need deep packet inspection that goes beyond what eBPF programs can efficiently do today.

Kmesh solves this with a smart split:

As you can see, most of your traffic (L4) takes the fast path — handled entirely in the kernel by eBPF with zero proxy overhead. Only the traffic that genuinely needs deep HTTP inspection gets routed to a shared Waypoint Proxy.

The key difference from traditional meshes: instead of every pod paying the sidecar tax, only the small slice of traffic that actually needs L7 processing touches a proxy. Everything else goes through the kernel.

Kmesh vs Traditional Mesh​

Let's get to the numbers. This performance chart from the Kmesh project tells the whole story:

Look at what happens as connections scale up. The blue line (raw Kubernetes, no mesh) and the yellow line (Kmesh) stay nearly flat — Kmesh adds almost zero overhead compared to having no mesh at all. The orange line (Istio with Envoy sidecars) shoots up dramatically.

Here's the full comparison:

When to use Kmesh: You care about latency or you're running a modern Linux kernel (5.10+), and you want the benefits of a service mesh without the resource overhead.

When to stick with traditional sidecars: You're on older kernels that don't support the required eBPF features, or you need every single request to go through deep L7 processing.

In Short​

Traditional service meshes use sidecar proxies that bounce traffic between user space and kernel space — adding latency and eating resources. Kmesh uses eBPF to handle traffic governance directly inside the Linux kernel. Same features (load balancing, mTLS, routing), but without the sidecar overhead.

Getting Started​

Kmesh installs via Helm on any Kubernetes cluster with a modern Linux kernel. If you're already running Istio, Kmesh plugs right in as an alternative data plane — your existing traffic policies keep working.

• 👉 Quick Start Guide — Get Kmesh running in under 5 minutes
• 👉 GitHub Repository — Star the project and explore the code
• 👉 Community Slack — Say hi, ask questions, find your first issue

Kmesh is a CNCF Sandbox project and actively welcomes contributors. Whether it's fixing a bug, improving documentation, or just exploring the codebase — every contribution matters.

If you found this useful, share it with someone who's new to cloud-native networking — and come build with us.

Hello everyone! I’m Yash Israni, an open-source enthusiast passionate about automation, DevOps practices, and building tools that eliminate repetitive manual work.

This summer, I had the privilege of participating in the Open-Source Promotion Plan (OSPP) 2025, where I collaborated with the Kmesh community to automate documentation and release workflows. Over the course of three months, I designed and implemented GitHub Actions pipelines that keep the Kmesh website always up-to-date, properly versioned, and reviewed for language quality.

In this blog, I’ll share my journey—from acceptance to project execution, the technical decisions I made, and the lessons I learned along the way.

OSPP Program – Overview​

The Open-Source Promotion Plan (OSPP), organized by the Institute of Software, Chinese Academy of Sciences (ISCAS), gives students and early-career contributors the opportunity to gain hands-on experience by working on impactful open-source projects under the guidance of mentors.

Each term runs for about three months (1 July – 30 September in my case). Contributors not only deliver real-world features but also learn how large open-source communities operate.

My Acceptance​

I have always enjoyed contributing to open source, and my interests naturally align with automation and cloud-native tooling. When I saw that Kmesh was offering projects under OSPP 2025, I was immediately drawn to their proposal for automating documentation workflows.

The project addressed a clear pain point: documentation updates and versioning were being done manually, often lagging behind releases. The opportunity to replace repetitive tasks with reliable automation felt both impactful and challenging.

I received my acceptance email on 28 June 2025, and the program officially ran from 1 July to 30 September.

Interestingly, I was able to complete the majority of my project work before the mid-term evaluation, so that checkpoint was skipped, giving me extra time to refine the workflows and write proper usage guidelines.

Project Workthrough​

1. Doc-Sync Workflow​

• Trigger: on every push to the main branch
• Action: opens a pull request in the website repository with the latest documentation updates
• Enhancements: automatically labels the PR for triage and runs the site’s CI pipeline to validate changes

2. Release Versioning Workflow​

• Trigger: when a new Git tag is pushed (release event)
• Action: generates a versioned snapshot of the documentation in the website repository
• Enhancements: automatically opens a PR for any versioning-related changes

3. Chinese Grammar Checker Workflow​

• Trigger: on pull requests that modify Chinese documentation
• Action: uses the LanguageTool API to detect grammar and style issues
• Enhancements: posts line-level review comments as warnings (non-blocking) so contributors receive suggestions without being blocked from merging

Results​

These workflows have effectively eliminated delays and manual errors, ensuring Kmesh documentation stays accurate and up-to-date.

All three workflows are now live in both the Kmesh main repository and website repository under .github/workflows.

Key Technical Decisions​

• Adopted repository dispatch for secure cross-repo communication, eliminating the need for long-lived personal tokens
• Granted the GitHub Actions token read & write permissions only where necessary, while delegating other operations to a scoped bot account for better security
• Implemented Docusaurus-compatible versioning by dynamically generating versions.json, keeping navigation in sync with releases
• Added robust error handling in the doc-sync workflow to gracefully manage missing folders or files, preventing workflow crashes

Mentorship Experience​

My mentors, Li Zhencheng and Zhonghu Xu, along with the Kmesh maintainers, were consistently supportive—whether through GitHub reviews or quick clarifications on Slack. Even though I delivered my main workflows ahead of schedule, their feedback helped me refine edge cases and improve overall reliability.

As a recognition of my contributions and active involvement, the Kmesh community welcomed me as a member of the organization. This acknowledgment was both humbling and motivating, and it strengthened my commitment to continue contributing to Kmesh and supporting its growth.

Lessons Learned​

1. Automation empowers humans – the goal isn’t to replace contributors but to free them from repetitive tasks so they can focus on meaningful reviews and design.
2. Start small and iterate – building workflows in incremental, testable steps made debugging and maintenance far easier than deploying everything at once.
3. Security matters – applying the principle of least privilege to tokens and permissions reduced risk while keeping automation safe.
4. Expect edge cases – workflows behave differently across environments; testing on forks and multiple platforms prevented surprises in production.
5. Documentation is part of the code – writing clear workflow descriptions and PR comments ensured maintainers trusted and understood what the automation was doing.

Acknowledgements​

I would like to sincerely thank my mentors Li Zhencheng and Zhonghu Xu for their guidance, quick reviews, and encouragement. Thanks also to the OSPP program staff for ensuring smooth operations throughout the term.

Links​

• Project issue & Pull requests
• OSPP website
• Yash Israni's github

Hello everyone! I'm Wu Xi, an open source enthusiast with deep interests in kernel networking, eBPF, and test engineering.

This summer, I had the privilege to participate in Open Source Promotion Plan (OSPP) 2025 and collaborate with the Kmesh community, focusing on eBPF program UT enhancement. Over three months, I primarily completed unit testing work for Kmesh eBPF programs. I wrote and successfully ran UT test code for sendMsg and cgroup programs, and supplemented testing documentation based on this work. Kmesh community developers can now verify eBPF program logic without depending on real kernel mounting and traffic simulation, significantly improving development efficiency. In this blog, I'll share my complete experience—from acceptance to project execution, technical choices, and lessons learned along the way.

OSPP Project Overview​

Open Source Promotion Plan (OSPP) is organized by the Institute of Software, Chinese Academy of Sciences (ISCAS), providing students and early-career developers with opportunities to collaborate on real open source projects under the guidance of experienced mentors.

Each session lasts approximately three months (my session was July 1st – September 30th). Participants not only deliver functional features but also experience firsthand how large open source communities operate.

My Acceptance Experience​

I've always enjoyed contributing to open source, and my interests happen to focus on network kernels and cloud-native tools. When I saw the "eBPF" and "unit testing" related topics offered by Kmesh in OSPP 2025, I was immediately attracted.

The pain points this project aimed to solve were very clear: eBPF program verification has long relied on black-box testing, which is not only inefficient but also has coverage that depends on testers' experience. By introducing a unit testing framework and supplementing key use cases, functional verification can be completed without requiring real kernel mounting, which is both valuable and challenging.

I received my acceptance email on June 28, 2025, with the official project cycle running from July 1st to September 30th.

Interestingly, I completed the main work of the project before the mid-term evaluation, so that stage was skipped. This gave me more time to refine the workflow and write usage documentation.

Project Work Content​

1. eBPF Unit Testing Framework Construction​

• Core Technology: eBPF kernel function simulation based on #define mock macro replacement
• Test Coverage: Covers sendmsg TLV encoding, cgroup sock connection management, cgroup skb traffic processing
• Innovation: Embedding test infrastructure in production code through conditional compilation #ifdef KMESH_UNIT_TEST

2. sendmsg TLV Encoding Verification​

• Test Objective: Verify correctness of TLV metadata encoding in waypoint scenarios
• Test Data: IPv4 (8.8.8.8:53) and IPv6 (fc00:dead:beef🔢🔡53) simulation data
• Verification Mechanism: Real-time parsing of TLV message format, verifying integrity of type, length, IP, and port

3. cgroup Lifecycle Management Testing​

• Hook Coverage: cgroup/connect4, cgroup/connect6, cgroup/sendmsg4, cgroup/recvmsg4
• Test Scenarios: kmesh management process registration/deregistration, backend connections without waypoint, tail call mechanism
• Verification Method: Verify netns cookie management correctness through km_manage map state changes

Project Results​

These testing frameworks effectively eliminated blind spots in eBPF program testing, ensuring the stability and correctness of Kmesh's data plane.

Currently, the testing framework has been integrated into the CI/CD pipeline, allowing execution of the complete eBPF unit test suite through the make run command, covering core components like workload, XDP, sockops, sendmsg, cgroup_skb, and cgroup_sock.

Key Technical Decisions​

• Used define mock for function replacement, replacing eBPF kernel functions at compile time through macro definitions like #define bpf_sk_storage_get mock_bpf_sk_storage_get, achieving dependency isolation in unit tests
• Adopted conditional compilation test infrastructure, embedding test-specific map definitions and data structures in production code through #ifdef KMESH_UNIT_TEST macros, ensuring consistency between test and production code
• Used Go + eBPF hybrid testing framework, combining C language eBPF program compilation with Go language test execution, implementing automated testing workflow through go test -v ./...

Mentor Guidance Experience​

My mentors Li Zhencheng and Xu Zhonghu, along with other Kmesh maintainers, provided tremendous support throughout the UT testing framework development process.

They not only patiently pointed out improvements in test design during GitHub reviews but also quickly answered my questions about bpf helper mocking and map validation on Slack.

Although I completed the core UT for sendMsg and cgroup programs relatively early, mentor feedback helped me notice more edge cases and pushed me to further improve test coverage and documentation.

Finally, the Kmesh community invited me to become an organization member as recognition of my contributions and active participation. This not only made me feel humble but also strengthened my determination to continue participating in and supporting Kmesh's development.

Lessons Learned​

1. Unit testing is a tool to enhance development efficiency — It doesn't replace black-box testing but complements and frees developers, allowing them to focus faster on feature implementation and optimization.
2. Start small and iterate gradually — First supplement UT for core eBPF programs (like sendMsg, cgroup_skb), then gradually expand to more scenarios, which is more stable than covering all logic at once.
3. Anticipate edge cases — eBPF programs may behave differently across kernel versions or environments; simulating various inputs and exceptions in UT in advance helps avoid production environment surprises.
4. Communication can accelerate learning progress — Every time I submitted a PR, mentors would comment with better solutions or questions, which taught me a lot in a short time.
5. Facing challenges head-on is the recipe for progress — When learning fields you're interested in but haven't had much exposure to, setbacks are inevitable. Don't give up at these times; believe in yourself and keep trying—you'll eventually find solutions to problems.

Acknowledgments​

I want to sincerely thank my mentors Li Zhencheng and Xu Zhonghu throughout the process. In every community meeting, they would actively solve my current problems and understand my progress. Whenever I submitted a PR, they would share their insights in the comments, making my thinking clearer and improving my problem-solving abilities. I also want to thank the OSPP organizing committee for providing us with a smoothly running environment. This open source participation was an extraordinary experience for me, and I will continue to dedicate myself to open source and love open source!

Related Links​

• Project Issue & Pull Requests
• OSPP Official Website
• Wu Xi's GitHub

Hello readers, I am Yash, a final Year student from India. I love building cool stuffs and solving real world problems. I’ve been working in the cloud-native space for the past three years, exploring technologies like Kubernetes, Cilium, Istio, and more.

I successfully completed my mentorship with Kmesh during the LFX 2025 Term-1 program, which was an enriching and invaluable experience. Over the past three months, I gained significant knowledge and hands-on experience while contributing to the project. In this blog, I’ve documented my mentorship journey and the work I accomplished as a mentee.

LFX Mentorship Program – Overview​

The LFX Mentorship Program, run by the Linux Foundation, is designed to help students and early-career professionals gain hands-on experience in open source development by working on real-world projects under the guidance of experienced mentors

Participants contribute to high-impact projects hosted by foundations like CNCF, LF AI, LF Edge, and more. The program typically runs in 3 terms throughout the year, each lasting about three months.

More-info

My Acceptance​

I am a regular opensource contributor and loves contributing to opensource. My interests heavily aligned with clound-native technologies. I was familiar with popular mentorship programs like LFX and GSoC, which are designed to help students get started in the open source world. Based on my work the Kmesh community also promoted for the member of Kmesh I had made up my mind to apply for LFX 2025 Term-1 and began exploring projects in early February. The projects under CNCF for LFX are listed in the cncf/mentoring GitHub repository. I came across the Kmesh project, a newly added CNCF sandbox project participating in LFX for the first time. I found the Kmesh project particularly exciting because of the problem it addresses—providing a sidecarless service mesh data plane. This approach can greatly benefit the community by improving performance and reducing overhead.

Kmesh came up with 4 projects in term-1, i selected long-connection-metrics projects as it allows me to works with eBPF a already have a prior experience on working with eBPF.

I began exploring the Kmesh project by reading the documentation and contributing to Good First Issues. As I became more involved, the mentors started to take notice. I also submitted a proposal for the long connection metrics project.

In late February, I received an email from LFX notifying me of my selection.

Project Workthrough​

The tcp long connection metrics project aims to implement access logs and metrics for TCP long connections, developing a continuous monitoring and reporting mechanisms that captures detailed, real-time data throughout the lifetime of long-lived TCP connections.

Ebpf hooks are used to collect connection stats such as send/received bytes, packets losts, retransmissions etc.

More-information

Mentorship Experience​

The Kmesh maintainers were always available to help me with any doubts, whether on Slack or GitHub. Additionally, there is a community meeting held regularly every Thursday, where I could ask questions and discuss various topics. I’ve learned a lot from them, including how to approach problems effectively and consider edge cases during development in these three months.

Based on my contributions and active involvement, the Kmesh community recognized my efforts and promoted me to a member of the organization. This acknowledgment was truly encouraging and motivated me to continue contributing to Kmesh and help the project grow.

Building on the foundation of v1.0.0, this release introduces significant enhancements to Kmesh’s architecture, observability, and ecosystem integration. The official Kmesh website has undergone a comprehensive redesign, offering an intuitive interface and streamlined documentation to empower both users and developers. Under the hood, we’ve refactored the DNS module and added metrics for long connections, providing deeper insights into more traffic patterns.

In Kernel-Native mode, we’ve reduced invasive kernel modifications. Also, we use global variables to replace the BPF config map to simplify the underlying complexity. Compatibility with ​​Istio 1.25​​ has been rigorously validated, ensuring seamless interoperability with the latest Istio version. Notably, the persistent TestKmeshRestart E2E test case flaky—a long-standing issue—has been resolved through long-term investigation and reconstruction of the underlying BPF program, marking a leap forward in runtime reliability.

Main Features​

Website overhaul​

The Kmesh official website has undergone a complete redesign, offering an intuitive user experience with improved documentation, reorganized content hierarchy and streamlined navigation. In addressing feedback from the previous iteration, we focused on key areas where user experience could be enhanced. The original interface presented some usability challenges that occasionally led to navigation difficulties. Our blog module in particular required attention, as its content organization and visual hierarchy impacted content discoverability and readability. From an engineering perspective, we recognized opportunities to improve the code structure through better component organization and more systematic styling approaches, as the existing implementation had grown complex to maintain over time.

To address these problems, we shifted to React with Docusaurus, a modern documentation framework that's much more developer-friendly. This allowed us to create modular components, eliminating redundant code through reusability. Docusaurus provides built-in navigation systems specifically designed for documentation and blogs, plus version-controlled documentation features. We've implemented multilingual support with both English and Chinese documentation, added advanced search functionality, and completely reorganized the content structure. The result is a dramatically improved experience that makes the Kmesh site more accessible and valuable for all users.

Long connection metrics​

Before this release, Kmesh provides access logs during termination and establishment of a TCP connection with more detailed information about the connection, such as bytes sent, received, packet lost, rtt and retransmits. Kmesh also provides workload and service specific metrics such as bytes sent and received, lost packets, minimum rtt, total connection opened and closed by a pod. These metrics are only updated after a connection is closed.

In this release, we implement access logs and metrics for TCP long connections, developing a continuous monitoring and reporting mechanism that captures detailed, real-time data throughout the lifetime of long-lived TCP connections. Access logs are reported periodically with information such as reporting time, connection establishment time, bytes sent, received, packet loss, rtt, retransmits and state. Metrics such as bytes sent and received, packet loss, retransmits are also reported periodically for long connections.

DNS refactor​

The current DNS process includes the CDS refresh process. As a result, DNS is deeply coupled with kernel-native mode and cannot be used in dual-engine mode.

In release 1.1 we refactored the DNS module of Kmesh. Instead of a structure containing cds, the data looped through the refresh queue in the Dns is now a domain, so that the Dns module no longer cares about the Kmesh mode, only providing the hostname to be resolved.

BPF config map optimization​

Kmesh has eliminated the dedicated kmesh_config_map BPF map, which previously stored global runtime configurations such as BPF logging level and monitoring toggle. These settings are now managed through global variables. Leveraging global variables simplifies BPF configuration management, enhancing runtime efficiency and maintainability.

Optimise Kernel Native mode to reduce intrusive modifications to the kernel The kernel-native mode requires a large number of intrusive kernel reconstructions to implement HTTP-based traffic control. Some of these modifications may have a significant impact on the kernel, which makes the kernel-native mode difficult to deploy and use in a real production environment. To resolve this problem, we have modified the kernel in kernel-native mode and the involved ko and eBPF synchronously. Through the optimization of this release. In kernel 5.10, the kernel modification is limited to four, and in kernel 6.6, the kernel modification is reduced to only one. This last one will be eliminated as much as possible, with the goal of eventually running kernel-native mode on native version 6.6 and above.

Adopt istio 1.25​

Kmesh has verified compatibility with istio 1.25 and has added the corresponding E2E test to CI. The Kmesh community maintains verification of the three istio versions in CI, so the E2E test of istio 1.22 has been removed from CI.

Critical Bug Fix​

kmeshctl install waypoint error (#1287)

root analysis:

Remove the extra v before the version number when building the waypoint image.

TestKmeshRestart flaky (#1192)

root analysis:

This issue is actually not related Kmesh restart, and it can also be produced in non-restart scenario.

The root case is that it's not appropriate to use sk as the key of map map_of_orig_dst, because it is reused and the value of map will be incorrectly overwritten, resulting in the metadata is not being encoded when it should be encoded in the connection sent to the waypoint, resulting the reset error in this issue.

TestServiceEntrySelectsWorkloadEntry flaky (#1352)

root analysis:

before this test case, there is a test TestServiceEntryInlinedWorkloadEntry which will generate two workload objects, for example, Kubernetes/networking.istio.io/ServiceEntry/echo-1-21618/test-se-v4/10.244.1.103 and ServiceEntry/echo-1-21618/test-se-v6/10.244.1.103.

In the current use case, WorkloadEntry will generate the workload object Kubernetes/networking.istio.io/WorkloadEntry/echo-1-21618/test-we.

If the test case runs fast enough, the removal operation of the first two workload objects will be aggregated with the creation operation of the latter object.

Kmesh will process the new object first and then remove the old resources, reference.

The IP addresses of these three objects are the same, which will eventually lead to the inability to find the IP address in the Kmesh workload cache, which will cause auth failure and connection timeout.

Acknowledgment​

Kmesh v1.1.0 includes 118 commits from 14 contributors. We would like to express our sincere gratitude to all contributors:

We have always developed Kmesh with an open and neutral attitude, and continue to build a benchmark solution for the Sidecarless service mesh industry, serving thousands of industries and promoting the healthy and orderly development of service mesh. Kmesh is currently in a stage of rapid development, and we sincerely invite people with lofty ideals to join us!

Reference Links​

• Kmesh Release v1.1.0
• Kmesh GitHub
• Kmesh Website

Hi everyone! I'm Jayesh Savaliya, a B.Tech student at IIIT Pune passionate about backend technologies and open source. Over the last two years, I've been selected for the C4GT program twice (2024 & 2025) - yes, they let me back in - and recently completed LFX Mentorship 2025 (Term 1), where I somehow went from fixing typos to being responsible for reviewing other people's code at Kmesh.

In this blog, I'll share my journey and the strategies that actually worked (no generic "just be passionate" advice, I promise).

My Background​

When I applied to LFX, I wasn't starting from scratch. I had already battle-tested myself with:

• Sunbird (EkStep Foundation) via C4GT, where I learned that education tech is harder than it looks
• Mifos, a GSoC organization focused on financial services (because debugging payment systems at 2 AM builds character)
• Various backend projects where I definitely didn't break production. Much.

Choosing Kmesh​

I shortlisted projects from the LFX portal based on three key criteria:

1. Tech stack relevance - Technologies I wanted to master
2. Learning potential - Projects that would challenge and grow my skills
3. Active maintainers - Communities with responsive, helpful mentors

I chose Kmesh, a high-performance service mesh data plane built on eBPF and programmable kernel technologies. Kmesh's sidecarless architecture eliminates proxy overhead, resulting in better performance and lower resource consumption.

Honestly? It had "eBPF" in the description and I wanted to sound cool at tech meetups. But it turned out to be genuinely fascinating work with a great community.

How to Succeed in Open Source Programs​

Here's my three-step approach that worked for LFX:

1. Make Meaningful Contributions​

Start small and scale up gradually. Don't be the person who says "I'll rewrite the entire architecture!" on day one.

Instead:

• Weeks 1-2: Fix typos, improve logs, update documentation
• Weeks 3-4: Fix small bugs, add tests
• Week 5+: Work on core features and refactoring

This progression shows mentors you're not just throwing random PRs at the wall hoping something sticks.

2. Write a Strong Proposal​

Your proposal should be:

• Clear: Explain your approach in straightforward language
• Structured: Include a realistic timeline with milestones
• Convincing: Demonstrate why you're the right person for the project

Make sure your proposal reflects genuine engagement with the project, not just surface-level research.

3. Be Actively Involved​

Stay engaged in project channels (Slack, Discord, mailing lists). Communicate regularly with mentors, ask thoughtful questions, and contribute to discussions.

But also: don't be that person who asks questions Google could answer or pings everyone at 3 AM with "quick question." Balance is everything.

The Formula: Consistent contributions + Strong proposal + Active communication = Standing out

The Path to Maintainership​

Becoming a maintainer wasn't planned. It happened naturally through sustained engagement after the mentorship period ended.

Consistency​

I continued contributing regularly after my initial PRs were merged:

• Fixing overlooked bugs
• Adding requested features
• Refactoring code for better maintainability

Learning Mindset​

I embraced every learning opportunity, even when I had no idea what I was doing. eBPF concepts? Started clueless, ended slightly less clueless. Performance optimization? Learned by making things slower first. CI/CD improvements? Broke the build a few times, but now I own it.

Patience & Feedback​

Code reviews can be humbling (read: brutal). I learned to take feedback seriously even when it stung, iterate quickly, and stay patient when things inevitably broke.

Taking Initiative​

I started acting like a maintainer before having the title:

• Suggesting project improvements
• Writing comprehensive tests (because flaky tests are the worst)
• Automating repetitive tasks (laziness is a virtue in programming)
• Reviewing other contributors' work

By the end of my mentorship, the trust I built with the team led to being granted maintainer access. Going from "hey, can I fix this typo?" to "you're now responsible for reviewing PRs" was equal parts surreal and terrifying.

Key Takeaways​

Here's what I learned that might help you:

Start small, stay consistent - Begin with simple contributions and build from there. Consistency matters more than individual genius.

Focus on learning - Getting selected is great, but learning enough to make real contributions is what counts.

Communicate effectively - Ask questions, share progress, and be helpful. Respectful, clear communication goes a long way.

Suggest improvements - If you see something that could be better, speak up. Good ideas are always welcome.

Embrace feedback - Your first PR won't be perfect. Nobody's is. Take feedback as learning opportunities, iterate, and move on. Arguing about semicolons is not a productive use of anyone's time.

You don't need to be a genius. You just need to show up, contribute meaningfully, and improve consistently.

Final Thoughts​

The LFX Mentorship taught me more than just technical skills. I learned how to work with distributed teams across timezones, think critically about production software (logs are your friends!), and grow into a leadership role in an open source community.

If you're considering applying to LFX or any open source program, take the leap. With consistent effort and genuine engagement, you can make a real impact. If I can go from nervous first-time contributor to maintainer, so can you.

Connect With Me​

Feel free to reach out if you want to discuss open source, eBPF, or systems programming:

• LinkedIn
• GitHub

Thanks for reading, and see you in the next PR!

阿里云服务网格（ASM）支持边车模式和无边车模式。边车模式中，每个服务实例旁边运行一个代理，这种模式目前是最常选且较为稳定的解决方案。然而，这种架构会引入延迟和资源开销。为了解决边车模式中固有的延迟和资源消耗问题，近年来出现了各种无边车模式的解决方案，例如 Istio Ambient。Istio Ambient 在每个节点上部署 ztunnel 对节点上运行的 Pod 进行 L4 流量代理，并部署 waypoint 来处理 L7 流量代理。虽然无边车模式可以降低延迟和资源消耗，但其稳定性和功能完整性仍有待提高。

ASM 目前支持多种无边车模式，例如 Istio Ambient 模式、ACMG 模式以及 Kmesh 等。Kmesh（详细信息请参见 https://kmesh.net/）是一款基于 eBPF 和可编程内核实现的高性能服务网格数据面软件。通过将流量管理卸载到内核中，Kmesh 使得网格内服务间的通信无需经过代理软件，从而显著缩短流量转发路径，并有效提升服务访问的转发性能。

Kmesh 简介​

Kmesh 的双引擎模式使用 eBPF 在内核空间截获流量，同时部署 Waypoint 代理来处理复杂的 L7 流量管理，从而实现内核空间（eBPF）和用户空间（Waypoint）间的 L4 与 L7 分离治理。与 Istio Ambient Mesh 相比，它降低了约 30% 的延迟；与内核原生模式相比，双引擎模式不需要内核增强，具有更广泛的适用性。

目前，ASM 支持将 Kmesh 的双引擎模式作为服务网格的数据面之一，从而实现更高效的服务管理。具体来说，ASM 可作为控制面使用，而 Kmesh 则可作为数据面部署在阿里云容器服务 Kubernetes（ACK）集群中。

在 ACK 中部署 Kmesh 并连接到 ASM​

前提条件​

首先需要创建一个 ASM 集群，并将 ACK 集群添加到 ASM 集群中进行管理。详细步骤请参阅文档：将集群添加到 ASM 实例。

安装 Kmesh​

运行以下命令将 Kmesh 项目克隆到本地。

git clone https://github.com/kmesh-net/kmesh.git && cd kmesh

检查 ASM 控制面的服务​

下载 Kmesh 后，首先需要执行以下命令以检查集群中当前 ASM 控制面的服务名称，从而配置 Kmesh 与 ASM 控制面之间的连接。

kubectl get svc -n istio-system | grep istiod

# istiod-1-22-6   ClusterIP   None   <none>   15012/TCP   2d

使用 Kubectl 安装 Kmesh​

你可以使用 kubectl 或 helm 在 ACK Kubernetes 集群中安装 Kmesh。但在安装前，请将 ClusterId 和 xdsAddress 环境变量添加到 Kmesh 的 DaemonSet 中。这些变量用于 Kmesh 与 ASM 控制面之间的身份验证和连接。ClusterId 为 Kmesh 部署所在 ACK 集群的 ID，而 xdsAddress 为 ASM 控制面的服务地址。

# 你可以在以下文件中找到资源定义：
# helm: deploy/charts/kmesh-helm/templates/daemonset.yaml
# kubectl: deploy/yaml/kmesh.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kmesh
  labels:
    app: kmesh
  namespace: kmesh-system
spec:
  template:
    spec:
      containers:
        - env:
          # ASM 控制面服务
          - name: XDS_ADDRESS
            value: "istiod-1-22-6.istio-system.svc:15012"
          # 添加 ACK 集群 ID
          - name: CLUSTER_ID
            value: "cluster-id"
    ...

完成修改后，可运行以下命令安装 Kmesh。

# 使用 kubectl 安装
kubectl apply -f deploy/yaml

# 使用 helm 安装
helm install kmesh deploy/charts/kmesh-helm -n kmesh-system --create-namespace

检查 Kmesh 启动状态​

安装完成后，运行以下命令检查 Kmesh 的启动状态。

kubectl get pods -A | grep kmesh

# kmesh-system   kmesh-l5z2j   1/1   Running   0    117m

运行以下命令查看 Kmesh 运行状态。

kubectl logs -f -n kmesh-system kmesh-l5z2j

# time="2024-02-19T10:16:52Z" level=info msg="service node sidecar~192.168.11.53~kmesh-system.kmesh-system~kmesh-system.svc.cluster.local connect to discovery address istiod.istio-system.svc:15012" subsys=controller/envoy
# time="2024-02-19T10:16:52Z" level=info msg="options InitDaemonConfig successful" subsys=manager
# time="2024-02-19T10:16:53Z" level=info msg="bpf Start successful" subsys=manager
# time="2024-02-19T10:16:53Z" level=info msg="controller Start successful" subsys=manager
# time="2024-02-19T10:16:53Z" level=info msg="command StartServer successful" subsys=manager
# time="2024-02-19T10:16:53Z" level=info msg="start write CNI config\n" subsys="cni installer"
# time="2024-02-19T10:16:53Z" level=info msg="kmesh cni use chained\n" subsys="cni installer"
# time="2024-02-19T10:16:54Z" level=info msg="Copied /usr/bin/kmesh-cni to /opt/cni/bin." subsys="cni installer"
# time="2024-02-19T10:16:54Z" level=info msg="kubeconfig either does not exist or is out of date, writing a new one" subsys="cni installer"
# time="2024-02-19T10:16:54Z" level=info msg="wrote kubeconfig file /etc/cni/net.d/kmesh-cni-kubeconfig" subsys="cni installer"
# time="2024-02-19T10:16:54Z" level=info msg="command Start cni successful" subsys=manager

你可以通过以下命令为特定命名空间启用 Kmesh。

kubectl label namespace default istio.io/dataplane-mode=Kmesh

流量切换演示​

部署示例应用及流量切换规则​

在为默认命名空间启用 Kmesh 后，运行以下命令安装示例应用。

kubectl apply -f samples/fortio/fortio-route.yaml
kubectl apply -f samples/fortio/netutils.yaml

运行以下命令检查示例应用的运行状态。

kubectl get pod
# NAME                         READY   STATUS    RESTARTS   AGE
# fortio-v1-596b55cb8b-sfktr   1/1     Running   0          57m
# fortio-v2-76997f99f4-qjsmd   1/1     Running   0          57m
# netutils-575f5c569-lr98z     1/1     Running   0          67m

kubectl describe pod netutils-575f5c569-lr98z | grep Annotations
# Annotations:      kmesh.net/redirection: enabled

Pod 的标签 kmesh.net/redirection: enabled 表示该 Pod 已启用 Kmesh 转发功能。

运行以下命令查看当前定义的流量路由规则。从输出中可以看出，90% 的流量被导向 fortio 的 v1 版本，而 10% 的流量被导向 v2 版本。

kubectl get virtualservices -o yaml

# apiVersion: v1
# items:
# - apiVersion: networking.istio.io/v1beta1
#   kind: VirtualService
#   metadata:
#     annotations:
#       kubectl.kubernetes.io/last-applied-configuration: |
#         {"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"fortio","namespace":"default"},"spec":{"hosts":["fortio"],"http":[{"route":[{"destination":{"host":"fortio","subset":"v1"},"weight":90},{"destination":{"host":"fortio","subset":"v2"},"weight":10}]}]}}
#     creationTimestamp: "2024-07-09T09:00:36Z"
#     generation: 1
#     name: fortio
#     namespace: default
#     resourceVersion: "11166"
#     uid: 0a07f283-ac26-4d86-b3bd-ce6aa07dc628
#   spec:
#     hosts:
#     - fortio
#     http:
#     - route:
#       - destination:
#           host: fortio
#           subset: v1
#         weight: 90
#       - destination:
#           host: fortio
#           subset: v2
#         weight: 10
# kind: List
# metadata:
#   resourceVersion: ""

为 Fortio 服务部署 Waypoint​

你可以在默认命名空间中执行以下命令部署 Waypoint，以处理服务级别的 L7 流量。

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  labels:
    istio.io/waypoint-for: service
  name: fortio-waypoint
  namespace: default
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - name: mesh
    port: 15008
    protocol: HBONE
EOF

接着，为 fortio 服务启用 Waypoint。

kubectl label service fortio istio.io/use-waypoint=fortio-waypoint

运行以下命令检查当前 Waypoint 的状态。

kubectl get gateway.gateway.networking.k8s.io

# NAME              CLASS            ADDRESS          PROGRAMMED   AGE
# fortio-waypoint   istio-waypoint   192.168.227.95   True         8m37s

开始测试流量​

你可以通过执行以下命令启动测试流量。结果应显示约 10% 的流量被导向 fortio 的 v2 版本。

for i in {1..20}; do kubectl exec -it $(kubectl get pod | grep netutils | awk '{print $1}') -- curl -v $(kubectl get svc -owide | grep fortio | awk '{print $3}'):80 | grep "Server:"; done

# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 2
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 2
# < Server: 1
# < Server: 1

Kmesh 是一个内核原生、无边车(sidecarless) 的服务网格数据平面。借助 ebpf 和可编程内核，它将流量治理下沉到操作系统内核，从而减少了服务网格的资源开销和网络延迟。

内核中可以直接获取流量数据，并通过 bpf map 将数据传递到用户态。这些数据用于构建指标和访问日志。

如何获取数据​

在内核中，可以直接从 socket 中获取携带的指标数据。

bpf_tcp_sock 中携带的数据如下：

struct bpf_tcp_sock {
 __u32 snd_cwnd;  /* 发送拥塞窗口 */
 __u32 srtt_us;  /* 平滑往返时延（左移 3 位，以微秒为单位） */
 __u32 rtt_min;
 __u32 snd_ssthresh; /* 慢启动阈值 */
 __u32 rcv_nxt;  /* 下一个期望接收的数据 */
 __u32 snd_nxt;  /* 下一个将要发送的序列号 */
 __u32 snd_una;  /* 第一个等待确认的字节 */
 __u32 mss_cache; /* 缓存的有效 MSS，不包括 SACKS */
 __u32 ecn_flags; /* ECN 状态位 */
 __u32 rate_delivered; /* 保存的速率采样：已交付的包数量 */
 __u32 rate_interval_us; /* 保存的速率采样：经过的时间（微秒） */
 __u32 packets_out; /* 正在“飞行”中的包 */
 __u32 retrans_out; /* 重传的包 */
 __u32 total_retrans; /* 整个连接的重传总数 */
 __u32 segs_in;  /* RFC4898 tcpEStatsPerfSegsIn：接收的总段数 */
 __u32 data_segs_in; /* RFC4898 tcpEStatsPerfDataSegsIn：接收的数据段总数 */
 __u32 segs_out;  /* RFC4898 tcpEStatsPerfSegsOut：发送的总段数 */
 __u32 data_segs_out; /* RFC4898 tcpEStatsPerfDataSegsOut：发送的数据段总数 */
 __u32 lost_out;  /* 丢失的包 */
 __u32 sacked_out; /* 被 SACK 确认的包 */
 __u64 bytes_received; /* RFC4898 tcpEStatsAppHCThruOctetsReceived：累计接收（或确认）的字节数 */
 __u64 bytes_acked; /* RFC4898 tcpEStatsAppHCThruOctetsAcked：累计确认的字节数 */
 __u32 dsack_dups; /* RFC4898 tcpEStatsStackDSACKDups：接收到的 DSACK 块数 */
 __u32 delivered; /* 包括重传在内的数据包总交付数 */
 __u32 delivered_ce; /* 同上，但仅限 ECE 标记的数据包 */
 __u32 icsk_retransmits; /* 未恢复（RTO）超时次数 */
};

注意： 上述数据并未全部用于构建指标和访问日志。Kmesh 后续会完善指标数据。目前使用的数据包括：

struct tcp_probe_info {
    __u32 type;
    struct bpf_sock_tuple tuple;
    __u32 sent_bytes;
    __u32 received_bytes;
    __u32 conn_success;
    __u32 direction;
    __u64 duration; // 单位：纳秒
    __u64 close_ns;
    __u32 state; /* TCP 状态 */
    __u32 protocol;
    __u32 srtt_us; /* 平滑往返时延（左移 3 位，以微秒为单位） */
    __u32 rtt_min;
    __u32 mss_cache;     /* 缓存的有效 MSS，不包括 SACKS */
    __u32 total_retrans; /* 整个连接的重传总数 */
    __u32 segs_in;       /* RFC4898 tcpEStatsPerfSegsIn：接收的总段数 */
    __u32 segs_out;      /* RFC4898 tcpEStatsPerfSegsOut：发送的总段数 */
    __u32 lost_out;      /* 丢失的包 */
};

除了上述可直接访问的数据外，Kmesh 在链路建立期间还会临时记录数据，例如在链路关闭时获取链路时长等信息。

如何处理数据​

当 Kmesh 完成对该链路数据的处理后，它会通过 ringbuf 将数据传递到用户态。

在用户态解析 ringbuf 中的数据后，Kmesh 根据链路的源和目的信息构建 metricLabels，然后更新 metricController 中的缓存。

这是因为通过 ringbuf 上报的数据是以 Pod 为粒度的链路数据，而呈现给用户的指标既有 Pod 级别也有服务级别，因此还需要进行聚合处理。

从目的工作负载的 Services 信息中获取集群内目的服务的主机名和命名空间：

namespacedhost := ""
for k, portList := range dstWorkload.Services {
    for _, port := range portList.Ports {
        if port.TargetPort == uint32(dstPort) {
            namespacedhost = k
            break
        }
    }
    if namespacedhost != "" {
        break
    }
}

在构建了工作负载粒度和服务粒度的 metricLabels 后，缓存会被更新。

每 5 秒，指标信息会通过 Prometheus API 更新到 Prometheus 中。

在处理指标的同时，会生成与访问日志相关的数据。每当链路关闭时，系统会利用这些数据生成访问日志，并将其打印到 Kmesh 的日志中。

下图展示了架构图：

结果​

当前阶段由 Kmesh L4 监控的指标如下：

工作负载：

服务：

指标示例结果：

kmesh_tcp_workload_received_bytes_total{connection_security_policy="mutual_tls",destination_app="httpbin",destination_canonical_revision="v1",destination_canonical_service="httpbin",destination_cluster="Kubernetes",destination_pod_address="10.244.0.11",destination_pod_name="httpbin-5c5944c58c-v9mlk",destination_pod_namespace="default",destination_principal="-",destination_version="v1",destination_workload="httpbin",destination_workload_namespace="default",reporter="destination",request_protocol="tcp",response_flags="-",source_app="sleep",source_canonical_revision="latest",source_canonical_service="sleep",source_cluster="Kubernetes",source_principal="-",source_version="latest",source_workload="sleep",source_workload_namespace="default"} 231

这些指标也可以通过 Prometheus 仪表板进行查看。参见 Kmesh 可观测性

当前阶段由 Kmesh L4 监控的访问日志包括：

访问日志示例结果：

accesslog: 2024-09-14 08:19:26.552709932 +0000 UTC
src.addr=10.244.0.17:51842, src.workload=prometheus-5fb7f6f8d8-h9cts, src.namespace=istio-system,
dst.addr=10.244.0.13:9080, dst.service=productpage.echo-1-27855.svc.cluster.local, dst.workload=productpage-v1-8499c849b9-bz9t9, dst.namespace=echo-1-27855, direction=INBOUND, sent_bytes=5, received_bytes=292, duration=2.733902ms

总结​

Kmesh 直接从 socket 中获取流量数据，并通过 ringbuf 将数据传递到用户态以生成 Metric 和 Accesslog，并将其暴露给 Prometheus。

这种方式避免了在用户态截取流量并以原生方式获取指标，同时通过定时批量更新用户态指标，避免在高流量时增加网络延迟。

后续，我们还将开发链路追踪功能，以补全 Kmesh 的可观测能力。

欢迎加入 Kmesh 社区!

Kmesh：业界首个基于内核的无边车流量管理引擎​

eBPF 与无边车：服务网格的未来​

近年来，服务网格越来越受欢迎，但边车模式仍面临资源开销、升级和部署以及延迟等挑战。如何降低代理开销、构建无边车服务网格成为业界长期存在的问题。

在项目初期，Kmesh 创新性地提出了业界首个基于内核的无边车流量管理引擎来解决这一问题。通过使用 eBPF 和可编程内核技术，将 L4–L7 流量治理下沉至操作系统。此时流量无需经过代理，服务通信路径由三跳减少到仅一跳，从而消除代理开销，实现无边车服务网格。

Kmesh 的优势​

• 高性能
  利用内核技术，提供原生的 L4–L7 流量治理支持，与边车相比，将微服务转发延迟降低了 60%，微服务启动性能提升了 40%。
• 低开销
  业务工作负载无需注入边车，数据平面开销降低了 70%。
• 高可用性
  内核流量治理不会中断连接，Kmesh 组件升级或重启不会影响现有服务连接。
• 零信任网络
  基于内核 mTLS 可实现透明的零信任网络。
• 安全隔离
  支持基于 eBPF 的 VM 安全和 cgroup 级别的治理隔离。
• 灵活的管理模式
  除了全内核管理外，Kmesh 还支持对 L4 和 L7 流量治理进行切分隔离。内核中的 eBPF 程序和 waypoint 组件分别处理 L4 和 L7 流量，使用户可以逐步实现从 L4 服务管理向 L7 服务管理的迁移。
• 无缝兼容
  理论上可无缝集成任意支持 xDS 协议的控制面。Istio 是 Kmesh 首次集成的控制面，支持 Istio API 和 Gateway API。同时，Kmesh 还能与边车模式协同工作，实现从边车向 Kmesh 的平滑迁移。

为什么选择 Kmesh？​

Kmesh 构建于无边车网络架构，目前已获得 Istio 社区和 Cilium 社区的认可，并广受用户接受。与边车模式相比，无边车模式避免了额外的资源开销；它将应用和代理的生命周期分离，消除了一对一绑定，从而简化了部署和维护。

Kmesh 利用 eBPF 技术在内核态执行流量治理，确保流量治理与流量传输无缝衔接。通过防止服务连接中断，Kmesh 减少了流量路径中的连接数量，最大程度降低了应用访问延迟。

用户态流量治理的一个明显缺陷是，代理升级可能导致服务流量中断。Kmesh 通过利用可编程内核技术解决了这一问题，从而获得了显著的业界优势。eBPF 技术的潜力已经显现，并有望推动更多网络创新。

Kmesh 还提供了一种高级模式，通过分离 L4 和 L7 流量治理进一步增强 L7 流量管理能力。这种分离方式实现了更细粒度的物理隔离，租户、命名空间或服务可以独享 L7 代理 waypoint，并可根据流量负载动态缩放，比传统的集中式网关更灵活可靠，且不存在单点故障。

因此，我们坚信，结合 eBPF 技术与 waypoint 的无边车架构是最佳方案。该方案旨在降低资源开销和延迟：具体而言，eBPF 在节点上处理 L4 路由和简单的 L7 流量治理，而更复杂的 L7 协议则交由 waypoint 进行全面管理。

为社区做出贡献​

Kmesh 由华为发起，并在 openEuler 社区孵化，目前作为一个独立项目托管在 GitHub 上。它为用户提供了性能卓越的流量治理技术解决方案。

作为中国首个参与服务网格的厂商，华为自 2018 年起为 Istio 社区做出贡献，并在亚洲贡献最多。华为还在 Istio Steering Committee 中占有一席之地，参与 Istio 社区的治理。

凭借在 Istio 社区积累的丰富经验，我们期望以开放、中立的方式推动 Kmesh 的成长。我们的目标是打造业界领先的无边车服务网格标杆解决方案，满足各行业需求，并促进服务网格技术的健康、有序演进。Kmesh 正在快速发展，我们热忱欢迎有志之士加入我们的行列。

Kmesh 社区： https://github.com/kmesh-net/kmesh

参考文献​

[1] CNCF 生态图谱: https://landscape.cncf.io/

[2] 介绍 Ambient Mesh: https://istio.io/latest/blog/2022/introducing-ambient-mesh/

[3] 华为云 ASM: https://support.huaweicloud.com/intl/en-us/asm/index.html

[4] Kmesh 快速上手: https://kmesh.net/en/docs/setup/quickstart/

Kmesh 简介​

基于 eBPF 和可编程内核技术，Kmesh 将流量管理下沉到操作系统，消除了数据路径上代理层的需求，从而实现了内核级无 Sidecar 的网格数据平面。

Kmesh 的关键能力​

• 高性能： 原生支持内核中 L4~L7 的流量管理功能，无需经过物理代理组件即可完成治理流程。这使得网格内服务通信路径由代理架构下的三跳降为一跳，显著提升了网格数据平面的转发性能。
• 低开销： 无需在工作负载 Pod 附侧部署 Sidecar，大幅降低了网格基础设施的资源开销。
• 安全隔离： 基于 eBPF 的运行时安全机制，支持 cgroup 级别的治理隔离。
• 无缝兼容： 支持与遵循 xDS 协议的服务网格控制平面（如 Istiod）集成，同时也能与现有的 Sidecar 网格协同工作。
  

Kmesh 的主要组件包括：

• kmesh-controller： 负责 BPF 生命周期管理、xDS 资源订阅、可观测性等功能。
• kmesh-api： 适配层，包含 xDS 转换后的编排 API、可观测性通道等。
• kmesh-runtime： 在内核中实现的运行时，支持 L4~L7 流量编排；第 7 层编排能力依赖于内核的增强。
• kmesh-orchestration： 基于 eBPF 实现 L4~L7 流量编排，如路由、金丝雀发布、负载均衡等。
• kmesh-probe： 提供端到端可观测性的探针工具。

性能测试​

我们使用 fortio 在相同流量管理场景下测试了 Istio（Envoy）与 Istio(Kmesh) 的性能，同时以基于 kube-proxy(iptables) 的服务通信延迟作为基准参考。

不同连接数下的延迟对比：

相同 QPS 下 CPU 开销对比：

从测试结果中可以看出：

• Kmesh 的转发延迟几乎接近原生 Kubernetes 的转发延迟，明显优于 Istio Sidecar 模式。
• 在相同 QPS 下，Kmesh 的 CPU 开销基本与原生 Kubernetes 持平，相较于 Istio Sidecar 模式有大幅降低。

详细演示测试细节，请观看我们的演示视频：

Kmesh 的关键技术解析​

内核级流量编排运行时​

在微服务通信中，通常在发送业务消息之前先建立连接。如果要对业务消息进行无缝编排，通常需要进行流量拦截，在完成编排后再根据拦截的内容进行消息转发。这是当前代理实现的方式。Kmesh 则旨在在流量传递过程中完成治理，并将连接建立延迟到业务消息发送阶段，以实现更高的编排处理性能。

伪连接建立​

在 pre_connect 过程中加载 BPF 程序。如果所访问的目标地址位于 xDS 监听器范围内，则调用 bpf_setsockopt，通过 TCP_ULP 将当前套接字的 TCP 协议钩子重新加载到 kmesh_defer 内核模块中。

延迟连接建立​

kmesh_defer 内核模块对 connect/send 钩子进行了重写（即对原生钩子的增强）：

• 当服务首次进入 connect 钩子时，会设置 bpf_defer_connect 标志，并不会触发握手过程。
• 在 send 钩子中，如果检测到套接字上设置了 bpf_defer_connect 标志，则触发 connect，此时通过扩展 BPF 程序调用 BPF_SOCK_OPS_TCP_DEFER_CONNECT_CB，完成流量治理后，再根据调整后的通信五元组和消息建立连接并发送数据。

整个治理过程大致如下图所示：

xDS 规则管理​

xDS 模型是一种分层树形规则表达，不同版本的模型定义可能有所调整。Kmesh 需要将模型信息转换为 eBPF map 存储，同时保持模型规则之间的层级关系。

将 xDS 模型转换为 eBPF map 数据​

具体过程：

1. Kmesh 订阅 Istiod 的 xDS 模型，并基于 protobuf-c 将 pb 模型转换为 C 数据结构风格。
2. 对于顶层模型（例如 listener），Kmesh 定义了对应的知名 map 表，其值的数据结构复用了 protobuf-c 导出的 C 结构体。
3. map 的更新从顶层模型的知名 map 表开始。对于记录中的指针成员，xds-adapter 会创建一个 inner-map 表，用于存储指针指向的实际数据区域；然后将 inner-map 的 map fd 添加到 map-in-map 表中，最终使用其在 map-in-map 表中的 key（索引）作为指针成员的值。

map-in-map 解决 xDS 模型的层级特性​

对于 map 记录中的值成员，如果它们是指针变量（涉及引用其他数据结构），则通过 inner-map 存储所指向的数据区域：

• 如果值成员为基本数据类型（如 int），则可直接访问。
• 如果值成员为指针类型，则指针存储的值为 inner-map 中实际数据所在的索引（注：该索引与 kmesh-daemon 的 xds-adapter 模块在更新 bpf map 时协调写入）。在访问时，首先根据该索引查找 inner-map 的 map fd，然后从 inner-map 表中获取实际数据。对于多级指针成员，此过程会重复进行，直至所有指针信息被剥离。

流量管理编排实现​

由于 xDS 的治理规则较为复杂，涉及层级匹配，其复杂度超出单个 eBPF 程序的处理能力。基于 eBPF Tail Calls 特性，Kmesh 将治理过程拆分为多个独立的 eBPF 程序，从而具备良好的可扩展性。

Kmesh 最新关键特性​

• 一键部署
  Kmesh 社区已发布 Kmesh 部署镜像，并支持通过 helm 一键部署 Kmesh。
• 基于命名空间的启用
  Kmesh 支持基于命名空间启用流量接管，例如：
  kubectl label namespace default label istio.io/dataplane-mode=Kmesh
• 与 Istio Sidecar 的无缝集成
  对于集群中未启用 Kmesh 数据平面的命名空间，如使用 Sidecar 代理（例如 Envoy），Kmesh 同样支持互联。此外，可使用 sockmap 加速 Sidecar 的流量转发，带来 10% 至 15% 的转发性能提升，同时不影响业务流程。
• 与服务网格控制平面的自动集成
  Kmesh 支持与 Istiod 自动集成，理论上任何遵循 xDS 协议的网格控制平面均可与 Kmesh 集成。通过修改 yaml 中的 MESH_CONTROLLER 环境变量即可指定。
• 支持 xDS/工作负载
  Kmesh 支持 xDS 模型，实现 TCP 流量转发、HTTP/1.1 头匹配、路由及灰度发布，同时支持随机和轮询负载均衡算法。此外，还基于工作负载模型提供基本的转发功能。

展望未来​

Kmesh 是一款基于 eBPF 和可编程内核实现的高性能流量管理引擎。与业内解决方案相比，它在转发性能上更高、资源开销更低。Kmesh 可在未打增强补丁的内核版本上以兼容模式运行，而对于完整的治理能力，目前 openEuler 23.03 版本已原生支持，其他操作系统则需基于增强补丁进行构建。
Kmesh 正在逐步演进为更受欢迎的流量管理引擎，还有大量工作待完成。目前已计划支持将 L7 流量转发到 waypoint 以及 mTLS 功能。欢迎大家尝试 Kmesh，并与 Kmesh 社区保持联系。我们也非常期待您的参与。

在 KubeCon + CloudNativeCon Europe 2024 与 Kmesh 相见​

在 KubeCon + CloudNativeCon Europe 2024 期间，Kmesh 将参与多项活动，包括：

Kmesh 展台​

3 月 20 日至 22 日全天
欢迎前往 KubeCon 的 J1 展位，与专家交流或观看演示！

Kmesh 开放演讲​

3 月 22 日（星期五），中欧时间 11:10-11:30
内核原生流量治理框架带来全新性能体验

参考链接​

[1] Kmesh 发布信息: https://github.com/kmesh-net/kmesh/releases

[2] Kmesh 部署镜像: https://github.com/orgs/kmesh-net/packages

[3] Kmesh 一键部署: https://github.com/kmesh-net/kmesh?tab=readme-ov-file#quick-start

[4] openEuler 23.03 版本: https://repo.openeuler.org/openEuler-23.03/

[5] 基于增强补丁的构建: https://github.com/kmesh-net/kmesh/blob/main/docs/kmesh_kernel_compile.md

[6] Kmesh 社区地址: https://github.com/kmesh-net/kmesh

服务网格的概念最初由开发 Linkerd 软件的公司 Buoyant 在 2016 年提出。Linkerd 的 CEO Willian Morgan 给出了服务网格的最初定义：

服务网格是专门用于处理服务间通信的一个层。它负责在构成现代云原生应用的复杂服务拓扑中可靠地传递请求。实际上，服务网格通常通过部署在应用代码旁边的一组轻量级网络代理来实现，而应用程序本身无需感知这一层。

简单来说，服务网格是一层处理服务间通信的机制。它通过部署一组轻量级网络代理，为现代云原生应用提供透明且可靠的网络通信。

服务网格的本质在于解决微服务如何高效通信的问题。通过实现负载均衡、金丝雀路由和熔断等治理规则，服务网格能够协调流量，最大化服务集群的能力。这是服务治理演进的产物。

我们可以将服务治理的演进分为三代，并进行比较。从这一演进中可以看出，服务治理能力逐步从业务逻辑中剥离，并下沉到更低层次。

作为处理服务间通信的一层，服务网格有效弥补了 Kubernetes（k8s）中微服务治理的不足。作为云原生环境的下一代技术，它已成为云计算的关键组件。

近年来，服务网格受到广泛关注，涌现出诸多服务网格软件解决方案，如 Linkerd、Istio、Consul Connect 和 Kuma。虽然它们在软件架构上可能存在细微差别，但以 Istio（最流行的服务网格项目之一）为例，可以说明服务网格的基本架构：

以 Kubernetes 集群为例，当创建一个 Pod 实例时，服务网格软件会透明地在应用代码旁边部署一个代理容器（也称为边车，Istio 默认的边车软件为 Envoy）。Pods 之间的基本通信流程如下：

• 流量通过 iptables 规则被透明拦截，并导向 Pod 内的代理组件。
• 代理组件应用流量治理逻辑（例如熔断、路由、负载均衡），确定目标服务实例并转发消息。
• 目标 Pod 内的代理组件拦截传入流量，应用基础流量治理逻辑（例如限流），然后将流量转发到 Pod。
• 处理完成后，响应沿原路径返回给请求的 Pod。

服务网格数据平面面临的挑战​

正如前文所述，服务网格通过在数据平面中引入代理层来实现透明的服务治理。然而，这也带来了一个问题：代理层的引入不可避免地增加了服务通信的延迟并降低了性能。

以 Istio 官方网站提供的数据为例，在集群环境中，微服务之间每跳的平均延迟增加了 2.65 毫秒。考虑到在微服务集群中，外部请求往往涉及多个微服务之间的调用，因此服务网格引入的延迟开销十分显著。随着服务网格应用的不断增长，代理架构带来的额外延迟已成为一个关键挑战。

为了解决这一问题，我们对 HTTP 服务的 L7 负载均衡进行了性能测试，以分析服务网格的通信性能。时间消耗的细分如下：

从对网格流量的详细分析中可以看出，服务间通信从一次连接建立变为三次，从两次协议栈遍历变为六次。时间消耗主要集中在数据拷贝、连接建立、上下文切换等方面，而流量治理实际引入的开销相对较小。

这就引出了一个问题：在保持应用透明治理的同时，是否能降低服务网格的延迟开销？

高性能服务网格数据平面：Kmesh​

基于上述性能分析，我们对服务网格数据平面进行了两阶段优化。

Sockmap：利用 Sockmap 加速服务网格数据平面​

Sockmap 是 Linux 4.14 引入的 eBPF 特性，它能够在节点内部在套接字之间重定向数据流，而无需经过复杂的内核协议栈，从而优化了网络路径上套接字之间数据转发的性能。

在服务网格场景中，Pod 内业务容器与本地代理组件之间的默认通信需要经过完整的内核协议栈，从而产生不必要的开销。通过 Sockmap 可以优化这一开销。下图展示了这一概念：

利用 Sockmap 加速服务网格数据平面的基本步骤如下：

• 在连接建立过程中，附加一个 eBPF 程序（类型为 BPF_PROG_TYPE_SOCK_OPS）拦截所有 TCP 连接建立动作：
  • 在 BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB 状态下，添加客户端侧的 Sockmap 记录。
  • 在 BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB 状态下，添加服务端侧的 Sockmap 记录。
  • 将双方的套接字信息存储到 Sockmap 表中。
• 在 sendmsg 过程中，附加一个 eBPF 程序（类型为 BPF_PROG_TYPE_SK_MSG）拦截消息发送动作：
  • 程序根据当前套接字信息查找 Sockmap 表，并将其与目标方的套接字信息关联，直接将流量重定向到目标套接字的接收队列。

通过利用 Sockmap 加速服务网格数据平面，我们在 60 个长连接场景下观察到服务访问的平均延迟降低了 10% 到 15%。

虽然 Sockmap 是优化服务网格数据平面的常用方案，但它并未完全解决服务网格延迟相关的性能挑战。

Offload：利用可编程内核将流量治理卸载至操作系统​

基于前述性能分析，显而易见服务网格引入的额外开销中有很大一部分花费在将流量重定向到代理组件上，而实际执行流量治理的开销相对较小。这就引出了一个问题：是否可以绕过代理组件，直接在内核中进行流量治理，而内核本身就是网络通信的天然参与者？

Kmesh 正是我们提出的高性能服务网格数据平面解决方案，它利用可编程内核将流量治理卸载到操作系统中。通过 Kmesh，数据平面不再经过代理组件，服务间通信从三跳降低到一跳，实现了沿流量传输路径的流量治理。下图展示了 Kmesh 中微服务之间的流量流程：

Kmesh 的软件架构包括以下组件：

• kmesh-controller：负责 Kmesh 生命周期管理、XDS 协议集成、可观察性等功能的管理程序。
• kmesh-api：Kmesh 提供的 API 接口层，包括由 XDS 转换而来的编排 API 以及可观察性通道。
• kmesh-runtime：在内核中实现的运行时，支持 L3-L7 流量编排。
• kmesh-orchestration：基于 eBPF 实现的 L3-L7 流量编排，包括路由、金丝雀发布、负载均衡等。
• kmesh-probe：提供端到端可观察性能力的探针。

我们在 Istio 网格环境中进行了对比测试，使用 Fortio 测试工具对 HTTP 服务的 L7 负载均衡进行性能测试，结果显示 Kmesh 在服务间通信方面比 Istio 原生数据平面（Envoy）提升了 5 倍的性能。

值得注意的是，我们还测试了基于 Kubernetes 的非网格环境下的服务间通信性能，其性能与 Kmesh 相当。这进一步验证了 Kmesh 的低延迟性能。（测试场景涉及实验室环境下的 L7 负载均衡，实际治理场景下的性能可能不尽相同，初步评估显示相比 Istio 有 2-3 倍的性能提升。）

结论​

作为云原生环境的下一代技术，服务网格为应用提供了透明的服务治理。然而，代理架构引入了额外的延迟开销，这已成为服务网格大规模采用的一大挑战。Kmesh 通过将流量治理卸载至操作系统、利用可编程内核提出了一种全新的解决方案，大幅提升了服务网格数据平面的性能，为服务网格数据平面的发展提供了全新的思路。

参考资料​

https://linkerd.io/2017/04/25/whats-a-service-mesh-and-why-do-i-need-one

https://istio.io/latest/docs/ops/deployment/architecture

https://istio.io/v1.16/docs/ops/deployment/performance-and-scalability/#performance-summary-for-istio-hahahugoshortcode-s0-hbhb

早期的微服务架构面临着服务发现、负载均衡以及认证/授权等诸多挑战。最初，微服务实践者们各自实现了分布式通信系统来应对这些挑战，但这种方式导致业务功能的重复造轮子。为了解决这一问题，提出了一种方案：将通用的分布式系统通信代码抽取成框架，并以库的形式供程序调用。然而，这个看似完美的方案存在几个致命的弱点：

• 框架需要对业务代码进行侵入式修改，迫使开发者学习如何使用该框架。
• 框架无法跨不同的编程语言使用。
• 在管理复杂的项目框架和库版本兼容性问题时，升级框架往往会迫使业务一同升级。

随着微服务架构的演进，第一代服务网格应运而生，其代表产品有 Linkerd/Envoy/NginxMesh 以及边车代理模式。作为一种基础设施层，边车代理与业务进程解耦，并与业务进程一起部署，接管了业务组件之间的通信，将网络数据传输抽象为一个独立的层。该层集中处理分布式系统所需的服务发现、负载均衡、认证/授权等功能，为微服务框架库遇到的问题提供了更为全面的解决方案。

然而，软件开发没有万全之策。虽然服务网格带来了诸多便利，但它也不可避免地存在一些问题。在传统的方式中，客户端与服务端之间传递消息只需经过一次内核协议栈即可完成。但在边车代理模式中，业务流量通常利用内核的 iptables 能力被拦截，导致业务数据多次经过内核协议栈。这就增加了延迟，并降低了吞吐量。

我们对服务网格性能进行了基准测试，发现边车模式（使用 Envoy）的延迟明显比非边车模式（无 Envoy）更高。

基于 eBPF 能力加速 ServiceMesh​

是否有方法可以在享受 ServiceMesh 带来的便利的同时减少乃至消除网络延迟的影响？在这里，不得不提 eBPF 技术。eBPF 是内核中的一项革命性技术，旨在更安全高效地扩展内核功能，而无需修改内核代码或加载内核模块。利用 eBPF 绕过内核网络协议栈，可以降低网络延迟，从而提升 ServiceMesh 的用户体验。这也是当前业界的普遍做法。

为了实现绕过内核网络协议栈的目标，我们需要利用 eBPF 提供的两个能力：sockops 和套接字重定向。

• Sockops 提供了在创建 TCP 连接时识别并将套接字（通常由四元组标识）存储到 sockmap 数据结构中的能力。
• 套接字重定向 支持在传输 TCP 数据时，根据关键字在 sockmap 中引用套接字。当匹配成功时，可以直接将数据转发到对应的套接字。
• 对于在 sockmap 中未找到的套接字，数据包将照常通过内核网络协议栈发送。

通过结合这两种能力，我们可以直接将数据包转发到对应的套接字，而无需经过内核网络协议栈，从而减少在内核网络协议栈中花费的时间。

在建立 TCP 套接字连接的过程中，实际上存在两个连接建立过程：正向连接和反向连接。通常在正向和反向连接建立过程中，会利用 iptables 信息获取实际的 IP 地址和端口号。通过调用 bpf_get_sockopt，在 eBPF 函数中可以主动获取经过 iptables 转换后的地址信息。这使得我们可以建立一个辅助映射，用于存储正向和反向连接之间的对应关系。在进行套接字重定向时，我们首先从辅助映射中查找对端的连接信息；如果成功找到，则执行套接字转发操作。其原理如下图所示：

我们在 openEuler 21.03 上进行了实际测试，以评估通过 sockmap 能力实现加速的效果。测试环境为 openEuler-21.03 / 5.10.0-4.17.0.28.oe1.x86_64，网络配置为 fortio-envoy-envoy:80 到 fortio_server:80。

根据测试结果，与不使用 ServiceMesh 的情况相比，在利用 sockmap 加速后，QPS 提升了约 18%，平均延迟降低了 15%。

服务网格的性能开销能否完全消除？​

然而，尽管利用 sockmap 为 ServiceMesh 带来了显著加速，但与不使用 ServiceMesh 相比，仍存在较大差距。这主要是因为当前服务网格代理架构引入了大量延迟开销。要完全消除服务网格带来的性能影响，关键在于从架构层面进行优化。

Kmesh 正在积极探索数据平面架构层面的新方法来解决这一挑战，业界在这方面也做出了大量努力。在后续文章中，我们将详细介绍这些举措和优化措施。