Query Response

Query data

Score

Prompt

System Prompt

You are a reporter for a major world newspaper. Write your response as if you were writing a short, high-quality news article for your paper. Limit your response to one paragraph. Use the following article for context: , 5.12, so anything in the last 5 or 6 years, eBPF will just work out of the box. You can write your code and attach it to any part of the running Linux kernel. No more dependencies on waiting for it to actually make its way to the actual upstream main kernel.What it looks like is we have our kernel. It will manipulate files, networking, memory, processes. We have our eBPF program, and we can effectively attach it to various areas inside the Linux kernel. Today we're going to look at attaching eBPF programs to the networking stack in order for us to manipulate these AI calls. eBPF code is low-level. It is code that you are writing that is going to attach into a Linux kernel. It does need to be safe and secure. In order to do that, eBPF comes with tooling to ensure that your code won't loop forever and cannot in any way crash a Linux kernel, because adding stuff into running kernels is quite scary. We want to ensure that the scaredness doesn't actually exist when we're doing these sorts of things. We can attach eBPF programs to a lot of different areas. Kprobes, kernel probes, user probes or Uprobes allows us to attach our eBPF programs to anything in userland.Network packets is the main use case for eBPF today. Then there's a bunch of other areas as well. Now hardware devices, Linux security modules, performance events and things like that. You can attach it to so many different areas. As I mentioned, various areas of the file system from the block device all the way to the virtual file system that sits on top of all of that. A number of different areas in the networking stack to various areas inside the Linux kernel itself as well. A huge amount of control over what we may or may not want to do with a running Linux system.Building an AI/API GatewayWe know what the problem is. We now have a technology that's going to allow us to start changing these sorts of behaviors by hooking in to a running system. Let's step through what that will look like. This is the shopping list for everything that we're going to need in order to build an AI gateway that's going to run within Kubernetes. Step one is we're going to want to transparently intercept the traffic from our AI agent program or whatever it is to our LLM endpoint. We're going to redirect that traffic transparently to a userland proxy where we'll be able to modify and change that. We want to be able to observe all of those AI requests and responses. We may want to modify that, change the model, look at the prompt, cap the tokens, all of the things to protect us from issues that I mentioned earlier.We may want to block those requests, again, based upon those rules. Where are we going to attach our eBPF programs? This is important because there are a lot of different areas that we can do all of this. The lower down the stack, the more fine-grained control that you're going to have. However, if I'm attaching my eBPF program to the network card itself, which I can do. I can attach directly into the driver of the network interface card. I'll just be receiving raw frames, which is going to be very hard for me to manipulate and understand that that is basically a bunch of JSON packets come out of order. Some things get corrupted and will need to be resent. It's not just going to be that packet. I will see all traffic leaving and entering the box and things like that. In order to make life a bit easier, we're going to go a little bit higher up, and we're going to connect to the sockets area.Effectively, whenever our program dials out, tries to do a connect to a other IP endpoint, which will be the web server that we're doing our API call to, that is where our eBPF program will hook in. It will see everything that's actually happening and allow us to start making changes. This is what it looks like. Our process on the left would be our AI agent that is going to try and dial out to OpenAI. It will create a socket in userland. That socket will go through iptables if needed. It will go through an area called traffic control in the Linux kernel, and then finally onto the network card, and traffic will go where it needs to go. This is the different areas in terms of what it actually looks like. There are a bunch of areas where we can attach our eBPF program to. The most performant would be on the NIC driver, but that's just going to give us raw frames.It's very hard for us to actually turn those raw frames back into a data stream that makes sense to the application. Traffic control, again, very performant, but is inside the kernel, and again, it's just raw packets. Attaching our eBPF program to the socket allows the kernel to take care of all of the actual data itself, but we will be able just to see those streams.This is what it would look like. We have our running process. We have the concept of a proxy that sits next to it. We attach our eBPF program to the socket when a new socket is created, and with the process, without realizing it, has its traffic then redirected to our proxy. All transparent. We can also do this with ensuring that there's no restarts required to the application as well. We can actually force a new connect to occur, meaning our eBPF program will actually capture that traffic again transparent to the application itself. That proxy now can start to see these requests. This is what the request would typically look like. Here we can see we're using an old model, so we may want to inline change that model. The application itself doesn't need changing. It doesn't realize anything has actually changed. We're doing that transparently to the application itself.We may want to make changes to the input request as well, so we may want to change the prompt so that we get jokes about giraffes instead of other animals and things like that. As mentioned as well, we may want to cap on other areas. There are a number of different areas through the API we can start to assert control on those behaviors. This is what it would typically look like from a response perspective. Traffic goes from our AI agent. eBPF has redirected it to the proxy. The proxy has looked at it, may have made changes based upon some of the rules that we've applied. The proxy is then dialed out on behalf of the original process to the AI provider, and then we get our response back. The original question, what's the joke about giraffes? AI LLMs make terrible jokers. The jokes are usually quite terrible.The answer to, what do you call it when a giraffe swallows a toy jet, is a plane in the neck. We can see the response gives us the amount of tokens that we've required in order to generate that. We can then filter based upon all of that as well. We have the capability of understanding everything going in and out from there. Finally, encryption as well. We can effectively enforce encryption between applications. If you're running your LLM process internally within the Kubernetes cluster, you typically will find that there'll be no encryption between the two. We can effectively do the same thing where we add a proxy on the backend, and then effectively just do mTLS between the two as well. The applications are none the wiser. It decrypts on the other end, and it will just receive the traffic as it looked before.DemoWe're going to do a demo of pretty much everything that I've talked about. It is going to be quite a lot of terminal action that we're going to get right now. Kubernetes cluster up and running. There is no workload that's actually running on here at the moment. However, at the bottom is Ollama, which allows me to run LLMs locally within a Kubernetes cluster. What we're going to do now is we're going to deploy our very basic workload. We have an AI pod, which is effectively an AI agent. Very simple, all it does is dials to our LLM and asks it to tell a joke about Go every few seconds. We can quickly look at that. There we go. It's very simple. We can see here the model that it used, how many tokens were required in the prompt, and how many tokens were in the response.This is hard-coded, like we are starting to see. The model isn't great, doesn't give us very good jokes and things like that. What we're going to do is we're going to transparently modify the behavior of this hard-coded AI agent. What we have here is, I think, called a watcher. You have lots of watchers inside Kubernetes, and they all watch for events that are actually happening within the cluster. Our watcher that we've created as part of this proof of concept is going to watch for annotations being added to a workload. The first annotation I'm going to add to our workload is an annotation called netflush. I mentioned this previously, that annotation means when the agent spins into life, it will use eBPF to reforce a brand-new socket connection. Then, finally, one other annotation, AI="true". The watcher will have basically seen that annotation be added to our pod.Here we go. Right at the top, now we can see our three annotations. Three, I only added two. What's actually happened is when we added the AI="true", the watcher saw that, and it added an additional container to our pod. If we look down at the bottom here, we will see just underneath that we added an additional container to our pod. Typically, this isn't possible. Kubernetes has very strict rules on what you can and can't change once things have been applied to a workload. For instance, has anybody heard of the concept of a sidecar? A sidecar typically requires something watching for workloads being applied, captures it before the API server, modifies it, adds the sidecar, and then it goes. Once the API server has started the workload, you can't add sidecars to it. What we've done here is we've added an ephemeral container to it, which is a relatively new thing added to the Kubernetes API, which allows us to add an additional workload next to the existing one.We now have our gateway set there. If we look at the logs, we can see our gateway has actually started, and it's doing a few things here. We can see here the netflush is true, and it's going to look for all of the networking sessions that already existed. It's already found our AI client. It's found its process ID, so it knows to look for that particular process. It's found that it already had a connection between itself, and that's the Ollama, so it knows where it was actually going. It's forced it to do a brand-new connection, so it's now speaking to the proxy, and now the proxy is now speaking to Ollama. We are now in complete control of all of the communication between our AI agent and whatever it's actually speaking to. Let's control it. I've added a config map. I'll quickly show you this config map so you know what I'm actually talking about.This is a policy that we're going to apply. This policy is going to do two things. One, it's going to swap the model out, so it's going to swap the old Llama model for this Gemma 2 model, and further down, instead of it saying jokes about Go, it's actually going to give us facts about Go. We can quickly see this now. Here we go. The model is now Gemma 2 as opposed to Llama, and the responses are facts as opposed to jokes about Go and things like that as well.We can demonstrate this a bit better. The config map has been changed, and we'll now see in the responses that instead of telling us jokes about Go, it's going to give us jokes about horses instead. Here we go. Horses have three kinds of teeth. Very useful information. We now have the capability of transparently to the application swapping out things like the model that it's using, changing the prompt behavior, omitting keywords and things like that. There may be things where we want to block certain behaviors. Let's have facts about rabbits. In a second, we'll see that this will be blocked because I'll show you, we have a filter which effectively says that if the response contains anything about rabbits, it shouldn't be allowed to come back to the actual user itself. There we go. Kube-gateway says no. We've effectively now transparently stopped any of those sorts of responses that contain keywords that we don't actually want to allow.Then, finally, from an observability perspective, we may want to actually understand what things look like from an API perspective. Next time there is a request, we'll be able to see what all of that actually looks like. Again, transparently to everything that's actually occurring within here. We can see here, earlier it found the thing about rabbits. Now we can actually see the JSON payload that it is actually sending over to the LLM that we are actually then parsing and determining if things should change, if we should block things, and things like that. Then, finally, we can see both the request and the response now. We can see the original request was telling me a joke about Go. We've actually changed a bunch of it. We can now see that the response is telling us about rabbits or frogs, and things like that. We've got frog facts or frog jokes. Transparently, we've changed the behavior of a running AI agent without it actually knowing that anything has actually changed at all. That's the demo.AI Programs - Observability and Security with eBPFAdditional ways that we can now use eBPF to apply additional security and observability. Everything we've talked about today from an observability and understanding perspective has all been around the networking and the API side of things. With eBPF we can hook into a variety of different areas inside the running system. There was a talk which talked a lot about how a lot of these AI agents will be shelling out additional tooling to eventually effectively do what tasks they are trying to do. With eBPF, we can hook in to the kernel itself and say, if anybody tries to run rm, just simply don't allow that program to ever actually run. If somebody tries to delete /etc/passwd or open /etc/passwd or anything like that, we can hook into the eBPF area of the file system and effectively say, this process is not allowed to do that. As shown, we can control the networking behavior to transparently redirect traffic to somewhere else so that we can do things transparently to the application itself.eBPF can enforce which syscalls are allowed as well, so if something is trying to elevate privileges, which there was another demo recently of an AI agent asked to do something and it didn't have the privileges to do it, and it effectively started messing around in the proc file system to get another root access to it. It basically broke itself out of the sandbox. We can typically say just that behavior is not allowed, you simply can't do that. You don't have the privileges to do these sorts of things. It's pretty limitless at the moment in terms of what we can allow and what we can block within a running system.SummaryThere's a lot of things happening at the moment. It's super easy these days to generate some AI code that scratches an itch that you have. Very quick and easy for that code to end up in production, for things to actually depend on it. No real ownership. These things are being thrown over the fence a lot of times because people are moving so quickly onto the next thing that no one is really taking ownership of what it is that they've created. Once these things are often dropped into production, that person who threw that together and has moved on, like, who's going to own that? Who's going to update that, should it need updating, should there be security issues in the libraries that it depends on, the model that it's using, the prompts that it's making use of? That can become a really large issue. As seen, we need a method for clear control over what is allowed from these parts of AI programs and AI agents.What I showed you today is a proof of concept based upon the AI egress working group as part of the Kubernetes project. A lot of different companies at the moment, ourselves included, are working together to come up with a standardization for having rules and an agreed upon approach in order to observe, control, and determine the behavior of AI-based applications within Kubernetes clusters. However, what I showed you today is also possible outside of a Kubernetes cluster as well. We're effectively redirecting traffic and parsing the API calls and things like that as well.ResourcesIf you want to learn more, ebpf.io is a fantastic resource. If you want to learn eBPF code, it's all there. On the Cilium lab section, so we have a number of free labs, you can go there. I think there's 45 different labs, but there is one about eBPF. If you want to learn about managing eBPF programs, we have free lab environments that you can basically just hit with a web browser, play with, and learn more. If you're really into eBPF, there is a documentary that you can watch. I think it's about an hour long. It details how eBPF came into creation. eBPF is pretty much everywhere at this point. It's on all Android phones. Facebook use it to manage the networking for all of their Meta fleet and things like that. It's in most Kubernetes clusters, either doing observability, networking, or enforcement of behaviors and things like that as well. There is also Liz Rice's book, "What Is eBPF?" available to download from the website as well.Questions and AnswersParticipant 1:Can you also show the eBPF code?Dan Finneran:Yes. This is the code that powers everything that I talked about today. Now, I can step you through this if you really want to. Effectively, this bit of code here, if you look at the area where it says SEC, that's the section header, and connect4, that is a TCP IPv4 connection event, which is what happens when you connect out to something. This is where our code will actually run before the kernel will do anything. eBPF code always happens before the kernel, or whatever it is will actually process it. It allows us the capability to change things and enforce behaviors before they can actually impact a running system. Typically, what this will do is, if we care about it, we will lower down. We will change the destination to where it's actually going, so we hijack the traffic in the eBPF, sending it to our proxy. This is all open source. There's a number of examples of doing this sort of thing, but yes, this is a quick example of eBPF code hooking into IPv4 TCP connect events.Bryant: Is there any advice on how to document some of this stuff? I'm an old-school Java dev, and I got into aspect-oriented programming, similar vibes but at the JVM level, and it became a nightmare to debug some stuff, because I had these aspects cutting in, and I was almost injecting functionality arbitrarily. How do I debug it? How do I understand what's going on? I'm conscious that it's very powerful, but is it a bit of a foot gun as well?Dan Finneran:We did this example on the webpage. When I first joined Isovalent, the first example I had was, look at this, connect this to running on your laptop, and it will print out all these events, and it will return ok. Now try saying return not ok, which all of a sudden meant all packets were just dropped. I basically denial of service my laptop from the example. It is incredibly easy to do that. There's no debugger. All you've got are effectively bpf_printk events, so that is a kernel logging mechanism. Effectively, you can write logs to the kernel, and the kernel will output those in sys tracing debug, trace pipe, or whatever it is. That's the only way you can understand what's actually happening in there. This is the thing. Not everybody needs to write eBPF code. The idea really being is that you write it once, expose the bits that you need to do. As I say, for all the networking side of stuff that you see in production, like from Cilium and Calico and things like that, you don't need to know eBPF. You just know that it's powering your network. It's looking at traffic, and it's redirecting it, and doing things like that.Bryant: The producer and the consumer are quite different people in some regards there. You produce it, you know all the stuff, but I'm a consumer at the other end.Dan Finneran:Yes. If you really want to learn eBPF, then brilliant. I think I've done my job. It's a bit like driving a car. You don't really need to know how the engine works, especially not with today's modern electric cars, because that is bonkers. It's the same with this. Use the things powered by eBPF. If you want to learn about eBPF, that's great, but you don't need to know the fundamentals fully about what it's doing.Participant 2:You mentioned restricting syscalls towards the end. I'm wondering, at what point does that just become an arms race, and you just have to sandbox it, since I can imagine it could start cloning itself, forking.Dan Finneran:The hooks that we apply are effectively events. Our eBPF code, we would have, for instance, file open. We would create our eBPF code, which attaches to the syscall file open, or open. Our code would execute first. If an AI agent tries to open /etc/passwd, our eBPF program will be the first thing that actually happens before the kernel syscall actually occurs. We can always be one step ahead of anything that's actually happening within the system itself. We see that syscall and we say, no, you're not allowed to do it, we return false, effectively that's a stop dead, there's nothing it can really do. It can keep trying that, or try a different syscall, but whatever it is, that is really the only mechanism for opening files through syscalls in the Linux kernel. It's always going to trigger our event. There is syscall ret, so we can actually attach to the return of the syscall. We can allow the kernel to run that syscall, and then we can capture the results of all of that, and have eBPF code that does something based upon the return of that as well. We can attach to the before and after the syscalls.See morepresentations with transcriptsRecorded at:Aug 21, 2026byDan FinneranRelated SponsorsThis content is in theDevOpstopicRelated Topics:DevelopmentArchitecture & DesignDevOpsAI, ML & Data EngineeringObservabilityAPI GatewayMonitoringKubernetesQCon London 2026APIeBPFLinuxCloud-NativeQCon Software Development ConferenceTranscriptsapplication performance managementContainersArtificial IntelligencePerformanceOperating SystemsSecurityArchitectureCloud ComputingInfoQRelated EditorialPopular across InfoQCloudflare Migrates JavaScript CDN Serving 9B Requests a Day to Its Developer PlatformHow PGSimCity Turns PostgreSQL Complexity into a Virtual City 3D SimulationNetflix Open-Sources Agentic Workflow for Causal InferenceCloudflare Turns CI Pipelines into TypeScript WorkflowsAWS Open-Sources Dogwood, Extending Cedar to Govern Sequences of Agent Tool CallsWill Agentic AI Bring Fantasia’s Sorcerer's Apprentice to Life?: A Conversation with Tracy BannonDevelopmentHow PGSimCity Turns PostgreSQL Complexity into a Virtual City 3D SimulationLLM-Generated GraphQL Mocks Arrive at Airbnb and Expedia, While the Spec Lags behindAdopting Memory-Safety and Fine-Grained Compartmentalisation with CHERIArchitecture & DesignWhatsapp Tests on Device ML for Scam Detection with Privacy Preserving AnalyticsUnderstanding Progressive Collapse: How To Avoid A Cascading FailureGrab Cuts Mechanical Analytics Work from 44% to 30% with AI AgentsCulture & MethodsHow Code in the Age of Artificial Intelligence Becomes Write-Only and DisposableTurning Outward: Growing From Code to InfluenceFounders, Friction, and Focus: Building Engineering Teams at Early-Stage StartupsAI, ML & Data EngineeringInfoQ Opens Enrollment for New AI-Assisted Engineering Online Certification ProgramHarper Argues Against the Multi-System Stack and Releases 5.2The Open-Sourcing of DeepSeek Harness Opens the Door to Modular, Unbundled AI Agent InfrastructureDevOpsWhy Fetch When You Can Sync? Building Local-First Apps on a Sync Engine ArchitectureFlux Mirror Uses Gitless GitOps to Keep Software Supply Chain Under ControlDocker Launches Fully Rebuilt Virtualization Layer to Boost Performance and Improve Dev ExperienceThe InfoQNewsletterA round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers.View an exampleGet a quick overview of content published on a variety of innovator and early adopter technologiesLearn what you don’t know that you don’t knowStay up to date with the latest information from the topics you are interested inEnter your e-mail addressSelect your countrySelect a countryI consent to InfoQ.com handling my data as explained in thisPrivacy Notice.We protect your privacy.InfoQ Online Certification ProgramsFor Senior Engineers, Architects, and Technical LeadersAI Security & Privacy Engineeringwith Katharine Jarmul | August 26Architectwith Luca Mezzalira | September 14Engineering Leadershipwith Michelle Brush | September 18AI-Assisted Engineeringwith Zichuan Xiong & Premanand Chandrasekaran | September 18As your role becomes more senior, the work changes. You're no longer just implementing decisions; you're shaping the systems, trade-offs, and technical direction other teams depend on.These 5-week online programs give you a structured way to work through real decisions from your current role with senior peers from other companies.RESERVE YOUR PLACELive online. 4 hours a week, for 5 weeks.HomeCreate accountLog InQCon ConferencesEventsWrite for InfoQInfoQ EditorsAbout InfoQAbout C4MediaMedia KitInfoQ Developer Marketing BlogDiversityEventsOnline InfoQ AI Security & Privacy Engineering ProgramAugust 26, 2026Online InfoQ Architect CertificationSeptember 14, 2026Online InfoQ Engineering Leadership CertificationSeptember 18, 2026Online InfoQ AI-Assisted Engineering CertificationSeptember 18, 2026QCon San FranciscoNovember 16-20, 2026QCon AI New YorkDecember 15-16, 2026QCon London 2027April 13-16, 2027Follow us onYoutube232K FollowersLinkedin26K FollowersInstagramNewRSS19K ReadersX57.1k FollowersFacebook21K LikesBlueskyNewStay in the knowThe InfoQ PodcastEngineering Culture PodcastThe Software Architects' NewsletterGeneral Feedbackfeedback@infoq.comAdvertisingsales@infoq.comEditorialeditors@infoq.comMarketingmarketing@infoq.comInfoQ.com and all content copyright © 2006-2026 C4Media Inc.Privacy Notice,Terms And Conditions,Cookie PolicyBT<img height="1" width="1" src="https://www.facebook.com/tr?id=842388869148196&ev=PageView&noscript=1"/>

User Query

Result

Source

https://www.infoq.com/presentations/ebpf-ai-gateway-kubernetes-security/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=DevOps

Response

At QCon London, Isovalent engineer Dan Finneran demonstrated a proof‑of‑concept that uses eBPF — the Linux in‑kernel programmable framework — to build a transparent AI API gateway inside Kubernetes, showing how lightweight programs attached to socket and syscall events can intercept an AI agent’s outbound traffic, redirect it to a userland proxy, swap models and prompts, cap tokens, block unwanted responses, enforce mTLS, and even forbid dangerous file or syscall activity without altering the application itself; the live demo used Ollama LLMs and a watcher that injects an ephemeral container when pods are annotated (AI="true", netflush) so existing connections are forced to reconnect through the proxy, while Finneran stressed that eBPF’s verifier and tooling protect kernel stability but warned of debugging limits (kernel logs/bpf_printk) and the potential for misconfiguration to cause outages — a capability now being debated for standardization by the Kubernetes AI egress working group, with learning resources such as ebpf.io, Cilium labs, and Liz Rice’s primer recommended for operators.