about this goblin thing, i made a video about! it was a lot of work, crazy. i had no idea how much work goes into putting yourself out there like this. but i did it! and i also wrote an article on my employer’s website, which i’m reposting below my video, enjoy!

I was wondering how little power I could give an AI agent and still make it useful. I ended up writing a custom AI agent as part of a custom Kubernetes operator. I called it the Goblin Operator, and the agent being the Goblin Scout. You can find all the code here: github.com/sguldemond/goblin-operator

How a Devoteam Tech Friday Sparked the Idea

In our Cloud Native team in the Netherlands, we organise a regular Tech Friday where we share our observations and findings on Kubernetes, cloud, AI and more. During one of these gatherings, my colleague Gopal Ramachandran showed a demo in which multiple AI agents worked together to fix a failing Pod. It was a short demo, showing off the capabilities of AI agents in cluster maintenance.

At that same gathering, there was a lot of interest within the team in getting more hands-on experience with writing Kubernetes operators. I had already started exploring operators by writing a small one with Kubebuilder, but I was struggling to find a good use case. Seeing the demo made it click: embed an AI agent in an operator’s lifecycle to help fix cluster issues.

How it works

  • Everything starts with an IncidentPolicy, one of the operator’s custom resources. It defines when an incident should be logged and which permissions the agent that picks it up receives.
  • A controller loop matches these policies against the cluster state and creates an Incident object whenever it detects a match.
  • The AI agent watches for these Incidents and picks them up as they come in.
  • The agent has some default read permissions to gather context around the Incident, such as reading its status and possibly some logs.
  • If the policy grants it the right permissions, the agent can propose a specific fix to resolve the Incident.
  • As the end user, you can discuss the solution with the agent and approve it if you agree.
  • The agent implements the fix, the Incident is resolved, and the entire flow is logged.

Taming the Agent: Securing AI Control in Kubernetes

Following the principle of least privilege, I implemented what I call “dynamic RBAC”. The agent only gets the permissions relevant to fixing a specific type of incident, and those permissions are automatically revoked once the incident is resolved. And because it’s a custom agent loop, human approval before implementing a fix is hard-coded into the logic. The context is limited as well, to the direct surroundings of the objects related to the incident. There’s no need for the agent to read the entire cluster.

Zooming out

The capabilities of this operator are still limited, and it hasn’t been proven in a production environment. I see this project as a reference implementation: an inspiration for how AI agents can be used in a wider context while keeping tight control over what they’re allowed to do.