Under the Hood

Imp turns Kubernetes objects into Firecracker microVMs. Its control plane decides what should exist. Its node agent does the host work that must happen on the selected machine.

This page separates current source behaviour, the implemented runner lifecycle, and live proof. That boundary matters: implemented source code is not the same thing as a working runner pool in a cluster.

Three views of the system

View	What it answers	Status
Current implementation	What the current Imp source is designed to do	Source-backed
Runner lifecycle implementation	How runner setup, ownership, cleanup, and replacement work in source	Source-backed
Demonstration evidence	What has been proven in unraid-lab	Verified for one controlled run

Current implementation

1. One desired state, two execution roles

flowchart LR
  user[User or controller] --> api[Kubernetes API]
  api --> operator[Imp operator]
  operator --> vm[ImpVM desired state]
  vm --> agent[Node agent]
  agent --> fc[Firecracker microVM]
  agent --> status[ImpVM status]
  status --> api

Plain explanation: the operator owns the cluster-wide decision. The node agent owns work that can only happen on the node: disks, networking, Firecracker, and guest communication.

Evidence in the repository:

• API contracts: api/v1alpha1/
• Pool reconciliation: internal/controller/impvmrunnerpool_controller.go
• Node-local runtime work: internal/agent/
• Firecracker/VSOCK path: internal/agent/vsock/

2. A runner VM is built from layers, then started

flowchart LR
  base[Base OCI image] --> rootfs[Ext4 root filesystem]
  runner[Runner OCI layer] --> rootfs
  agentbin[Injected guest agent] --> rootfs
  rootfs --> disk[VM disk]
  disk --> fc[Firecracker]
  fc --> guest[Guest agent over VSOCK]

Plain explanation: Imp does not run a container as the job. It builds a VM filesystem from container image layers, starts a microVM, and talks to a small guest agent over VSOCK, a host-to-guest channel.

When the guest agent is enabled, Imp writes a small /.imp/init wrapper into the VM disk. That wrapper starts the guest agent in the background, then runs the image's normal /sbin/init. Runner launch happens later through VSOCK.

Evidence in the repository:

• Root filesystem assembly: internal/agent/rootfs/
• Guest init injection: internal/agent/rootfs/init.go
• Runner launch boundary: internal/agent/runnerlaunch/launcher.go

3. An ImpVM is not a Kubernetes Pod

The Imp agent is a Kubernetes DaemonSet Pod. Kubernetes schedules that Pod onto a Linux node. The agent container mounts the host's /dev/kvm, the Firecracker binary, and the guest kernel.

An ImpVM is different. It is not another Pod created for the job. Once an ImpVM is assigned to a node, the agent starts Firecracker as a Linux process on that node. Firecracker uses KVM through /dev/kvm to boot a separate guest kernel and the ext4 root filesystem Imp built from OCI layers.

flowchart TD
  k8s[Kubernetes schedules one agent Pod per enabled node] --> agent[Linux Imp agent Pod]
  agent --> kvm[Host /dev/kvm]
  agent --> kernel[Guest kernel image]
  agent --> rootfs[Ext4 root filesystem built from OCI layers]
  kvm --> fc[Firecracker Linux process]
  kernel --> fc
  rootfs --> fc
  fc --> guest[Guest Linux kernel and processes]

Plain explanation: a Pod runs container processes that share the node's Linux kernel. An ImpVM is a guest Linux system with its own kernel. Kubernetes knows the agent Pod; Imp records the VM as an ImpVM custom resource and starts its Firecracker process on the selected node.

	Agent Pod	ImpVM
Kubernetes object	DaemonSet Pod	ImpVM custom resource
Who selects the node	Kubernetes scheduler	Imp's VM scheduler
What starts the work	Kubelet starts the agent container	Agent starts a Firecracker process
Kernel seen by workload	Node's shared Linux kernel	A separate guest Linux kernel
Image use	Container image becomes the process filesystem	OCI layers become an ext4 VM disk
Main runtime record	Pod status	ImpVM.status.runtimePID and phase

Evidence in the repository:

• Linux-only Firecracker driver: internal/agent/firecracker_driver.go:1-4
• Agent DaemonSet and host mounts: charts/imp/templates/agent/daemonset.yaml:1-104
• Root filesystem, kernel, KVM, and Firecracker launch: internal/agent/firecracker_driver.go:153-323, 473-530
• VM runtime PID: api/v1alpha1/impvm_types.go:159-175

4. GitHub App to one runner

sequenceDiagram
  participant O as Pool controller
  participant G as GitHub API
  participant S as VM-owned Secret
  participant A as Node agent
  participant V as Guest VM

  O->>G: App JWT
  G-->>O: Installation token
  O->>G: JIT runner setup
  O->>S: Store one-time setup
  A->>V: VSOCK Exec environment
  V-->>A: Runner exits
  A->>S: Delete setup

Plain explanation: the GitHub App key stays with the operator. The one-time setup reaches the guest through the VSOCK execution environment. It is not baked into the VM disk or placed in command arguments.

Evidence in the repository:

• App JWT and installation-token exchange: internal/runner/githubapp.go
• JIT configuration contract: internal/runner/jitconfig_contract_test.go
• VM-owned JIT Secret: internal/controller/impvmrunnerpool_controller.go
• VSOCK handoff and post-exit deletion: internal/agent/runnerlaunch/launcher.go

For the full chronological paths, read how a runner is configured and how a queued job reaches a runner.

5. The v0.10.1 runner image boundary

Runner images are assembled from two OCI layers so the operating-system base and CI software can be versioned independently:

flowchart LR
  base[Debian-slim glibc base] --> compose[Imp rootfs composer]
  layer[Actions runner OCI layer] --> compose
  compose --> disk[Cached ext4 disk]
  disk --> boot[Firecracker guest boot]
  boot --> wrapper[/usr/local/bin/runner]
  wrapper --> jit[One-time JIT setup via VSOCK Exec]
  jit --> job[Register and run one job]

The base supplies the guest userspace, certificates, .NET runtime libraries, and an unprivileged runner user. The runner layer supplies /home/runner/actions-runner and the wrapper executable. Firecracker attaches the guest NIC; the image/wrapper is responsible for bringing up eth0 and DNS before the runner registers. Alpine/musl images are not the default contract because compatibility with the official Actions runner is not established.

The JIT value is deliberately not part of either image, the VM manifest, the rootfs cache key, or a command-line argument. The operator creates a VM-owned one-time Secret, the node agent reads that exact Secret, and guest-agent Exec passes the value through the process environment. Cleanup deletes the Secret after the handoff/runner boundary, with bounded failure handling if the guest or node disappears.

Security invariants

Must happen	Must not happen
Keep App credentials in the operator-side credentials Secret	Put App keys or JIT data in ImpVM.spec.env
Send the one-time setup through guest Exec environment	Bake JIT data into the root filesystem or its cache key
Keep the JIT Secret owned by exactly one VM	Print tokens, private keys, or JIT payloads in logs
Delete the JIT Secret only after runner exit or bounded cleanup	Treat request acceptance as proof that the runner completed

Runner lifecycle implementation

The runner lifecycle is implemented in the current source. The pool controller creates an ephemeral VM, creates or safely recovers its VM-owned JIT Secret, and removes the new VM if setup cannot be minted or persisted. The agent hands the one-time setup to the guest and removes that Secret after the runner exits. Terminal VMs are deleted and the next reconciliation restores capacity within the pool limits.

flowchart TD
  reconcile[Pool controller reconciles capacity] --> vm[Create ephemeral VM]
  vm --> mint[Mint or recover VM-owned JIT Secret]
  mint --> handoff[Agent delivers one-time setup]
  handoff --> exit[Runner exits]
  exit --> cleanup[Agent deletes setup; controller deletes terminal VM]
  cleanup --> replace[Pool restores capacity if needed]

  mint -. mint or persistence failure .-> remove[Delete the new VM]

Plain explanation: one VM owns one setup Secret. If setup cannot be made safely, the controller removes that VM instead of letting it boot without a runner. When the runner finishes, the temporary setup and terminal VM are removed, then the pool can create replacement capacity.

The implementation also validates the named credential fields before use, keeps JIT data tied to the VM identity, and avoids creating a second Secret when a reconcile repeats after an interrupted write.

This is source-backed implementation. A controlled live run has also proven the registration, assignment, successful completion, one-time Secret cleanup, and ephemeral runner deregistration path in unraid-lab; see Demonstration evidence for the exact workflow and release references. The evidence proves that run, not every possible pool configuration or a VM deletion event.

Implementation evidence:

• internal/controller/impvmrunnerpool_controller.go:284-391
• internal/runner/auth.go
• internal/agent/runnerlaunch/launcher.go:73-145

Read the system in order

• Architecture for the core operator/agent split.
• Runner pool for capacity and job execution.
• Runner configuration for the GitHub App and JIT setup flow.
• Job scheduling for capacity, node placement, and GitHub job assignment.
• Lifecycle for VM lifecycle concepts.
• Demonstration evidence for the live-proof boundary.
• 20-minute technical presentation for a speaker-ready walkthrough.