# Kernel Features Behind Containers

> **Supported Versions**: Linux 6.1 / 6.12 / 6.18 (Amazon Linux 2023), Kubernetes 1.25+ (cgroup v2)
> **Last Updated**: October 16, 2026

## What This Document Covers

- Which kernel features combine to make a container, and the fact that there is no "container" kernel object
- What actually changed operationally in the cgroup v1 → v2 move (especially OOM diagnosis)
- Where netfilter and conntrack sit in Kubernetes networking

## First: There Is No "Container" in the Kernel

This is the starting point for understanding containers. There is no `struct container`, and no single syscall that creates one.

A container is **a convention built by combining several independent kernel features**. When a runtime (containerd, runc) starts a process, it applies these together.

| Purpose | Kernel feature |
|---|---|
| **What can it see** (isolation) | namespaces |
| **How much can it use** (limits) | cgroups |
| **What can it do** (privileges) | capabilities, seccomp, LSM (AppArmor/SELinux) |
| **How is the filesystem composed** | overlayfs (union mount) |
| **How does traffic flow** | veth, bridge/route, netfilter |

Two practical conclusions follow from it being a combination.

**First, isolation is not all-or-nothing.** Some namespaces can be shared while others are isolated. A Kubernetes Pod is exactly that: containers in the same Pod **share** the network and IPC namespaces while mount and PID namespaces are usually **separate**. That is why containers in a Pod can reach each other over `localhost` (shared network) but cannot see each other's filesystems (separate mounts).

**Second, isolation you forget becomes a silent hole.** The kernel does not know "make a container," so if the runtime does not apply a seccomp profile, the workload simply runs without one. This is why container security is a matter of runtime and policy configuration.

## Namespaces: What Can It See

A namespace **separates the "name space" of a kernel resource**, so the same name or number refers to different things in different namespaces.

| Namespace | Isolates | In a Pod |
|---|---|---|
| **mnt** | Mount points | Per container |
| **pid** | Process IDs | Per container (`shareProcessNamespace: true` shares within the Pod) |
| **net** | Interfaces, routing tables, netfilter rules, sockets, ports | **Shared per Pod** |
| **ipc** | System V IPC, POSIX message queues | Shared per Pod |
| **uts** | hostname, domainname | Shared per Pod |
| **user** | UID/GID mapping | Not used by default (see below) |
| **cgroup** | cgroup root path | Per container |
| **time** | Boot time, monotonic clock (5.6+) | Not used |

### Why the net namespace is the Pod boundary

A Pod's identity is decided here. Kubernetes creates one net namespace per Pod (held by the pause container) and puts **all of that Pod's containers into the same net namespace**.

What follows:

- Containers in a Pod share **the same IP and the same port space** → two containers in one Pod cannot both bind 8080
- `localhost` communication works → the foundation of the sidecar pattern
- **netfilter rules are also per net namespace** → this is why a sidecar mesh's init container can install iptables rules inside the Pod's net namespace, and why those rules do not affect the whole node (see [VPC Lattice Kernel Datapath](https://docs.atomai.click/kubernetes-docs/llms/en/service-mesh/vpc-lattice/07-kernel-datapath.md))
- Routing tables are separate too → `ip route` inside a Pod is not the node's

### user namespaces: why they were not the default for so long

A user namespace maps container UIDs/GIDs to a different host range. This reduces the privileges of many escaped operations but does **not guarantee** containment against a kernel vulnerability or other privilege-escalation path.

Yet it was not the default for a long time. The reason is **file ownership.** Files on a volume are recorded with host UIDs; if the container sees a different UID, permissions do not line up. Solving it requires translating UIDs at mount time (idmapped mounts, kernel 5.12+), and storage drivers and CSI must support it too.

### Kubernetes user namespace support status

Per the [Kubernetes feature gates reference](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) and the [v1.36 GA announcement](https://kubernetes.io/blog/2026/04/23/kubernetes-v1-36-userns-ga/), the maturity stages are (design: [KEP-127](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/127-user-namespaces/kep.yaml)):

| Stage | Version |
|---|---|
| alpha | v1.25 |
| beta | v1.30 (enabled by default since v1.33) |
| **stable (GA)** | **v1.36** |

The feature gate is `UserNamespacesSupport`, applying to kubelet and kube-apiserver. **From 1.36 it is GA, so `hostUsers: false` works without enabling a feature gate.**

::: warning Needs verification
Those maturity stages are upstream Kubernetes. **Whether EKS offers that version, and whether your container runtime and CSI drivers support idmapped mounts, are separate questions.** Confirm the EKS supported version and your runtime/storage combination before adopting.
:::

## cgroups: How Much Can It Use

cgroups (control groups) **measure and limit resource use for a group of processes.** This is where Kubernetes `requests`/`limits` ultimately land.

### Structural differences between v1 and v2

| Item | cgroup v1 | cgroup v2 |
|---|---|---|
| **Hierarchy** | A **separate tree per controller** (cpu, memory, blkio…) | **A single unified tree** |
| **Process membership** | Can be in different groups per controller | Belongs to exactly one group |
| **Memory + IO cooperation** | Hard (separate trees cannot coordinate) | Possible (same tree) |
| **Pressure information** | None | **PSI** (`cpu.pressure`, `memory.pressure`, `io.pressure`) |
| **CPU limit representation** | `cpu.cfs_quota_us` / `cpu.cfs_period_us` | `cpu.max` (one file: "quota period") |
| **Memory limit representation** | `memory.limit_in_bytes` | `memory.max`, plus `memory.high` (soft pressure) |
| **AL2023 EKS AMI** | - | **Default** |

Where v1's "separate tree per controller" actually hurt was **coordinating memory reclaim with IO.** When memory runs low and page cache must be dropped, that reclaim itself causes disk IO, and in v1 the two controllers knew nothing about each other. v2's unified tree handles both in the same hierarchy.

### October 2026 Update: cgroup v1 deprecation moving forward

The Kubernetes blog summarized the state of the cgroup v1→v2 transition ([The Shift to cgroup v2 in Kubernetes](https://kubernetes.io/blog/2026/10/06/kubernetes-cgroups-v2-shift/), 2026-10-06). cgroup v2 management has been stable since v1.25 and cgroup v1 support has been in maintenance mode since v1.31; **starting with v1.35, the kubelet's `failCgroupV1` defaults to `true`, so the kubelet no longer starts on a cgroup v1 node by default.** Setting `failCgroupV1: false` in the kubelet configuration is a temporary override; full removal is tracked in KEP-5573. On kubeadm clusters, the SystemVerification preflight check treats cgroup v1 as an error with kubelet v1.35 or later (a warning on older kubelets). If you are on a release older than v1.35, migrate every Linux node to cgroup v2 before upgrading.

### The biggest operational change: OOM diagnosis

Here is the fact you must know about cgroup v2.

> **`memory.current` includes page cache.**

So even when the memory the application actually holds (anon/RSS) is far below the limit, reading many files piles up page cache and pushes `memory.current` to the limit.

There is an important distinction here. **Page cache is reclaimable.** In the normal case the kernel hits the limit, drops page cache to make room, and no OOM occurs. The problem arises **when reclaim cannot keep up with allocation**. That is when the OOM killer acts.

Practical implications:

| Misconception | Reality |
|---|---|
| "`memory.current` near the limit means OOM is imminent" | It may include reclaimable file cache; inspect anon/file/kernel usage and pressure instead of assuming its composition |
| "Just look at RSS" | OOM can happen with low RSS (when reclaim can't keep up) |
| "Raising the limit fixes it" | If the cause is slow reclaim, it will recur |

**Values to look at when diagnosing:**

| File/value | Meaning |
|---|---|
| `memory.current` | Current usage (includes page cache) |
| `memory.stat` → `anon` | Anonymous memory (what the application actually holds) |
| `memory.stat` → `file` | Page cache |
| `memory.events` → `oom` / `oom_kill` | OOM occurrences / kills |
| `memory.events` → `high` / `max` | Times the soft/hard limit was hit |
| `memory.pressure` (PSI) | **Fraction of time stalled** by memory pressure |

**PSI is especially useful** because it reports **pressure (how long you waited as a result)** rather than usage (how much you use). A rising `some avg10` in `memory.pressure` means time is being spent on reclaim, something a usage graph alone will not show.

### CPU limits and throttling: why low utilization can still be slow

A CPU limit is a **bandwidth limit.** `cpu.max` of `20000 100000` means "up to 20ms per 100ms period."

Something counterintuitive follows. If the application does its work in a short burst across several threads, it **spends the whole quota early in the period and is forcibly stopped until the period ends.** Average utilization looks like a low 20% while latency spikes.

It is worse with multiple threads. With 4 threads running concurrently, a 20ms quota is consumed in 5ms of wall time. The remaining 95ms is waiting.

**Diagnosis:** `cpu.stat` → `nr_throttled` (periods throttled) and `throttled_usec` (total throttled time). If `nr_throttled` is a meaningful fraction of `nr_periods`, the limit is the cause.

**Directions for response** (see [Resource Optimization](https://docs.atomai.click/kubernetes-docs/llms/en/ops/10-resource-optimization.md) for request/limit design):

- Raise or remove the limit (trading off node stability)
- Align the application's thread count with the limit (JVM `-XX:ActiveProcessorCount`, Go `GOMAXPROCS`, etc.); **a mismatch between the CPU count the container perceives and its actual quota is often the root cause**
- Check actual stall time with `cpu.pressure` PSI

## Privileges: What Can It Do

Even with isolation (namespaces) and limits (cgroups) in place, reducing the **actions** a process can take is a separate layer.

| Feature | What it does | In Kubernetes |
|---|---|---|
| **capabilities** | Grant/drop root privileges as fine-grained units (`CAP_NET_ADMIN`, `CAP_SYS_ADMIN`, …) | `securityContext.capabilities.add/drop` |
| **seccomp** | Restrict the allowed set of **syscalls** | `securityContext.seccompProfile` (`RuntimeDefault` recommended) |
| **LSM** (AppArmor/SELinux) | Control file and network access by policy | `securityContext.appArmorProfile`, etc. |
| **no_new_privs** | Block privilege escalation via setuid binaries | `allowPrivilegeEscalation: false` |

The three layers answer different questions: capabilities ask "do you hold this privilege," seccomp asks "may you call this syscall," LSM asks "may you touch this object." **One alone is insufficient; layering them is the norm.**

`CAP_NET_ADMIN` deserves a mention. A sidecar mesh's init container needs it to install iptables rules, which is why adopting a mesh raises the security-review question "why does this Pod hold NET_ADMIN?"

## netfilter and conntrack: The Reality of Kubernetes Networking

### netfilter

netfilter is a framework providing **hooks** at defined points in the kernel network stack. `iptables`, `nftables`, and `ipvs` are all userspace tools using those hooks, or implementations on top of them.

Main hook points:

| Hook | When |
|---|---|
| `PREROUTING` | Packet arrives, **before** the routing decision (the DNAT point) |
| `INPUT` | Packets bound for local processes |
| `FORWARD` | Packets passing through |
| `OUTPUT` | Packets leaving locally |
| `POSTROUTING` | **After** routing, just before leaving (the SNAT/MASQUERADE point) |

Where these hooks are used in Kubernetes:

- **Service ClusterIP → Pod IP translation**: DNAT at `PREROUTING`/`OUTPUT`
- **Source translation for Pod → external traffic**: MASQUERADE at `POSTROUTING`
- **NetworkPolicy**: CNI inserts rules at `FORWARD` and elsewhere (Calico's iptables dataplane)
- **Sidecar mesh traffic interception**: REDIRECT at `OUTPUT`/`PREROUTING` inside the Pod net namespace

### kube-proxy modes: iptables, IPVS, nftables

Service implementation comes in three flavors, and **the landscape shifted in 2025–2026.**

| Mode | Rule evaluation | Status |
|---|---|---|
| **iptables** | Rule-chain lookup cost depends on rule layout; current kube-proxy optimizes updates | Default where not explicitly changed; verify the installed implementation |
| **IPVS** | In-kernel L4 load balancer, hash-based O(1) | **Deprecated in Kubernetes 1.35 (December 2025)**; planned default disablement in 1.40 and removal in 1.43 |
| **nftables** | O(1) lookup plus **incremental rule updates** | **GA in Kubernetes 1.33** (alpha 1.29 → beta 1.31). Requires **kernel 5.13+** on worker nodes |

How to read this:

- **In large clusters, the iptables-mode bottleneck is rule count and update cost.** The more Services and Endpoints, the longer kube-proxy's sync takes, and during that window the rules are not current.
- **If you run IPVS, you need a migration plan.** Upstream plans default disablement in 1.40 and removal in 1.43; confirm the maintained [KEP-5495 schedule](https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/5495-deprecate-ipvs-mode-in-kube-proxy/README.md) when planning. The recommended replacement is nftables mode.
- AL2023 nodes run kernel 6.x, so they satisfy the nftables mode kernel requirement.
- Even with nftables GA, **the default is still iptables**; you must switch explicitly.

### conntrack: the most frequent source of incidents

For netfilter to do NAT, it must **remember connections.** If you rewrote the address on the way out, you have to undo it on the way back. The kernel table holding that memory is `nf_conntrack`.

In kube-proxy netfilter modes, Service NAT relies on connection tracking. However, **non-NAT traffic may also be tracked**, and headless Services, external endpoints and eBPF implementations have different paths. Do not equate every Kubernetes Service with one mandatory DNAT path.

**What happens on exhaustion is the crux of the problem.** There is no loud error. New connections are **silently dropped**, and the application sees connection timeouts or refusals. From the application side there is no way to know why.

| Observation point | Meaning |
|---|---|
| `/proc/sys/net/netfilter/nf_conntrack_count` | Current entries |
| `/proc/sys/net/netfilter/nf_conntrack_max` | Ceiling |
| `conntrack -S` → `insert_failed` | Insert failures; correlate with count/max, drops and kernel logs rather than treating this as proof of exhaustion alone |
| `conntrack -S` → `drop` | Dropped packets |
| `dmesg` → `nf_conntrack: table full, dropping packet` | Kernel warning |

**One EKS-specific caution.** kube-proxy also manages conntrack values, and **EKS ships a `kube-proxy-config` ConfigMap by default that takes precedence over command-line arguments.** So you can raise the sysctl on the node and have kube-proxy set it back. The correct path is to adjust `conntrack.maxPerCore`/`conntrack.min` in the ConfigMap and restart the kube-proxy DaemonSet.

Raising `nf_conntrack_max` **increases node memory use.** Each entry costs memory, so you cannot raise it without bound; it must match node size. Concrete settings are in [EKS Node Kernel Tuning](https://docs.atomai.click/kubernetes-docs/llms/en/kernel/03-eks-node-tuning.md).

::: warning Verify the active kube-proxy configuration
A historical Bottlerocket report shows node sysctl values being overwritten by kube-proxy. When `--config` is used, edit the **active configuration** (`conntrack.maxPerCore` and `conntrack.min`), not CLI flags that it overrides. Setting both to 0 intentionally delegates the ceiling to node sysctl; confirm behavior against the deployed add-on/version and verify the resulting node value. Preserve the node-memory budget.
:::

The historical issue is not evidence that every current Bottlerocket release has the same behavior. Read the effective configuration and actual sysctl after rollout.

### Reducing conntrack pressure

There are approaches that reduce the load itself.

- Headless Services avoid Service VIP DNAT, but **do not inherently bypass conntrack**.
- Cilium can replace kube-proxy/netfilter functions with eBPF maps; measure its own tracking/map pressure and any remaining netfilter path.
- Connection reuse reduces connection churn; verify both established capacity and timeout behavior.

## overlayfs: How Image Layers Are Composed

Container images being layered, and those layers appearing as one filesystem, is **union mount**, specifically `overlayfs`.

Three parts:

| Layer | Role |
|---|---|
| **lowerdir** | Read-only: image layers (several can stack) |
| **upperdir** | Writable: the container's changes |
| **merged** | The combined view the container sees |

The operationally important property is **copy-up.** Modifying a file from lowerdir copies **the entire file to upperdir first**, then modifies it. So:

- **Modifying a large file slightly still pays the full copy cost.** Changing one byte of a 1GB file copies 1GB
- Heavy writes inside a container consume node disk (ephemeral storage)
- **Write-heavy paths belong on volumes** (emptyDir, PVC, and so on)

## Summary

- There is no "container" in the kernel. It is a **combination** of namespaces (isolation) + cgroups (limits) + capabilities/seccomp/LSM (privileges) + overlayfs (filesystem) + netfilter (network). That is why isolation is selective, and forgotten isolation becomes a silent hole.
- **The net namespace is the Pod boundary.** The shared IP/port space, `localhost` communication, and Pod-scoped netfilter rules all follow from it.
- In cgroup v2, **`memory.current` includes page cache.** OOM diagnosis needs `memory.stat` → `anon`, `memory.events`, and **PSI (`memory.pressure`)** together.
- A CPU limit is a **bandwidth limit**, so throttling spikes latency even at low utilization. `cpu.stat` → `nr_throttled` is the evidence.
- For kube-proxy, **nftables is GA in 1.33 and IPVS is deprecated in 1.35**; upstream plans default disablement of IPVS in 1.40 and removal in 1.43. The default is still iptables.
- Conntrack exhaustion can drop new connections. Diagnose with count/max, drop/insert counters and logs; check the effective kube-proxy configuration before changing limits.

Next: [Kernel Networking Stack](https://docs.atomai.click/kubernetes-docs/llms/en/kernel/02-network-stack.md) walks the full path a packet travels.

## References

- [Control Group v2: Linux kernel documentation](https://docs.kernel.org/admin-guide/cgroup-v2.html)
- [PSI - Pressure Stall Information](https://docs.kernel.org/accounting/psi.html)
- [namespaces(7): Linux manual](https://man7.org/linux/man-pages/man7/namespaces.7.html)
- [KEP-127: Support User Namespaces](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/127-user-namespaces/README.md)
- [bottlerocket-os/bottlerocket#4221: conntrack limit not applied](https://github.com/bottlerocket-os/bottlerocket/issues/4221)
- [NFTables mode for kube-proxy (Kubernetes Blog)](https://kubernetes.io/blog/2025/02/28/nftables-kube-proxy/)
- [KEP-5495: Deprecate IPVS mode in kube-proxy](https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/5495-deprecate-ipvs-mode-in-kube-proxy/README.md)
- [Running kube-proxy in nftables Mode: EKS Best Practices](https://docs.aws.amazon.com/eks/latest/best-practices/nftables.html)
- [Increase nf_conntrack_max limit on EKS nodes](https://repost.aws/knowledge-center/eks-increase-nf-conntrack-max-limit)
- [Amazon EKS-Optimized Amazon Linux 2023 AMIs](https://aws.amazon.com/blogs/containers/amazon-eks-optimized-amazon-linux-2023-amis-now-available/)