# Network Configuration

> **Supported Versions**: EKS 1.36 examples; AWS-maintained Cilium 1.18.3-0 reference, compatible host/kernel required
> **Last Updated**: October 18, 2026

Validate routing, DNS, TLS, credentials and application traffic separately. These examples were checked with local schemas and fixtures, including a mocked Terraform provider; no AWS resources, routers, firewalls or live clusters were changed. The diagrams are repository illustrations based on AWS concepts, not AWS-published validation of this configuration.

![Hybrid prerequisites and bidirectional routing.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-eks-hybrid-nodes-prereq-0.png)

[🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-eks-hybrid-nodes-prereq-0.html)

**Security-team companion:** [Hybrid Nodes network-separation review](https://docs.atomai.click/kubernetes-docs/llms/en/eks-hybrid-nodes/11-network-separation-security.md) explains new control-plane-to-on-premises connections, endpoint types, permission and data boundaries, and review evidence. Private connectivity alone does not establish compliance.

## Network Architecture Overview

Control-plane→hybrid-node traffic and private Kubernetes API traffic use the cluster VPC network path. Kubelet traffic to a public API endpoint uses its configured public route instead; “all traffic always traverses the VPC ENIs” was too broad. Direct Connect public VIFs, private connectivity and public internet paths are distinct choices.

EKS control-plane ENIs/IPs can change. Review actual cluster ownership and approved control-plane subnet ranges instead of treating every `Amazon EKS*` ENI in a shared VPC as belonging to this cluster.

Bind the account and Kubernetes context before read-only diagnostics:

```bash
set -euo pipefail
: "${EXPECTED_ACCOUNT_ID:?Set the intended account}"
: "${AWS_REGION:?Set the cluster Region}"
: "${CLUSTER_NAME:?Set the reviewed cluster name}"
: "${KUBECONFIG:?Set the reviewed kubeconfig}"
export KUBECONFIG KUBE_CONTEXT="${KUBE_CONTEXT:-$CLUSTER_NAME}"
check_account() {
  local account
  account=$(aws sts get-caller-identity --region "$AWS_REGION" --query Account --output text) || return
  test "$account" = "$EXPECTED_ACCOUNT_ID" || { printf 'Account mismatch.\n' >&2; return 1; }
}
check_account
umask 077
export WORK_DIR
WORK_DIR=$(mktemp -d "$PWD/hybrid-network.XXXXXXXX")
aws eks describe-cluster --region "$AWS_REGION" --name "$CLUSTER_NAME" --output json \
  > "$WORK_DIR/cluster.json"
endpoint=$(kubectl --context "$KUBE_CONTEXT" config view --minify \
  -o jsonpath='{.clusters[0].cluster.server}')
jq -e --arg endpoint "$endpoint" '
  .cluster.status=="ACTIVE" and .cluster.endpoint==$endpoint and
  (.cluster.remoteNetworkConfig.remoteNodeNetworks|length)>0
' "$WORK_DIR/cluster.json" >/dev/null
printf 'Private diagnostics: %s\n' "$WORK_DIR"
```

## CIDR Range Requirements

Use non-overlapping IPv4 **RFC1918 or CGNAT** remote node/Pod networks, separate from VPC and Kubernetes Service CIDRs. Up to 15 CIDRs of each remote kind are supported. Current EKS supports remote-network configuration on existing clusters through its API workflow; it is not creation-only.

| Network behavior | Meaning |
|------------------|---------|
| Routable Pod IPs | Approved routes let cloud/control-plane clients initiate connections to Pod IPs |
| Masqueraded egress | SNAT can provide a return path for connections initiated by Pods; it does not automatically permit new inbound connections |
| Unroutable Pod network | Direct cloud→Pod traffic needs another supported path; use cloud-hosted webhooks/API services for the conventional design |

Unroutable does not mean a Pod cannot initiate any AWS API call. For direct hybrid/cloud Pod communication and hybrid-hosted webhooks, supply the actual Pod routes. Gateway/proxy alternatives have separate requirements.

## Required Firewall Ports

| Flow | Protocol / port |
|------|-----------------|
| Node/Pod → Kubernetes API | TCP443 to the actual cluster endpoint |
| Control-plane ENI → kubelet | TCP10250 with kubelet authentication/authorization |
| Control plane → webhook or aggregated API Pod | Its configured TCP port, not a generic “8443+” range |
| DNS clients ↔ actual resolvers | UDP/TCP53 and stateful return traffic |
| Cilium VXLAN between participating nodes | UDP8472 |
| Cilium Geneve, only if selected and supported for the design | UDP6081 |
| Cilium health checks | TCP4240 and the required ICMP/health-endpoint reachability |
| BGP node↔router | TCP179 for the configured active/passive peers |
| VPN gateway transport | UDP500/4500 and applicable IPsec transport requirements |
| Applications / AWS credential and registry services | Only their actual destinations and ports |

The network administrator should change firewall rules while preserving existing rules and connection tracking. Allow only the IP addresses, CIDRs and ports needed for communication. Avoid INPUT rules that allow the entire `10.0.0.0/8` range or DNS/VXLAN rules without source and destination restrictions. Before persisting changes, review which rules to save rather than saving the entire ruleset indiscriminately.

Use TCP `10250` for the kubelet API with authentication and authorization. Do not open TCP `10255`, the legacy read-only kubelet port that allows unauthenticated access.

## AWS Endpoint Access

The **EKS management API PrivateLink endpoint is not the Kubernetes API server endpoint**.

| Service suffix for `com.amazonaws.<region>.*` | Purpose / when needed |
|------------------------------------------------|-----------------------|
| `eks` | AWS EKS management calls such as DescribeCluster |
| `eks-auth` | EKS Pod Identity, when used |
| `ecr.api`, `ecr.dkr` | Private ECR API/registry; image layers also need S3 access |
| `s3` | Private S3 access; on-premises cannot directly use a VPC gateway endpoint |
| `ssm`, applicable SSM messaging services | SSM credential/management functions |
| `rolesanywhere` | IAM Roles Anywhere credentials when that provider is used |
| `sts` | Actual client STS/IRSA/AssumeRole calls; signing an EKS token locally is not itself a client STS network request |
| `logs`, `monitoring`, other selected services | Only if the chosen agents/workloads call them |
| `oidc-eks` | Current EKS OIDC discovery/JWKS PrivateLink service, where available |
| `eks-proxy` | AWS console resource views; not a public application SDK/API |

Confirm service availability in the target Region. Private ECR endpoints do not make **public ECR**, CloudFront or arbitrary package repositories private. For example, the AWS Cilium OCI chart in public ECR needs an approved reachable/mirrored distribution path.

OIDC discovery/JWKS is anonymous public-key material. `oidc-eks` accepts only its default full-access endpoint policy; use SG/routing for reachability and IAM trust `aud`/`sub` conditions for role authorization. STS validates IRSA tokens inside AWS, independently of this VPC endpoint.

For a Roles Anywhere CreateSession endpoint policy, the principal must be `*` because evaluation precedes certificate authentication; restrict the approved trust-anchor resource and supported certificate conditions as documented. Do not copy one generic endpoint policy across these different services. Endpoint policies filter endpoint traffic; they do not replace IAM/role trust or globally disable public service endpoints.

```bash
check_account
vpc_id=$(jq -er '.cluster.resourcesVpcConfig.vpcId' "$WORK_DIR/cluster.json")
aws ec2 describe-vpc-endpoints --region "$AWS_REGION" \
  --filters "Name=vpc-id,Values=$vpc_id" --output json |
  jq '[.VpcEndpoints[]|{id:.VpcEndpointId,service:.ServiceName,type:.VpcEndpointType,
      state:.State,privateDNS:.PrivateDnsEnabled,dnsOptions:.DnsOptions,
      subnets:.SubnetIds,groups:.Groups,dnsEntries:.DnsEntries}]'
```

### S3 Private DNS and Artifact Distribution

S3 **interface endpoints support private DNS**. The inbound-Resolver-only option directs on-premises queries through interface endpoints while in-VPC traffic uses a required S3 gateway endpoint. Keep that gateway while the option is enabled. Alternatively, clear the option to use the interface endpoint for all relevant S3 traffic.

Private DNS is not a TLS rewrite. A PHZ/CNAME mapping `hybrid-assets.eks.amazonaws.com` to an S3 endpoint does not give S3 the CloudFront hostname's certificate or object/Host routing behavior. Do not disable TLS verification to make such a mirror work. Use a supported artifact-preparation/client configuration path, an approved mirror with its own hostname/certificate, or preinstalled dependencies in a validated image.

## VPC Private Endpoints (Restricted-Internet Connectivity) {#vpc-private-endpoints-air-gap-private-connectivity}

Here “air-gap” means restricted internet access with required AWS connectivity, not a disconnected cluster.

The following complete Terraform example uses an existing VPC, endpoint subnets and TGW. It does not create a VPN/DX circuit, TGW attachments, on-premises routes or EKS private DNS configuration. Verify VPC DNS support/hostnames and distinct actual AZs; two different subnet IDs alone do not prove AZ diversity.

Use the original infrastructure owner and import/adopt existing resources before planning replacements. The default endpoint set illustrates SSM; change it for the chosen provider/workloads. Endpoints and Resolver ENIs incur charges. The provider account guard and scoped ingress are deliberate; response traffic uses stateful SG tracking.

```hcl
terraform {
  required_version = ">= 1.9, < 2.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "= 6.69.1"
    }
  }
}

provider "aws" {
  region              = var.region
  allowed_account_ids = [var.expected_account_id]
}

variable "expected_account_id" {
  type = string
  validation {
    condition     = can(regex("^[0-9]{12}$", var.expected_account_id))
    error_message = "Set the reviewed 12-digit account ID."
  }
}

variable "region" {
  type = string
}

variable "name_prefix" {
  type    = string
  default = "hybrid-network"
}

variable "vpc_id" {
  type = string
}

variable "endpoint_subnet_ids" {
  type = set(string)
  validation {
    condition     = length(var.endpoint_subnet_ids) >= 2
    error_message = "Provide subnets in at least two verified Availability Zones."
  }
}

variable "s3_gateway_route_table_ids" {
  type = set(string)
  validation {
    condition     = length(var.s3_gateway_route_table_ids) > 0
    error_message = "Provide the reviewed VPC route tables for the S3 gateway endpoint."
  }
}

variable "client_ipv4_cidrs" {
  type = set(string)
  validation {
    condition = length(var.client_ipv4_cidrs) > 0 && alltrue([
      for c in var.client_ipv4_cidrs : can(cidrnetmask(c)) && c != "0.0.0.0/0"
    ])
    error_message = "Provide scoped IPv4 CIDRs for the actual VPC/on-premises clients."
  }
}

variable "onprem_dns_client_cidrs" {
  type = set(string)
  validation {
    condition = length(var.onprem_dns_client_cidrs) > 0 && alltrue([
      for c in var.onprem_dns_client_cidrs : can(cidrnetmask(c)) && c != "0.0.0.0/0"
    ])
    error_message = "Scope inbound DNS to the actual on-premises resolvers."
  }
}

variable "onprem_dns_servers" {
  type = set(string)
  validation {
    condition = length(var.onprem_dns_servers) > 0 && alltrue([
      for ip in var.onprem_dns_servers : can(cidrnetmask("${ip}/32"))
    ])
    error_message = "Provide actual IPv4 addresses of the on-premises DNS servers."
  }
}

variable "onprem_domain" {
  type    = string
  default = "corp.example.internal"
  validation {
    condition = length(var.onprem_domain) <= 253 && length(split(".", trimsuffix(var.onprem_domain, "."))) >= 2 && alltrue([
      for label in split(".", trimsuffix(var.onprem_domain, ".")) :
      length(label) <= 63 && can(regex("^[A-Za-z0-9](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/eks-hybrid-nodes/[A-Za-z0-9-]*[A-Za-z0-9])?$", label))
    ]) && !can(regex("(^|\\.)(amazonaws\\.com|api\\.aws|cluster\\.local)\\.?$", lower(var.onprem_domain)))
    error_message = "Use a specific owned DNS suffix; do not forward root, AWS or Kubernetes service zones back to on-premises."
  }
}

variable "interface_services" {
  type    = set(string)
  default = ["eks", "ecr.api", "ecr.dkr", "ssm", "ssmmessages"]
  validation {
    condition     = !contains(var.interface_services, "s3")
    error_message = "S3 has its own gateway/interface configuration below."
  }
}

variable "endpoint_policy_json" {
  type    = map(string)
  default = {}
  validation {
    condition = !contains(keys(var.endpoint_policy_json), "oidc-eks") && alltrue([
      for policy in values(var.endpoint_policy_json) : can(jsondecode(policy))
    ])
    error_message = "Use valid service-specific JSON policies; oidc-eks supports only its default full-access policy."
  }
}

variable "controlplane_route_table_ids" {
  type = set(string)
}

variable "remote_ipv4_cidrs" {
  type = set(string)
  validation {
    condition = alltrue([
      for c in var.remote_ipv4_cidrs : can(cidrnetmask(c)) && c != "0.0.0.0/0"
    ])
    error_message = "Use reviewed remote node, Pod and required DNS/service IPv4 CIDRs."
  }
}

variable "existing_transit_gateway_id" {
  type = string
}
```
```hcl
# Import/adopt existing resources through their owner before using this example.
# The existing VPC must have DNS support/hostnames and working hybrid routes.
resource "aws_security_group" "endpoints" {
  name_prefix = "${var.name_prefix}-vpce-"
  description = "HTTPS clients for interface endpoints"
  vpc_id      = var.vpc_id
}

resource "aws_vpc_security_group_ingress_rule" "endpoint_https" {
  for_each          = var.client_ipv4_cidrs
  security_group_id = aws_security_group.endpoints.id
  cidr_ipv4         = each.value
  ip_protocol       = "tcp"
  from_port         = 443
  to_port           = 443
}

resource "aws_vpc_endpoint" "service" {
  for_each            = var.interface_services
  vpc_id              = var.vpc_id
  service_name        = "com.amazonaws.${var.region}.${each.value}"
  vpc_endpoint_type   = "Interface"
  private_dns_enabled = true
  subnet_ids          = var.endpoint_subnet_ids
  security_group_ids  = [aws_security_group.endpoints.id]
  policy              = lookup(var.endpoint_policy_json, each.key, null)
  tags                = { Name = "${var.name_prefix}-${each.key}" }
}

# S3 inbound-Resolver-only private DNS requires this gateway endpoint.
resource "aws_vpc_endpoint" "s3_gateway" {
  vpc_id            = var.vpc_id
  service_name      = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = var.s3_gateway_route_table_ids
  tags             = { Name = "${var.name_prefix}-s3-gateway" }
}

resource "aws_vpc_endpoint" "s3_interface" {
  vpc_id              = var.vpc_id
  service_name        = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type   = "Interface"
  private_dns_enabled = true
  subnet_ids          = var.endpoint_subnet_ids
  security_group_ids  = [aws_security_group.endpoints.id]
  dns_options {
    private_dns_only_for_inbound_resolver_endpoint = true
  }
  depends_on = [aws_vpc_endpoint.s3_gateway]
  tags       = { Name = "${var.name_prefix}-s3-interface" }
}

resource "aws_security_group" "dns_inbound" {
  name_prefix = "${var.name_prefix}-dns-in-"
  description = "DNS from on-premises resolvers"
  vpc_id      = var.vpc_id
}

resource "aws_security_group" "dns_outbound" {
  name_prefix = "${var.name_prefix}-dns-out-"
  description = "DNS to reviewed on-premises resolvers"
  vpc_id      = var.vpc_id
}

locals {
  inbound_dns_rules = {
    for pair in setproduct(var.onprem_dns_client_cidrs, toset(["tcp", "udp"])) :
    "${pair[0]}-${pair[1]}" => { cidr = pair[0], protocol = pair[1] }
  }
  outbound_dns_rules = {
    for pair in setproduct(var.onprem_dns_servers, toset(["tcp", "udp"])) :
    "${pair[0]}-${pair[1]}" => { ip = pair[0], protocol = pair[1] }
  }
}

resource "aws_vpc_security_group_ingress_rule" "dns" {
  for_each          = local.inbound_dns_rules
  security_group_id = aws_security_group.dns_inbound.id
  cidr_ipv4         = each.value.cidr
  ip_protocol       = each.value.protocol
  from_port         = 53
  to_port           = 53
}

resource "aws_vpc_security_group_egress_rule" "dns" {
  for_each          = local.outbound_dns_rules
  security_group_id = aws_security_group.dns_outbound.id
  cidr_ipv4         = "${each.value.ip}/32"
  ip_protocol       = each.value.protocol
  from_port         = 53
  to_port           = 53
}

resource "aws_route53_resolver_endpoint" "inbound" {
  name                   = "${var.name_prefix}-inbound"
  direction              = "INBOUND"
  resolver_endpoint_type = "IPV4"
  security_group_ids     = [aws_security_group.dns_inbound.id]
  dynamic "ip_address" {
    for_each = var.endpoint_subnet_ids
    content {
      subnet_id = ip_address.value
    }
  }
}

resource "aws_route53_resolver_endpoint" "outbound" {
  name                   = "${var.name_prefix}-outbound"
  direction              = "OUTBOUND"
  resolver_endpoint_type = "IPV4"
  security_group_ids     = [aws_security_group.dns_outbound.id]
  dynamic "ip_address" {
    for_each = var.endpoint_subnet_ids
    content {
      subnet_id = ip_address.value
    }
  }
}

resource "aws_route53_resolver_rule" "onprem" {
  domain_name          = var.onprem_domain
  name                 = "${var.name_prefix}-onprem"
  rule_type            = "FORWARD"
  resolver_endpoint_id = aws_route53_resolver_endpoint.outbound.id
  dynamic "target_ip" {
    for_each = var.onprem_dns_servers
    content {
      ip   = target_ip.value
      port = 53
    }
  }
}

resource "aws_route53_resolver_rule_association" "onprem" {
  resolver_rule_id = aws_route53_resolver_rule.onprem.id
  vpc_id           = var.vpc_id
}

output "inbound_resolver_ips" {
  value = [for address in aws_route53_resolver_endpoint.inbound.ip_address : address.ip]
}

# VPC return routes only. Existing TGW attachment routes/propagation and
# on-premises routing must be managed separately by their infrastructure owner.
locals {
  remote_routes = {
    for pair in setproduct(var.controlplane_route_table_ids, var.remote_ipv4_cidrs) :
    "${pair[0]}-${pair[1]}" => { table = pair[0], cidr = pair[1] }
  }
}

resource "aws_route" "hybrid" {
  for_each               = local.remote_routes
  route_table_id         = each.value.table
  destination_cidr_block = each.value.cidr
  transit_gateway_id     = var.existing_transit_gateway_id
  # A VGW topology uses gateway_id instead; do not set both target fields.
}
```

The S3 gateway dependency is explicit. Include required on-premises DNS/service ranges in `remote_ipv4_cidrs`, not only node/Pod networks: the example DNS servers `192.168.1.10/11` need an approved `192.168.1.0/24` route or corresponding host routes. For a VGW return route, use `gateway_id` instead of `transit_gateway_id`; do not put a TGW ID into a VGW/gateway field or configure both targets. VPC routes alone do not establish TGW/VPN/on-premises routing.

## DNS Configuration

On-premises resolvers can conditionally forward selected AWS/service and actual cluster-endpoint names to Route 53 Resolver inbound IPs. Use the addresses returned for the real endpoint. A broad `amazonaws.com` forward can affect unrelated services; choose zones deliberately and avoid forwarding loops.

```text
// Example service zones only. Replace these Resolver IPs with actual outputs.
zone "eks.ap-northeast-2.amazonaws.com" {
    type forward;
    forward only;
    forwarders { 10.0.1.10; 10.0.2.10; };
};
zone "s3.ap-northeast-2.amazonaws.com" {
    type forward;
    forward only;
    forwarders { 10.0.1.10; 10.0.2.10; };
};
```

This BIND fragment covers the shown service zones, not the Kubernetes API hostname automatically. Add the actual cluster endpoint's DNS name/suffix and other required names from the verified service inventory. Resolver outbound rules handle the on-premises zone; that zone must be authoritative/reachable on the selected DNS servers, not forwarded back into the same loop.

### CoreDNS Custom Domain Configuration

If the chosen design forwards directly from CoreDNS, merge a reviewed server block into the existing Corefile; do not overwrite the whole managed ConfigMap:

```text
# Fragment to merge through the CoreDNS configuration owner.
corp.example.internal:53 {
    errors
    cache 30
    forward . 192.168.1.10 192.168.1.11 {
        max_concurrent 1000
    }
}
```

Preserve existing Kubernetes zones, health/readiness and reload behavior. Check the actual resolver file and systemd-resolved/stub layout; forwarding a DNS server back to itself can loop. Choose either the appropriate VPC forwarding path or explicit CoreDNS forwarding for a zone rather than layering contradictory routes.

For the managed EKS add-on, fetch the **installed version's** configuration schema and validate the complete proposed values, preserving unrelated existing settings:

```bash
check_account
aws eks describe-addon --region "$AWS_REGION" --cluster-name "$CLUSTER_NAME" \
  --addon-name coredns --output json > "$WORK_DIR/coredns-addon.json"
addon_version=$(jq -er '.addon.addonVersion' "$WORK_DIR/coredns-addon.json")
aws eks describe-addon-configuration --region "$AWS_REGION" --addon-name coredns \
  --addon-version "$addon_version" --output json > "$WORK_DIR/coredns-schema-response.json"
jq -r '.configurationSchema' "$WORK_DIR/coredns-schema-response.json" \
  > "$WORK_DIR/coredns-schema.json"
# Prepare the full intended values, preserving unrelated existing configuration.
: "${COREDNS_CANDIDATE_JSON:?Set the reviewed full configurationValues JSON file}"
export COREDNS_CANDIDATE_JSON
python3 - <<'PY'
import json, os
from pathlib import Path
import jsonschema
folder = Path(os.environ["WORK_DIR"])
schema = json.loads((folder / "coredns-schema.json").read_text())
candidate = json.loads(Path(os.environ["COREDNS_CANDIDATE_JSON"]).read_text())
validator = jsonschema.validators.validator_for(schema)
validator.check_schema(schema)
validator(schema).validate(candidate)
print("Configuration matches the fetched schema; rollout and DNS behavior are not yet verified")
PY
```

This is a local schema check, not a rollout. Coordinate managed add-on/GitOps ownership, autoscaling and recovery before applying configuration.

### CoreDNS Placement and Locality

AWS recommends at least one CoreDNS replica on cloud nodes and one on hybrid nodes in mixed clusters. Two per location can be a resilience choice; four replicas is not a universal minimum or a guarantee.

Verify real zone labels on all intended DNS nodes. Hybrid nodes need an owner-defined `topology.kubernetes.io/zone` value such as `onprem-dc1`; a compute-type label is not automatically that zone or a taint. Merge placement preferences without deleting unrelated affinity/tolerations:

```json
{
  "affinity": {
    "podAntiAffinity": {
      "preferredDuringSchedulingIgnoredDuringExecution": [
        {
          "weight": 100,
          "podAffinityTerm": {
            "labelSelector": {
              "matchLabels": {
                "k8s-app": "kube-dns"
              }
            },
            "topologyKey": "kubernetes.io/hostname"
          }
        },
        {
          "weight": 50,
          "podAffinityTerm": {
            "labelSelector": {
              "matchLabels": {
                "k8s-app": "kube-dns"
              }
            },
            "topologyKey": "topology.kubernetes.io/zone"
          }
        }
      ]
    }
  }
}
```

Soft affinity/spread is a preference, not a guaranteed 2+2 distribution or successful bootstrap. Placement alone also does not ensure clients choose a local DNS replica.

AWS's documented Service Traffic Distribution example uses `PreferClose`. With Cilium, configure the supported `loadBalancer.serviceTopology` setting and roll the affected agents through the owner before relying on it. Review the actual dataplane/version and healthy local endpoints.

```json
{
  "spec": {
    "trafficDistribution": "PreferClose"
  }
}
```
```bash
kubectl --context "$KUBE_CONTEXT" --request-timeout=15s -n kube-system \
  get service kube-dns -o json |
  jq '{name:.metadata.name,uid:.metadata.uid,clusterIP:.spec.clusterIP,
       clusterIPs:.spec.clusterIPs,ports:.spec.ports,trafficDistribution:.spec.trafficDistribution}'
kubectl --context "$KUBE_CONTEXT" --request-timeout=15s -n kube-system \
  get endpointslices -l kubernetes.io/service-name=kube-dns -o json |
  jq '[.items[]|{name:.metadata.name,addressType,ports,
       endpoints:[.endpoints[]?|{addresses,nodeName,zone,conditions,hints}]}]'
kubectl --context "$KUBE_CONTEXT" --request-timeout=15s -n kube-system \
  get pods -l k8s-app=kube-dns -o json |
  jq '[.items[]|{name:.metadata.name,node:.spec.nodeName,phase:.status.phase,
       ready:([.status.conditions[]?|select(.type=="Ready")|.status]|first // "NotReported")}]'
```

Inspect actual Service IPs, EndpointSlice zones/hints and Pod readiness. `10.100.0.10` is an example for a particular Service CIDR, not a universal cluster DNS address. Local replicas do not turn a disconnected EKS cluster into a fully independent DNS/control plane.

## Traffic Flow Patterns

The drawings use illustrative addresses and simplified processing stages. Check whether your actual Service dataplane is kube-proxy iptables, nftables/IPVS or Cilium's eBPF replacement.

### Pattern 1: Kubelet → EKS Control Plane

The kubelet resolves and connects to the configured Kubernetes API endpoint. Private and public access have different routes; neither should be confused with the EKS management PrivateLink endpoint.

![Kubelet API access paths for public and private endpoint configurations.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-eks-hybrid-nodes-02-network-configuration-10.png)

[🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-eks-hybrid-nodes-02-network-configuration-10.html)

### Pattern 2: EKS Control Plane → Kubelet

The control plane reaches the reported, routable node address over TCP10250. This path supports logs, exec and port-forward and needs reverse routing, firewall permission and kubelet authentication.

![Control-plane connection to the routable kubelet address over TCP10250.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-eks-hybrid-nodes-02-network-configuration-11.png)

[🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-eks-hybrid-nodes-02-network-configuration-11.html)

### Pattern 3: Pod → EKS Control Plane

A Pod using the Kubernetes Service IP needs Service translation to the selected API endpoint. If egress SNAT applies, responses target the node address and connection tracking reverses translation; without SNAT, the Pod address needs a return route.

![Logical Service translation and optional SNAT effects; the pictured order is not a universal hook sequence.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-eks-hybrid-nodes-02-network-configuration-12.png)

[🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-eks-hybrid-nodes-02-network-configuration-12.html)

The drawing's SNAT-before-DNAT numbering is not a universal hook order. In an iptables path, Service DNAT normally precedes routing and applicable POSTROUTING SNAT. eBPF paths differ. Capture the actual packet/connection state before drawing conclusions.

### Pattern 4: EKS Control Plane → Pod (Webhooks)

The API server needs the selected webhook Pod IP/port to be reachable. Use the configured port; the old “8443+” legend is not a valid port-range requirement.

![Control-plane path to a webhook Pod; use its actual configured TCP port and dataplane.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-eks-hybrid-nodes-02-network-configuration-13.png)

[🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-eks-hybrid-nodes-02-network-configuration-13.html)

### Pattern 5: Pod ↔ Pod on Hybrid Nodes

With the supported VXLAN overlay, the destination node is reached using **outer node IPs**. The underlay does not need a route for the inner destination Pod CIDR merely to carry the encapsulated packet.

![VXLAN Pod communication. Outer forwarding uses node IPs; the older Pod-CIDR forwarding labels need correction.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-eks-hybrid-nodes-02-network-configuration-14.png)

[🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-eks-hybrid-nodes-02-network-configuration-14.html)

The diagram's forwarding labels that use `10.85.x.0/24` after encapsulation need to be read as an older simplification: the outer packet routes to `10.80.0.x`. Nodes on one L2 segment may communicate directly without a router hop.

VXLAN encapsulates an inner Ethernet frame using UDP. In the IPv4/no-extra-encapsulation example, 50 bytes of overhead explains a 1500→1450 MTU adjustment; additional tunneling changes that calculation. Cilium VXLAN uses UDP8472; standard VXLAN commonly uses 4789. Geneve uses UDP6081. Current Cilium tunnel configuration distinguishes VXLAN/Geneve; IP-in-IP is not an interchangeable default overlay selected by the old `--tunnel` advice.

The VNI field is 24 bits; Cilium can carry security identity in encapsulation metadata. It is not cryptographic tenant isolation. Keep network policies and actual identity propagation separate.

### Pattern 6: Cloud Pod ↔ Hybrid Pod

Direct Pod-IP traffic needs the relevant Pod routes across VPC, WAN and on-premises. Service translation is needed only when the request actually targets a Service VIP.

![Direct cloud-to-hybrid Pod routing; kube-proxy Service translation is not required for a direct Pod-IP destination.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-eks-hybrid-nodes-02-network-configuration-15.png)

[🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-eks-hybrid-nodes-02-network-configuration-15.html)

The picture's kube-proxy/iptables block is dataplane-dependent; a direct Pod-IP packet does not inherently require kube-proxy DNAT.

### kube-proxy and kubelet Details

In kube-proxy **iptables mode**, a common chain path is:

```text
KUBE-SERVICES → KUBE-SVC-* → KUBE-SEP-* → endpoint DNAT
```

With three eligible equal-weight endpoints and no overriding affinity/locality policy, conditional probabilities 1/3, then 1/2 of the remaining packets, then the remainder produce roughly equal selection. This is an illustration, not captured output or the rule structure of every dataplane:

```text
# KUBE-SERVICES chain (nat table)
-A KUBE-SERVICES -d 172.20.0.10/32 -p tcp -m tcp --dport 80 -j KUBE-SVC-XXXXXX

# KUBE-SVC chain (load balancing)
-A KUBE-SVC-XXXXXX -m statistic --mode random --probability 0.33333 -j KUBE-SEP-AAAAAA
-A KUBE-SVC-XXXXXX -m statistic --mode random --probability 0.50000 -j KUBE-SEP-BBBBBB
-A KUBE-SVC-XXXXXX -j KUBE-SEP-CCCCCC

# KUBE-SEP chain (DNAT)
-A KUBE-SEP-AAAAAA -p tcp -j DNAT --to-destination 10.85.0.15:8080
-A KUBE-SEP-BBBBBB -p tcp -j DNAT --to-destination 10.85.0.16:8080
-A KUBE-SEP-CCCCCC -p tcp -j DNAT --to-destination 10.85.1.20:8080
```

| Secure kubelet endpoint | Purpose |
|------------------------|---------|
| `/pods` | Pod information |
| `/exec/{namespace}/{pod}/{container}` | Container exec stream |
| `/containerLogs/{namespace}/{pod}/{container}` | Container logs; not the former `/logs/...` path |
| `/metrics`, `/healthz` | Authorized metrics/health endpoints |

Use supported API-server-mediated diagnostics and appropriate authorization. The node's real `status.addresses` matters; do not substitute a hostname or the first address from an unrelated object.

## Choosing a CNI Design: Cilium Only or Mixed with VPC CNI

The VPC CNI is not compatible with hybrid nodes, so **hybrid nodes always run Cilium (or Calico).** The question "Cilium only or mixed with the VPC CNI?" is therefore really about **whether the cluster has cloud nodes, and if so, which CNI they run.**

| Criterion | ① Hybrid-only cluster (Cilium only) | ② Mixed mode (VPC CNI in the cloud + Cilium on hybrid) | ③ Cilium on cloud nodes too |
|---|---|---|---|
| AWS support | Supported (Cilium on hybrid nodes) | Supported. The AWS-recommended mixed mode design | Cloud side unsupported: Cilium running on nodes in AWS Cloud is not supported by AWS |
| Webhook placement | Runs on hybrid nodes. The control plane connects to pod IPs, so the on-premises pod CIDR must be routable with BGP or static routes | Run on cloud nodes. The on-premises pod CIDR does not need to be routable | Depends on the IPAM and routing of Cilium in the cloud; must be validated yourself |
| Cloud to on-premises pod traffic | Not applicable | Routable on-premises pod CIDR or the [Hybrid Nodes gateway](https://docs.atomai.click/kubernetes-docs/llms/en/eks-hybrid-nodes/10-hybrid-nodes-gateway.md) | Design tunnels or routing yourself. The Hybrid Nodes gateway requires the VPC CNI (or Auto Mode built-in networking) on cloud nodes, including the gateway nodes, so it cannot be used |
| Bursting and Auto Mode | Not possible without cloud capacity | Karpenter can add cloud nodes. Auto Mode uses built-in networking | Auto Mode cannot replace its CNI. Cilium on Karpenter nodes is also self-operated |
| Policy and observability | One CNI (NetworkPolicy, Cilium policies, Hubble) | Each CNI enforces standard Kubernetes NetworkPolicy. Cilium-specific policies and Hubble exist only on the hybrid side | One CNI, but incident response and upgrades for the cloud side are on you or a vendor |

How to choose:

- **Cloud nodes present: ②.** The mixed design within AWS support. Run webhooks in the cloud and use bursting and the Hybrid Nodes gateway. In mixed mode, run at least one CoreDNS replica on each side; features where the control plane or VPC must reach hybrid pod IPs (ALB/NLB IP targets, Metrics Server, AMP collection) need a routable on-premises pod CIDR ([Mixed-Mode Webhook Operations](https://docs.atomai.click/kubernetes-docs/llms/en/eks-hybrid-nodes/08-operations.md#mixed-mode-webhook-operations)).
- **On-premises only: ①.** Simpler with one CNI, but webhooks run on hybrid nodes, so a [routable pod CIDR](#routable-pod-cidr-configuration) is required.
- **Consider ③ carefully.** Cilium on cloud nodes is outside AWS support, and the Hybrid Nodes gateway and Auto Mode cannot be used. Consider it only with a vendor support agreement and when a single CNI outweighs these limits.

References:
- [Configure CNI for hybrid nodes](https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-cni.html)
- [Configure webhooks for hybrid nodes: considerations for mixed mode clusters](https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-webhooks.html)
- [Get started with EKS Hybrid Nodes gateway](https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-gateway-getting-started.html)

## Routable Pod CIDR Configuration

![Illustrative remote Pod CIDRs and the on-premises router.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-eks-hybrid-nodes-02-network-configuration-0.png)

[🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-eks-hybrid-nodes-02-network-configuration-0.html)

### Option 1: BGP (Recommended)

![Illustrative BGP Pod-prefix advertisements.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-eks-hybrid-nodes-02-network-configuration-1.png)

[🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-eks-hybrid-nodes-02-network-configuration-1.html)

AWS's dedicated CNI support page lists AWS-maintained Cilium 1.17/1.18 builds. The reference here is 1.18.3-0 on a compatible kernel/OS; do not blindly replace it with upstream 1.19. Other AWS pages still mention Calico BGP and its retained examples. That is not evidence that the Calico project is deprecated; confirm support scope for an existing deployment.

Enable BGP through the owner of the **existing pinned Cilium release**, merging values and reviewing operator/agent rollout:

```yaml
bgpControlPlane:
  enabled: true
operator:
  rollOutPods: true
```

The AWS-style `v2alpha1` APIs remain served in the reviewed 1.18.3 CRDs (which also serve `v2`). There is no need to replace CRDs simply to use this example. On Cilium 1.19+ use the `cilium.io/v2` BGP CRDs instead (the `v2alpha1` BGP CRDs are deprecated), and the legacy `CiliumBGPPeeringPolicy` was removed in 1.19.

The following selects hybrid nodes, links the peer's advertisement selector to the advertisement labels, and advertises only their Pod CIDRs:

```yaml
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPClusterConfig
metadata:
  name: hybrid-bgp-config
spec:
  nodeSelector:
    matchLabels:
      eks.amazonaws.com/compute-type: hybrid
  bgpInstances:
  - name: hybrid-instance
    localASN: 65001
    peers:
    - name: on-prem-router
      peerASN: 65000
      peerAddress: 10.80.1.1
      peerConfigRef:
        name: on-prem-peer
```
```yaml
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeerConfig
metadata:
  name: on-prem-peer
spec:
  timers:
    holdTimeSeconds: 90
    keepAliveTimeSeconds: 30
  gracefulRestart:
    enabled: true
    restartTimeSeconds: 120
  families:
  - afi: ipv4
    safi: unicast
    advertisements:
      matchLabels:
        advertise: hybrid-pods
```
```yaml
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPAdvertisement
metadata:
  name: hybrid-pod-cidrs
  labels:
    advertise: hybrid-pods
spec:
  advertisements:
  - advertisementType: PodCIDR
```

The example assumes the selected nodes can reach router `10.80.1.1` with the intended peering topology. Different racks/loopbacks may need distinct non-overlapping selectors and reviewed multihop settings. TCP179, ASN, authentication, route filters, negotiated timers and graceful-restart stale-route behavior need network-owner review.

BGP session establishment alone does not prove the intended prefixes were advertised, accepted or installed in the router's forwarding table. The Cilium BGP control plane advertises reachability; it does not replace all kernel/underlay routing.

```bash
cilium --context "$KUBE_CONTEXT" bgp peers
cilium --context "$KUBE_CONTEXT" bgp routes
kubectl --context "$KUBE_CONTEXT" --request-timeout=15s \
  get ciliumbgpclusterconfigs,ciliumbgppeerconfigs,ciliumbgpadvertisements -o json |
  jq '[.items[]|{kind,name:.metadata.name,status:.status}]'
```

#### ASN and Router Configuration

RFC6996 private ranges are **64512–65534** and **4200000000–4294967294**. The former blanket “only the 16-bit range” rule was incorrect. Do not describe every value 1–64511 as a freely usable public ASN; public/reserved/documentation assignments have their own rules.

Use existing coordinated network ASNs. `localASN=65001` represents Cilium nodes and `peerASN=65000` their on-premises router in these examples. A TGW's ASN is a different upstream relationship: Cilium does not automatically peer with a TGW merely because it exists. Site-to-Site VPN BGP terminates on the configured VPN/customer gateway path; TGW Connect is another separate transport/design.

The following vendor fragments are illustrative starting points, not device-tested configurations. Merge through the router owner with platform/version-specific import/export prefix filters, limits and recovery. Do not change a live router's global ASN indiscriminately.

**Cisco IOS / IOS-XE**

```text
router bgp 65000
 neighbor 10.80.1.10 remote-as 65001
 neighbor 10.80.1.10 description "EKS Hybrid Node - Cilium BGP"
 !
 address-family ipv4 unicast
  neighbor 10.80.1.10 activate
  neighbor 10.80.1.10 soft-reconfiguration inbound
 exit-address-family
```

**Cisco NX-OS (Nexus)**

```text
router bgp 65000
  address-family ipv4 unicast
  neighbor 10.80.1.10
    remote-as 65001
    description EKS-Hybrid-Cilium
    address-family ipv4 unicast
      soft-reconfiguration inbound
```

**Juniper Junos (MX / QFX / SRX)**

```text
set protocols bgp group eks-hybrid type external
set protocols bgp group eks-hybrid peer-as 65001
set protocols bgp group eks-hybrid neighbor 10.80.1.10 description "EKS Hybrid Node"
set protocols bgp group eks-hybrid family inet unicast
set routing-options autonomous-system 65000
```

**Arista EOS**

```text
router bgp 65000
   neighbor 10.80.1.10 remote-as 65001
   neighbor 10.80.1.10 description EKS-Hybrid-Cilium
   !
   address-family ipv4
      neighbor 10.80.1.10 activate
```

**MikroTik RouterOS 7.20+**

```text
/routing/bgp/instance
add name=hybrid as=65000
/routing/bgp/connection
add name=hybrid-node-001 instance=hybrid remote.address=10.80.1.10 remote.as=65001 local.role=ebgp address-families=ip disabled=yes
# Review input/output filters and routing before enabling the connection.
```

**FRRouting (FRR), reference 10.7.1**

```text
ip prefix-list HYBRID_PODS seq 10 permit 10.85.0.0/16 ge 25 le 25
route-map FROM_HYBRID permit 10
 match ip address prefix-list HYBRID_PODS
route-map TO_HYBRID deny 10
router bgp 65000
 bgp router-id 10.80.1.1
 bgp ebgp-requires-policy
 neighbor 10.80.1.10 remote-as 65001
 address-family ipv4 unicast
  neighbor 10.80.1.10 activate
  neighbor 10.80.1.10 route-map FROM_HYBRID in
  neighbor 10.80.1.10 route-map TO_HYBRID out
 exit-address-family
```

RouterOS 7.20+ explicitly defines the BGP instance. FRR's traditional defaults require eBGP policy; an established session without filters can show `(Policy)` and exchange no routes. The FRR example accepts only the reviewed `/25` Pod blocks and exports no routes to Cilium; adapt filters to the actual IPAM design and upstream routing.

### Option 2: Static Routes

![Illustrative static Pod-prefix routes; derive current next hops from observed state.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-eks-hybrid-nodes-02-network-configuration-2.png)

[🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-eks-hybrid-nodes-02-network-configuration-2.html)

In Cilium **cluster-pool IPAM**, read all allocated `CiliumNode.spec.ipam.podCIDRs`. Allocation is not guaranteed to follow node registration order. A `/16` contains 512 `/25` blocks geometrically, and each `/25` has 128 addresses; this does not guarantee 512 supported nodes or 128 usable application Pod IPs per node. Reserved addresses, node/CNI use and kubelet/resource limits also matter.

This is a Cilium Helm-values fragment for a reviewed new pool, not a kubelet-wide `podCIDR` setting. Do not change existing allocated CIDRs or block size as a shortcut to migration:

```yaml
ipam:
  mode: cluster-pool
  operator:
    clusterPoolIPv4PodCIDRList:
    - 10.85.0.0/16
    clusterPoolIPv4MaskSize: 25
```

Do not use `.addresses[0]` as the next hop: it can be a Cilium-internal address. The following cross-checks IPv4 InternalIP with the Kubernetes Node, validates all Pod prefixes against approved remote ranges, rejects overlap/missing state and produces **JSON candidates**, not executable shell:

```bash
kubectl --context "$KUBE_CONTEXT" --request-timeout=15s get nodes \
  -l eks.amazonaws.com/compute-type=hybrid -o json |
  jq '{items:[.items[]|{metadata:{name:.metadata.name,uid:.metadata.uid},
       status:{addresses:.status.addresses}}]}' > "$WORK_DIR/hybrid-nodes.json"
kubectl --context "$KUBE_CONTEXT" --request-timeout=15s get ciliumnodes.cilium.io -o json |
  jq '{items:[.items[]|{metadata:{name:.metadata.name,uid:.metadata.uid},
       spec:{addresses:.spec.addresses,ipam:{podCIDRs:.spec.ipam.podCIDRs}}}]}' \
  > "$WORK_DIR/cilium-nodes.json"
```
```bash
python3 - <<'PY'
import ipaddress, json, os
from pathlib import Path
folder = Path(os.environ["WORK_DIR"])
network = json.loads((folder / "cluster.json").read_text())["cluster"]["remoteNetworkConfig"]
node_ranges = [ipaddress.ip_network(c, strict=True) for n in network["remoteNodeNetworks"] for c in n["cidrs"]]
pod_ranges = [ipaddress.ip_network(c, strict=True) for n in network.get("remotePodNetworks", []) for c in n["cidrs"]]
if not pod_ranges:
    raise SystemExit("A reviewed routable remote Pod range is required for this route plan")
nodes = {n["metadata"]["name"]: n for n in json.loads((folder / "hybrid-nodes.json").read_text())["items"]}
claims = json.loads((folder / "cilium-nodes.json").read_text())["items"]
rows, seen = [], []
for item in claims:
    name = item["metadata"]["name"]
    if name not in nodes:
        continue
    node = nodes[name]
    ips = [ipaddress.ip_address(a["address"]) for a in node["status"]["addresses"]
           if a["type"] == "InternalIP" and ":" not in a["address"]]
    if len(ips) != 1 or not any(ips[0] in n for n in node_ranges):
        raise SystemExit(f"Review the unique IPv4 InternalIP and remote-node range for {name}")
    cilium_ips = [ipaddress.ip_address(a["ip"]) for a in item["spec"].get("addresses", [])
                  if a["type"] == "InternalIP" and ":" not in a["ip"]]
    if cilium_ips != ips:
        raise SystemExit(f"Kubernetes/Cilium InternalIP mismatch for {name}")
    cidrs = item["spec"]["ipam"].get("podCIDRs") or []
    if not cidrs:
        raise SystemExit(f"No allocated cluster-pool Pod CIDRs for {name}; do not invent a route")
    for raw in cidrs:
        cidr = ipaddress.ip_network(raw, strict=True)
        if cidr.version != 4 or not any(cidr.subnet_of(p) for p in pod_ranges):
            raise SystemExit(f"Unapproved Pod CIDR for {name}: {cidr}")
        if any(cidr.overlaps(previous) for previous in seen):
            raise SystemExit("Overlapping or duplicate Pod routes require investigation")
        seen.append(cidr)
        rows.append({"node": name, "nodeUID": node["metadata"]["uid"],
                     "ciliumNodeUID": item["metadata"]["uid"], "destination": str(cidr), "nextHop": str(ips[0])})
if set(nodes) != {row["node"] for row in rows}:
    raise SystemExit("Some hybrid nodes have no matching Cilium allocation")
(folder / "reviewed-route-candidates.json").write_text(json.dumps(rows, indent=2) + "\n")
print(json.dumps(rows, indent=2))
PY
```

This is a point-in-time plan for cluster-pool IPAM, not an atomic lease on node identity or a routing controller. Revalidate before changes. Calico BlockAffinity is a different model; inspect its state, borrowing/pool behavior and actual routes rather than reusing this generator blindly.

After owner review, manual router syntax can look like:

```text
# Illustrative syntax after validating the route plan on the intended router:
# Linux
ip route add 10.85.0.0/25 via 10.80.1.10
# Cisco IOS / IOS-XE
ip route 10.85.0.0 255.255.255.128 10.80.1.10 name hybrid-node-001-pods
# FRR
ip route 10.85.0.0/25 10.80.1.10
```

Persist changes through the actual network manager/device configuration. An `up ip route ...` line belongs to an ifupdown stanza, not a standalone Bash script and not every modern Linux network manager. Static routes need drift/failure tracking; the original “1–5 nodes” threshold was a planning heuristic, not a technical limit.

### Option 3: ARP Proxying

AWS describes proxy ARP as a possible L2 approach. It needs the appropriate on-link neighbor-discovery behavior and a specifically configured CNI/host implementation; merely enabling generic Cilium does not prove this path is ready.

![Proxy ARP concept for a verified L2/on-link design; upstream routes are still required.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-eks-hybrid-nodes-02-network-configuration-3.png)

[🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-eks-hybrid-nodes-02-network-configuration-3.html)

ARP broadcasts do not traverse TGW/VPN/DX Layer-3 routing. This option does not remove VPC/WAN return-route requirements. Validate actual L2 behavior and failover before replacing a BGP/static design.

## Network Policies

Policies depend on selection, direction and the enforcing dataplane. Kubernetes NetworkPolicy allows are additive: another matching policy can permit traffic. Cilium explicit deny and L7 behavior need their own evaluation. These examples are alternatives to test in a controlled namespace, not an instruction to layer every allow rule and expect stricter behavior.

### Kubernetes NetworkPolicy

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: bookinfo
spec:
  podSelector:
    matchLabels:
      app: reviews
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: productpage
    ports:
    - protocol: TCP
      port: 9080
```

This selects `reviews` ingress and permits matching `productpage` Pods in the same namespace on TCP9080. It does not isolate every Pod/direction or override every other policy.

### CiliumNetworkPolicy and L7

```yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: bookinfo
spec:
  endpointSelector:
    matchLabels:
      app: reviews
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: productpage
        k8s:io.kubernetes.pod.namespace: bookinfo
    toPorts:
    - ports:
      - port: '9080'
        protocol: TCP
```
```yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: frontend-http-contract
  namespace: bookinfo
spec:
  endpointSelector:
    matchLabels:
      app: reviews
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: productpage
        k8s:io.kubernetes.pod.namespace: bookinfo
    toPorts:
    - ports:
      - port: '9080'
        protocol: TCP
      rules:
        http:
        - method: GET
          path: /api/v1/.*
```

The HTTP rule is an alternative to the unrestricted L4 allow, not an automatic restriction layered on it. Use the actual application path. HTTP inspection requires appropriate visibility; encrypted mesh/TLS traffic is not automatically inspectable.

### DNS-Based Egress

The separate `external-api-client` example permits DNS through **Cilium-identified CoreDNS Pods** and HTTPS to the observed API addresses:

```yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-external-api
  namespace: bookinfo
spec:
  endpointSelector:
    matchLabels:
      app: external-api-client
  egress:
  - toEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: kube-system
        k8s:k8s-app: kube-dns
    toPorts:
    - ports:
      - port: '53'
        protocol: ANY
      rules:
        dns:
        - matchPattern: '*'
  - toFQDNs:
    - matchName: api.example.com
    toPorts:
    - ports:
      - port: '443'
        protocol: TCP
```

Verify the real resolver/identity layout. NodeLocal DNS, host DNS or non-Cilium-managed endpoints may need a different supported rule. FQDN IP observation is not remote API authentication; account for DNS caching and shared addresses. Cilium-specific L7/FQDN features also extend beyond the default Kubernetes NetworkPolicy support scope listed by AWS.

## Webhook Configuration

The conventional direct-routing design requires the control plane to reach webhook Pod IPs. If no suitable Pod return path exists, use appropriately placed cloud-hosted components. Verify alternative gateway/proxy designs separately.

```yaml
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: eks.amazonaws.com/compute-type
          operator: NotIn
          values:
          - hybrid
```

This is a Pod-template affinity fragment, not a complete Deployment. A `NotIn hybrid` condition is not proof that matching healthy cloud capacity or every other scheduling constraint exists.

AWS Load Balancer Controller, CloudWatch/ADOT operators and cert-manager have webhook placement requirements. Distinguish their operators from node collectors. **Metrics Server is an aggregated API service, not an admission webhook**, but still needs control-plane-to-Pod reachability. Test real API calls and webhooks, not only Pod phase.

## Read-Only Connectivity Diagnostics

### Kubernetes API TLS and Timing

```bash
set -euo pipefail
endpoint=$(jq -er '.cluster.endpoint' "$WORK_DIR/cluster.json")
case "$endpoint" in https://*) ;; *) printf 'HTTPS endpoint required.\n' >&2; exit 1;; esac
jq -er '.cluster.certificateAuthority.data' "$WORK_DIR/cluster.json" |
  base64 --decode > "$WORK_DIR/cluster-ca.pem"
openssl x509 -in "$WORK_DIR/cluster-ca.pem" -noout >/dev/null
curl --silent --show-error --connect-timeout 5 --max-time 15 \
  --cacert "$WORK_DIR/cluster-ca.pem" --output "$WORK_DIR/api-response.txt" \
  --write-out '{"httpCode":%{http_code},"remoteIP":"%{remote_ip}","dnsTotalSeconds":%{time_namelookup},"connectTotalSeconds":%{time_connect},"tlsTotalSeconds":%{time_appconnect},"totalSeconds":%{time_total}}\n' \
  "$endpoint/readyz" > "$WORK_DIR/api-timing.json"
cat "$WORK_DIR/api-timing.json"
```

The cluster CA and hostname must validate. An HTTP401/403 response can demonstrate a reachable TLS endpoint while authorization is missing; it is not a successful application health check. Curl timing fields are cumulative phases, not a pure RTT measurement. An unanswered ICMP ping is not proof that the EKS API is down.

### VPN State and Metrics

```bash
: "${VPN_ID:?Select the reviewed VPN connection}"
: "${TUNNEL_IP:?Select its actual AWS tunnel outside IP}"
check_account
# Select telemetry only: do not dump customer gateway configuration or pre-shared keys.
aws ec2 describe-vpn-connections --region "$AWS_REGION" --vpn-connection-ids "$VPN_ID" \
  --query 'VpnConnections[].{id:VpnConnectionId,state:State,telemetry:VgwTelemetry}' \
  --output json > "$WORK_DIR/vpn-state.json"
jq -e --arg ip "$TUNNEL_IP" 'length==1 and any(.[0].telemetry[]?; .OutsideIpAddress==$ip)' \
  "$WORK_DIR/vpn-state.json" >/dev/null
export VPN_ID TUNNEL_IP
python3 - <<'PY'
import ipaddress, json, os
from datetime import datetime, timedelta, timezone
from pathlib import Path
ipaddress.ip_address(os.environ["TUNNEL_IP"])
now = datetime.now(timezone.utc)
end = now.replace(minute=now.minute - now.minute % 5, second=0, microsecond=0)
body = {"Namespace": "AWS/VPN", "MetricName": "TunnelState",
        "Dimensions": [{"Name": "VpnId", "Value": os.environ["VPN_ID"]},
                       {"Name": "TunnelIpAddress", "Value": os.environ["TUNNEL_IP"]}],
        "StartTime": (end - timedelta(minutes=15)).isoformat(), "EndTime": end.isoformat(),
        "Period": 300, "Statistics": ["Minimum", "Maximum"]}
(Path(os.environ["WORK_DIR"]) / "vpn-metric-request.json").write_text(json.dumps(body, indent=2) + "\n")
PY
aws cloudwatch get-metric-statistics --region "$AWS_REGION" \
  --cli-input-json "file://$WORK_DIR/vpn-metric-request.json" --output json \
  > "$WORK_DIR/vpn-metric-result.json"
jq '{label:.Label,datapoints:(.Datapoints|sort_by(.Timestamp))}' "$WORK_DIR/vpn-metric-result.json"
```

`available` is a VPN resource state, not tunnel health. TunnelState is 1 for UP/static or ESTABLISHED/BGP and 0 for other states; aggregates can be fractional. Keep absent data distinct from DOWN and verify both tunnels, routes and actual workload behavior. The query avoids dumping customer gateway configurations/pre-shared keys.

AWS's general connectivity recommendation is **100 Mbps bandwidth and RTT of 200ms or less**. When choosing VPN or Direct Connect, assess actual round-trip latency, packet loss, failover paths and workload response times alongside the circuit's nominal bandwidth.

## References

- [Hybrid networking](https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-networking.html)
- [EKS PrivateLink: management, OIDC and console endpoints](https://docs.aws.amazon.com/eks/latest/userguide/vpc-interface-endpoints.html)
- [S3 interface endpoints and private DNS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/privatelink-interface-endpoints.html)
- [Roles Anywhere endpoint policies](https://docs.aws.amazon.com/rolesanywhere/latest/userguide/vpc-interface-endpoints.html)
- [Current hybrid CNI support](https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-cni.html)
- [AWS hybrid BGP procedure](https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-cilium-bgp.html)
- [Mixed-mode DNS and webhooks](https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-webhooks.html)
- [Hybrid routing concepts](https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-concepts-kubernetes.html)
- [Hybrid traffic-flow reference](https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-concepts-traffic-flows.html)
- [Cilium 1.18.3 routing source](https://github.com/cilium/cilium/blob/v1.18.3/Documentation/network/concepts/routing.rst)
- [Cilium 1.18.3 DNS-policy source](https://github.com/cilium/cilium/blob/v1.18.3/Documentation/security/dns.rst)
- [RFC6996 private ASNs](https://www.rfc-editor.org/rfc/rfc6996.html)
- [RouterOS BGP reference](https://help.mikrotik.com/docs/spaces/ROS/pages/328220/BGP)
- [FRR 10.7.1 BGP reference source](https://github.com/FRRouting/frr/blob/frr-10.7.1/doc/user/bgp.rst)
- [VPN metrics](https://docs.aws.amazon.com/vpn/latest/s2svpn/monitoring-cloudwatch-vpn.html)
- [Kubernetes 1.36.5 kubelet server source](https://github.com/kubernetes/kubernetes/blob/v1.36.5/pkg/kubelet/server/server.go)
- [Kubernetes NetworkPolicy semantics](https://kubernetes.io/docs/concepts/services-networking/network-policies/)


< [Previous: Prerequisites](https://docs.atomai.click/kubernetes-docs/llms/en/eks-hybrid-nodes/01-prerequisites.md) | [Table of Contents](https://docs.atomai.click/kubernetes-docs/llms/en/eks-hybrid-nodes/README.md) | [Next: Restricted-Internet Setup](https://docs.atomai.click/kubernetes-docs/llms/en/eks-hybrid-nodes/03-airgap-setup.md) >