Virtualization & Containers: From Hypervisors to Docker & K8s

Virtualization and containerization are the foundational technologies of modern cloud infrastructure. Virtual machines provide strong hardware-level isolation, enabling multiple operating systems to share a single physical server. Containers provide lightweight, portable application packaging with process-level isolation, enabling higher density and faster startup. Understanding both technologies — their mechanisms, trade-offs, and appropriate use cases — is essential for architects and engineers building and deploying software at scale.

Virtual Machine Architecture

A virtual machine (VM) is a software emulation of a physical computer. The hypervisor, or Virtual Machine Monitor (VMM), manages the physical hardware and provides each VM with virtualized CPU cores, memory, storage, and network interfaces. Each VM runs its own operating system (guest OS), which is unaware that it is sharing hardware with other VMs.

Type 1 Hypervisors (Bare Metal)

Type 1 hypervisors run directly on the physical hardware, without a host operating system. They include a minimal kernel that manages CPU virtualization, memory virtualization, and I/O device emulation or passthrough. Examples:

• VMware ESXi: The dominant enterprise hypervisor, supporting advanced features like vMotion (live migration), DRS (Distributed Resource Scheduler), and HA (High Availability).
• KVM (Kernel-based Virtual Machine): Built into the Linux kernel as a loadable module. KVM transforms Linux into a Type 1 hypervisor — each VM runs as a normal Linux process (managed by QEMU for device emulation), scheduled by the kernel’s CFS alongside regular processes.
• Xen: Used by AWS (EC2) as the foundational hypervisor. Xen uses a privileged domain (Dom0) for driver management and unprivileged domains (DomU) for guest VMs.
• Hyper-V: Microsoft’s Type 1 hypervisor, integrated into Windows Server and Windows 10/11 Pro and Enterprise.

Type 1 hypervisors achieve near-native performance because the VM’s vCPU threads execute directly on physical CPU cores (with hardware virtualization extensions — Intel VT-x or AMD-V). Memory virtualization uses Extended Page Tables (EPT) or Nested Page Tables (NPT) to accelerate address translation.

Type 2 Hypervisors (Hosted)

Type 2 hypervisors run as applications on top of a host operating system. The host OS handles device drivers and resource scheduling; the hypervisor provides the VM abstraction on top. Examples include Oracle VirtualBox, VMware Workstation, and QEMU in full-emulation mode.

Type 2 hypervisors are primarily used for development, testing, and desktop virtualization. Performance is lower than Type 1 because each VM operation passes through an additional layer (the host OS kernel). However, their ease of use — no need to configure hardware drivers or manage a separate hypervisor OS — makes them popular for local development environments.

Hardware Virtualization Extensions

Before Intel VT-x (2005) and AMD-V (2006), x86 virtualization required binary translation — the hypervisor would intercept and translate privileged instructions from the guest OS. This was correct but slow. Modern hardware provides root mode operation: the hypervisor runs in VMX root mode, and guest VMs run in VMX non-root mode. Privileged instructions in non-root mode trap to the hypervisor automatically (via VM exits). The VM entry/exit cost is approximately 1,000–10,000 cycles, dramatically lower than binary translation.

Containers: OS-Level Virtualization

Containers virtualize at the operating system level rather than the hardware level. Multiple containers share the same host kernel but are isolated from each other through kernel primitives. This eliminates the per-VM overhead of running a separate guest OS.

Linux Namespaces

Namespaces partition global kernel resources so that each container sees only its own view. The Linux kernel implements eight namespace types:

A container is a process (or group of processes) running in its own namespace isolation. When the container exits, its namespaces are destroyed, and all associated resources are cleaned up.

Control Groups (cgroups)

Namespaces provide isolation (what a container can see); cgroups provide resource limits (what a container can use). Cgroups v2 (the unified hierarchy, default since Linux 5.0) manages CPU, memory, I/O, PID, and hugetlb controllers.

Each cgroup is a directory in /sys/fs/cgroup/. Limits are set by writing to interface files:

/sys/fs/cgroup/mycontainer/memory.max     → "512M"
/sys/fs/cgroup/mycontainer/cpu.max        → "50000 100000"  (50% of one CPU core)
/sys/fs/cgroup/mycontainer/io.max         → "8:0 rbps=104857600 wbps=104857600"

Cgroups prevent a single container from exhausting system resources and starving other containers or the host processes. The OOM killer operates within the cgroup’s memory limit — if a container exceeds its memory limit, the kernel kills a process inside that container (not a process on another container or the host).

Union Filesystems and Container Images

Container images use layered filesystems (OverlayFS, AUFS, device mapper) to enable efficient storage. An image consists of read-only layers, each representing a filesystem change (like a commit in version control). The top layer is writable — changes made by the running container are written here.

Container (writable layer)
    ↑ (copy-on-write)
Layer 4: "RUN pip install flask"
    ↑
Layer 3: "COPY app.py /app/"
    ↑
Layer 2: "RUN apt-get update && apt-get install -y python3"
    ↑
Layer 1: python:3.12-slim (base image)

Multiple containers running from the same image share the read-only layers. Only the writable container layer is unique per container. This reduces disk usage and image download time. Docker images are built in layers via Dockerfile instructions (FROM, RUN, COPY). Each instruction creates a new layer.

Docker and OCI Containers

Docker popularized containers by providing a user-friendly CLI, image format, registry (Docker Hub), and orchestration tooling. Under the hood, Docker uses runc (the reference implementation of the OCI Runtime Specification) to create and run containers, and containerd (graduated to CNCF) for image management and container lifecycle.

The OCI (Open Container Initiative) standardizes the container image format (OCI Image Spec) and the runtime behavior (OCI Runtime Spec). This means Docker images can be run by any OCI-compatible runtime: Podman (Red Hat), CRI-O (Kubernetes), containerd, and others.

Docker Compose

Docker Compose defines multi-container applications in YAML. Each service specifies its image, ports, volumes, environment variables, and dependencies. Compose creates a dedicated network for the application, resolving service names via DNS.

Container Orchestration with Kubernetes

Managing hundreds or thousands of containers across a cluster requires orchestration. Kubernetes (K8s), originally developed by Google and now the CNCF’s flagship project, is the industry standard.

Kubernetes abstracts physical or virtual servers into a cluster. Containers run in Pods (the smallest deployable unit, containing one or more containers sharing a network namespace). Deployments manage replicated pods with rolling update and rollback. Services provide stable network endpoints. ConfigMaps and Secrets manage configuration.

Kubernetes uses the kubelet agent on each node, which communicates with the control plane (API server, scheduler, controller manager, etcd). The scheduler places pods on nodes based on resource requirements and constraints. The kubelet communicates with the container runtime (containerd, CRI-O) via the CRI (Container Runtime Interface).

VMs vs Containers: When to Use Which

The fundamental difference is isolation boundary:

Choose VMs when: you need to run different operating systems (Linux VMs on a Windows host, or vice versa), your workload requires strong security isolation (multi-tenant environments with untrusted workloads), or you need to run software that requires kernel-mode components (some VPN clients, security software).

Choose containers when: you need density and fast startup for microservices, you want consistent deployment across environments (development, staging, production), or you are building cloud-native applications that benefit from orchestrator-managed scaling and rolling updates.

Security Considerations

VM Security

The hypervisor’s attack surface is relatively small (especially for Type 1 hypervisors). Side-channel attacks — Spectre, Meltdown, L1TF, MDS — demonstrated that information can leak between co-located VMs through shared CPU resources (cache, branch predictors, store buffers). Mitigations include CPU microcode updates, hypervisor patches (flushing buffers on VM entry/exit), and dedicated hardware (TEE — Trusted Execution Environments like Intel SGX and AMD SEV).

Container Security

Containers share a kernel, which means a kernel vulnerability can potentially escape container isolation. The container breakout attack on runC (CVE-2019-5736) demonstrated overwriting the host runC binary from within a container. Mitigations include:

• Running containers as non-root (user namespace remapping maps container root to host unprivileged user)
• Dropping capabilities with --cap-drop=ALL and adding only required ones
• Enabling seccomp profiles (restrict system calls available to the container)
• Applying AppArmor/SELinux profiles
• Using rootless Podman or rootless Docker (available since Docker 20.10)
• Keeping the host kernel updated with security patches

FAQ

What is the difference between KVM and QEMU?

KVM is a kernel module that provides hardware-assisted virtualization (CPU and memory virtualization). QEMU is a userspace emulator that provides device emulation (disk, network, USB, graphics). QEMU uses KVM for CPU virtualization when available, falling back to software emulation. Together, KVM+QEMU is the standard open-source virtualization stack on Linux.

Can containers run on virtual machines?

Yes — this is the dominant deployment model in cloud computing. A physical server runs a hypervisor (ESXi, KVM). Each VM runs a Linux kernel. Containers run inside the VMs. The container provides application isolation within the VM; the VM provides security isolation between tenants. AWS Fargate and Google Cloud Run use this layered approach.

What is a pod in Kubernetes, and why does it exist?

A pod is the smallest deployable unit in Kubernetes — one or more containers that share a network namespace (same IP address and port space) and can share storage volumes. The sidecar pattern (adding a logging or proxy container to a pod) is the primary use case for multi-container pods. Pods are ephemeral — they are created and destroyed by controllers (Deployments, StatefulSets, Jobs).

How does live migration work for VMs?

Live migration (vMotion on VMware) transfers a running VM from one physical host to another with zero downtime. The hypervisor copies memory pages from source to destination while the VM continues running. Pages modified during the copy are re-copied (dirty page tracking). When only a small number of pages remain dirty, the VM is paused for milliseconds, final state is transferred, and execution resumes on the destination host. Live migration requires shared storage (both hosts access the same VM disk) or storage vMotion (disk migration as well).

Why are containers considered less secure than VMs?

Containers share the host kernel with all other containers on the same host. A kernel vulnerability (CVE) can potentially allow code in one container to access memory or execute code in another container or the host. VMs have a hardware-enforced boundary — a VM escape requires compromising the hypervisor itself, which has a smaller codebase and attack surface. For multi-tenant environments with untrusted workloads (PaaS platforms), VMs are strongly preferred for the outer isolation boundary, with containers used inside each VM.

The Distributed OS Guide explains how virtualization enables resource pooling and fault tolerance in distributed systems. The Kernel Architecture Guide covers the kernel-mode mechanisms (namespaces, cgroups, KVM module) that make virtualization possible. The OCI specifications are maintained at opencontainers.org. Kubernetes documentation at kubernetes.io provides comprehensive deployment and administration guidance. For container internals, Liz Rice’s Container Security (O’Reilly) and the Docker documentation at docs.docker.com cover the operational details.