Operating System Security Guide

Operating system security is the foundation of all computer security. If the OS is compromised, nothing above it is safe. The OS mediates every access to hardware resources, enforces isolation between processes, and provides the trust model that all applications depend on.

The OS Security Model

Layers of Trust

Applications (user-space)     ← Most vulnerable
Libraries (system calls)      
Kernel (privileged)           ← Most protected
Hardware (CPU, memory)

Privilege Rings

Most CPUs support multiple privilege levels:

Ring 0 (Kernel)    ─── OS kernel, device drivers
Ring 1 (System)    ─── Some OS services (rarely used)
Ring 2 (System)    ─── Some OS services (rarely used)
Ring 3 (User)      ─── Applications

Key protection: User-space code cannot directly access kernel memory or hardware. All hardware access must go through system calls, which the kernel validates before executing.

User Authentication

Authentication Factors

Password Storage

Never store plaintext passwords.

import hashlib, os

def hash_password(password):
    salt = os.urandom(32)  # 32 bytes random salt
    hash = hashlib.pbkdf2_hmac(
        'sha256',
        password.encode(),
        salt,
        100000  # Iterations (100K+ in 2026)
    )
    return salt + hash  # Store both

def verify_password(password, stored):
    salt = stored[:32]
    hash = stored[32:]
    new_hash = hashlib.pbkdf2_hmac('sha256', password.encode(), salt, 100000)
    return hash == new_hash

Multi-Factor Authentication

Combine factors for stronger security. A password plus a TOTP code from an authenticator app raises the bar dramatically: an attacker must both guess your password and gain access to your phone. Hardware security keys (FIDO2/WebAuthn) provide the strongest second factor, resisting phishing attacks that can bypass TOTP.

Access Control

DAC (Discretionary Access Control)

Users control access to their own files. Standard in Linux/Unix.

-rwxr-xr--  1 alice  staff  1024 Jan 1  file.txt
 │││││││
 ││││││└── Other: read (r--)
 │││││└─── Group: read+execute (r-x)
 │││└──── Owner: read+write+execute (rwx)
 ││└─────── ACL indicator (. for standard)
 └───────── File type (-=file, d=directory)

MAC (Mandatory Access Control)

System-wide policy enforced by the kernel. Users can’t override it.

Capability-Based Security

Instead of “root vs. user”, grant specific capabilities:

Linux capabilities:
  CAP_NET_RAW    — Use raw sockets
  CAP_SYS_ADMIN  — Many admin operations
  CAP_NET_BIND   — Bind to privileged ports (<1024)

# Grant a binary only the capability it needs
setcap cap_net_bind_service=+ep /usr/bin/my_web_server

Principle of Least Privilege

Every process should run with only the minimum privileges needed to perform its function. This principle is violated more often than any other in system security. Services that run as root when they only need network access are responsible for the majority of privilege escalation vulnerabilities. Capabilities, containers, and dedicated user accounts are all tools for implementing least privilege.

Memory Protection

ASLR (Address Space Layout Randomization)

Randomizes memory addresses to make buffer overflow exploitation harder:

Without ASLR:
  Stack:    0xBFFFF000 (fixed)
  Heap:     0x08048000 (fixed)
  Libraries: 0x40000000 (fixed)

With ASLR:
  Stack:    0xBFFD2000 (changes per run)
  Heap:     0x08934000 (changes per run)
  Libraries: 0x4A500000 (changes per run)

NX Bit (No-Execute)

Marks memory pages as non-executable. Code injection fails.

Stack:        ┌─────────────────┐
              │ Read + Write    │
              │ (NO execute)    │ ← Injected code can't run here
              └─────────────────┘

Code segment: ┌─────────────────┐
              │ Read + Execute  │
              │ (NO write)      │ ← Can't modify code
              └─────────────────┘

Stack Canaries

A guard value placed on the stack to detect buffer overflows:

void vulnerable(char *input) {
    int canary = 0xDEADBEEF;  // Random value
    char buffer[64];
    strcpy(buffer, input);    // If overflow, canary is corrupted
    if (canary != 0xDEADBEEF) {
        abort();  // Buffer overflow detected!
    }
---

Control Flow Integrity (CFI)

Modern operating systems and compilers implement CFI to prevent control-flow hijacking attacks like ROP (Return-Oriented Programming). CFI instruments indirect branches to ensure they only target valid destinations. Windows Control Flow Guard, LLVM CFI, and Intel CET (Control-flow Enforcement Technology) are production implementations. These defenses make exploitation significantly harder even when a memory corruption vulnerability exists.

Secure Boot

UEFI Secure Boot

Each component verifies the next before executing:

UEFI firmware → Bootloader → Kernel → OS modules
    ↓              ↓           ↓          ↓
  signed         signed      signed     signed

Measured Boot (TPM)

Records hash measurements in TPM (Trusted Platform Module):

Step 1: BIOS measures bootloader → TPM records hash
Step 2: Bootloader measures kernel → TPM records hash
Step 3: Kernel measures modules → TPM records hash

Result: Remote attestation can verify the exact boot chain

Full Disk Encryption

Encrypts the entire disk so data is protected when the device is off:

Linux Kernel Hardening

Sandboxing

Container Isolation (Docker, Podman)

Container A      Container B
┌──────────┐    ┌──────────┐
│ Process  │    │ Process  │
│ FS, Net  │    │ FS, Net  │
└────┬─────┘    └────┬─────┘
     │               │
     └───────┬───────┘
        Host Kernel (shared)

seccomp (Secure Computing)

Restricts system calls available to a process:

// Allow only: read, write, exit, sigreturn
scmp_filter_ctx ctx = seccomp_init(SCMP_ACT_KILL);
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(read), 0);
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 0);
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(exit), 0);
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(rt_sigreturn), 0);
seccomp_load(ctx);

Common OS Vulnerabilities

Spectre and Meltdown

CPU speculative execution vulnerabilities (2018):

Mitigation: Kernel Page Table Isolation (KPTI), microcode updates, compiler barriers. Spectre mitigations continue to evolve as researchers discover new speculative execution variants. Each mitigation imposes performance costs, typically 5-30 percent depending on workload.

Defense in Depth

Application security (OAuth, CSP, input validation)
        ↓
Sandboxing (containers, seccomp, AppArmor)
        ↓
Kernel hardening (ASLR, NX, canaries, SELinux)
        ↓
Hardware security (Secure Boot, TPM, AES-NI)
        ↓
Physical security (locked doors, HSM)

Each layer assumes the layer above it is compromised. A defense-in-depth strategy ensures that an attacker who breaches one layer still faces multiple additional barriers before achieving their objective.

An OS that’s hardened at every layer is exponentially harder to exploit than one relying on a single defense.

FAQ

What is the most important OS security setting? Enabling full-disk encryption is the single most impactful security measure. If a device is lost or stolen, encryption ensures the data cannot be read without the decryption key. Combine this with a strong password and a TPM for the best protection.

How does ASLR prevent exploits? ASLR randomizes the memory layout of a process, making it difficult for an attacker to predict the address of specific functions or gadgets needed to exploit a vulnerability. Combined with NX and stack canaries, ASLR makes buffer overflow exploitation exponentially harder.

What is the difference between SELinux and AppArmor? Both implement mandatory access control (MAC) for Linux. SELinux uses security labels attached to every file, process, and socket, with policy rules defined across the entire system. AppArmor uses path-based profiles that are easier to configure but provide less granular control. SELinux offers stronger security at the cost of complexity.

Can containers provide full isolation? No. Containers share the kernel with the host, so a kernel vulnerability can escape container boundaries. Virtual machines provide stronger isolation with separate kernels. Containers are suitable for defense in depth but not for hard multi-tenant security boundaries.

Kernel Architecture Guide — Memory Management Guide — Virtualization Guide