Operating Systems: Core Concepts & Architecture Overview

An operating system is the software layer that manages computer hardware and provides a consistent, abstracted interface for application programs. It mediates between the physical resources of the machine — CPU cores, memory banks, storage devices, network interfaces — and the software that makes those resources useful. Understanding OS fundamentals is essential for systems programming, performance engineering, security analysis, and debugging at any level of the software stack.

This overview serves as a roadmap to the major OS subsystems, each of which is examined in detail in dedicated guides in this series. The textbook references throughout are Silberschatz, Galvin, and Gagne’s Operating System Concepts (10th edition) and Tanenbaum and Bos’s Modern Operating Systems (4th edition), the two most widely used OS textbooks in computer science education.

What an Operating System Does

The OS has three primary roles. As a resource allocator, it decides which process gets the CPU, which memory pages remain in RAM, and in what order disk I/O requests are serviced. As a control program, it prevents one user’s process from interfering with another’s and enforces security policies. As a hardware abstraction layer, it presents a consistent API — the same system calls (open, read, write, fork, mmap) work across diverse hardware configurations.

The OS services can be categorized into:

• Process management: Creating, scheduling, synchronizing, and terminating processes and threads.
• Memory management: Allocating and deallocating memory, translating virtual addresses to physical addresses, and enforcing isolation.
• File system management: Organizing data on persistent storage into files and directories.
• I/O management: Interfacing with input/output devices through device drivers.
• Security and access control: Authentication, authorization, and resource protection.
• Networking: Managing network protocol stacks and communication end points.

Process Management Fundamentals

A process is a program in execution. Each process has its own address space, file descriptors, and execution state. The OS maintains a Process Control Block (PCB) for each process — a kernel data structure that stores the process state (running, ready, waiting), program counter, CPU registers, memory management information, and accounting data.

Processes can create child processes (via fork() on Unix, CreateProcess() on Windows). The process hierarchy forms a tree. Each process (except PID 0, the idle process) has a parent. In Linux, the pstree command visualizes this hierarchy.

Processes transition through states: New (being created) → Ready (in memory, waiting for CPU) → Running (executing on CPU) → Waiting (blocked on I/O or event) → Terminated (finished). The scheduler moves processes between Ready and Running.

Threads — lightweight processes that share the address space of their parent — enable parallel execution within a single process. The OS scheduler treats threads as the unit of execution. Multithreading improves CPU utilization on multicore systems and simplifies programming for I/O-bound workloads.

CPU Scheduling

The scheduler decides which ready process runs on which CPU core. Scheduling policies trade off throughput (work completed per unit time), fairness (each process gets its share), response time (time to first execution for interactive processes), and overhead (context switch cost).

The Completely Fair Scheduler (CFS) in Linux, introduced in kernel 2.6.23, maintains a red-black tree of processes keyed by virtual runtime (the amount of time each process has run, weighted by priority and nice value). CFS always runs the process with the smallest virtual runtime — ensuring that over any time window, each process receives its proportional share of CPU time. The time slice is dynamic, typically 4–6 milliseconds per scheduling round.

Windows uses a priority-driven preemptive scheduler with 32 priority levels (0 for idle, 1–15 for dynamic, 16–31 for real-time). Priorities can be boosted temporarily when a thread’s foreground window receives input, improving interactive responsiveness.

Memory Management and Virtual Memory

Memory management gives each process its own virtual address space, translated to physical memory by the Memory Management Unit (MMU). The page table maps virtual pages to physical frames. Demand paging loads pages from disk only when they are accessed — most processes use only a fraction of their virtual address space at any time, so demand paging dramatically reduces memory requirements.

The page cache caches file content in physical memory. When a process reads a file, the kernel checks the page cache first. On a cache hit (common for actively used files), the read completes in microseconds instead of the milliseconds required for disk I/O. The page cache and the virtual memory system are unified in Linux — pages in the page cache are fully integrated with the VM subsystem and can be reclaimed when memory pressure increases.

Synchronization and Concurrency

Concurrent access to shared resources creates race conditions. The OS provides synchronization primitives to enforce mutual exclusion: mutexes (simple locks), semaphores (counting locks for multiple resources), condition variables (for signaling between threads), and read-write locks (for read-mostly data). In the kernel, spinlocks provide lightweight locking for short critical sections where sleeping is not allowed.

Classic concurrency problems illustrate the challenges. The producer-consumer problem requires coordinating bounded buffer access between producing and consuming threads. The readers-writers problem balances concurrency for readers with exclusive access for writers. The dining philosophers problem demonstrates deadlock and livelock scenarios with multiple contending processes.

Deadlocks

A deadlock occurs when each process in a set is waiting for a resource held by another process in the set. The four Coffman conditions (mutual exclusion, hold-and-wait, no preemption, circular wait) must all hold. The OS can prevent deadlocks (by ensuring at least one condition never holds), avoid them (by evaluating resource requests against the Banker’s Algorithm), detect them (by finding cycles in the wait-for graph), or recover from them (by killing processes or preempting resources).

File Systems

The file system organizes data on persistent storage. Files are named data objects; directories form a hierarchical namespace. The Virtual File System (VFS) layer in the kernel provides a uniform interface (open, read, write, lseek, close) regardless of the underlying storage implementation.

Popular file systems optimize for different workloads: ext4 for general Linux use, NTFS for Windows (with per-file encryption and compression), APFS for macOS (optimized for SSDs with copy-on-write and snapshots), and XFS for large-scale storage servers. The choice of file system affects performance under concurrent access, crash recovery time, maximum file and volume sizes, and data integrity guarantees.

I/O Systems

The I/O subsystem bridges the CPU and peripherals. Three methods control data transfer: programmed I/O (CPU busy-waits), interrupt-driven I/O (device interrupts the CPU when data is ready), and Direct Memory Access (device transfers data directly to/from memory, with a single interrupt per transfer). DMA is mandatory for high-throughput devices — without it, a 10 Gbps network link would consume 100% of a CPU core just copying packets.

OS Design Space

The kernel architecture debate between monolithic and microkernel designs shaped OS research for decades. Linux uses a monolithic kernel (all services in kernel mode, maximizing performance), while QNX and seL4 use microkernels (minimal kernel, services as user processes, maximizing fault isolation). Windows NT and macOS XNU use hybrid approaches, running critical services in kernel mode while maintaining modular subsystem boundaries. The practical outcome is that monolithic kernels dominate general-purpose computing while microkernels dominate embedded, safety-critical, and security-focused environments.

FAQ

Which OS is best for learning operating system concepts?

Linux is the preferred platform for OS learning because the complete source code is available, the kernel is well-documented in Documentation/, and tools like QEMU/GDB enable kernel debugging without dedicated hardware. MINIX, the system Tanenbaum designed for teaching, remains an excellent educational resource with a clean microkernel architecture.

How do Linux and Windows handle processes differently?

Linux uses fork()/exec() for process creation — a two-step approach where fork() clones the parent and exec() loads a new program. Windows uses CreateProcess() as a single call that creates the process and loads the executable. Linux threads are implemented as processes that share an address space (clone() with CLONE_VM); Windows threads are explicit kernel objects.

What is the role of the kernel in security?

The kernel enforces the user/kernel privilege boundary, validates all system call arguments, checks file permissions against the process’s credentials, and manages audit logs. On Linux, security modules (SELinux, AppArmor) implement mandatory access control policies. The kernel’s correctness is the foundation of system security — an exploit that compromises the kernel can bypass all security mechanisms.

How does the OS manage multiple CPU cores?

The scheduler maintains a per-CPU run queue and implements load balancing (pulling processes from busy CPUs to idle ones). Memory management uses per-CPU page allocator caches and NUMA-aware allocation. Synchronization primitives (spinlocks) are optimized for multi-core systems with MCS locks and qspinlocks to reduce cache line bouncing.

What happens when memory runs out?

When physical memory is exhausted, the kernel’s page reclamation code (kswapd kernel thread and direct reclaim path) frees pages: evicting file-backed clean pages, writing dirty pages back to disk, and swapping anonymous pages to the swap device. If reclaim cannot free enough memory, the Out-Of-Memory (OOM) killer selects a process to terminate based on an heuristic score (oom_score).

This overview provides the foundation for the detailed OS subsystem guides. The Process Scheduling Guide expands on CPU scheduling algorithms. The Memory Management Guide covers paging, page replacement, and TLB management. The Synchronization and Concurrency Guide details mutexes, semaphores, and race conditions. For further reading, Silberschatz, Galvin, and Gagne’s Operating System Concepts (10th ed.) is the standard academic reference; Tanenbaum and Bos’s Modern Operating Systems (4th ed.) provides deeper coverage of design trade-offs.