Systemd Guide: Services, Timers & Logging

Understanding Systemd

Systemd is the init system and service manager used by most modern Linux distributions, including Debian (since 2015), Ubuntu (since 15.04), Fedora, Red Hat Enterprise Linux (since 7), Arch Linux, and SUSE. It replaces the traditional System V init with parallel service startup, socket-based activation, centralized logging (journald), and unified resource management through cgroups. Systemd manages services, timers, mounts, sockets, devices, swap files, and targets (groupings of units that define system states). The core design philosophy is that the init system should manage everything related to process lifecycle and resource tracking. While systemd has been controversial in the Linux community — particularly regarding complexity and deviation from the Unix philosophy — it has become the de facto standard due to its technical advantages in modern server environments (Poettering, “Systemd for Administrators,” 0pointer.de/blog/projects). The adoption by all major distributions ensures that understanding systemd is essential for professional Linux administration.

Unit Types and File Locations

Systemd manages system resources through unit files. Each unit type corresponds to a resource category:

Unit files are loaded from three directories, with later directories overriding earlier ones: /lib/systemd/system/ (distribution defaults), /run/systemd/system/ (runtime overrides), /etc/systemd/system/ (admin overrides). The precedence hierarchy allows administrators to customize any system service without modifying distribution-provided files.

Service Unit File Structure

A comprehensive service unit includes metadata, execution configuration, and target integration:

[Unit]
Description=Web Application Server
After=network.target postgresql.service
Wants=postgresql.service
Requires=network-online.target

[Service]
Type=simple
User=webapp
Group=webapp
WorkingDirectory=/opt/webapp
ExecStart=/usr/bin/webapp --config /etc/webapp/config.yaml
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
RestartSec=5s
TimeoutStopSec=30s

[Install]
WantedBy=multi-user.target

The Wants directive is a soft dependency. Requires is a hard dependency. The Type directive has several variants: simple (the process is the main process), forking (the process forks and the parent exits), oneshot (for short-lived tasks), notify (the process sends sd_notify()), and dbus (service registers on D-Bus). The Restart directive supports always, on-failure, on-abnormal, on-abort, and on-watchdog strategies.

Systemctl Essential Commands

systemctl status webapp              # Detailed status with recent logs
systemctl list-units --type=service  # All loaded services
systemctl list-unit-files            # All installed unit files
systemctl show webapp                # All properties of a unit
systemctl enable --now webapp        # Enable and start in one command
systemctl mask webapp.service        # Prevent any activation
systemctl list-dependencies webapp   # Show dependency tree

systemctl list-units --state=failed shows all failed services. systemctl list-dependencies --reverse shows which units depend on the specified unit, useful for understanding shutdown ordering.

Journalctl for Logging

Journald stores logs as structured binary data. journalctl provides powerful filtering:

journalctl -u webapp                  # Specific unit
journalctl --since "1 hour ago"       # Time-based filter
journalctl -p err -b                  # Errors from current boot
journalctl -o json                    # Machine-parsable output
journalctl --disk-usage               # Storage consumption

Persistent journal storage requires /var/log/journal/. When absent, logs disappear on reboot. journalctl --vacuum-size=500M limits disk usage.

Timers Replacing Cron

Systemd timers run service units on schedules or after events:

# /etc/systemd/system/db-backup.timer
[Unit]
Description=Daily database backup

[Timer]
OnCalendar=daily
Persistent=true
RandomizedDelaySec=1h

[Install]
WantedBy=timers.target

Persistent=true catches up on missed runs after system downtime. RandomizedDelaySec prevents all services from starting simultaneously, solving the “thundering herd” problem. Monotonic timers (OnUnitActiveSec=1h) trigger relative to the last activation.

Boot Analysis

systemd-analyze blame shows boot time by service. systemd-analyze critical-chain shows the boot time bottleneck chain. systemd-analyze plot > boot.svg generates a detailed SVG visualization. Use systemd-analyze time for total boot time. These tools are essential for optimizing boot performance.

Managing Dependencies and Ordering

Systemd’s dependency system ensures reliable service startup ordering. The After= directive only affects ordering, not activation — use Requires= or Wants= for activation. The PartOf= directive stops or restarts a dependent unit when the specified unit changes state. BindsTo= creates a stronger relationship: stopping the bound unit automatically stops the dependent. The Conflicts= directive ensures mutually exclusive services cannot run simultaneously. For complex multi-service applications, target units group related services: WantedBy=multi-user.target is the standard target for most services. Custom targets like myapp.target can group application-specific services. The systemctl list-dependencies command visualizes the dependency tree. Common ordering pitfalls include: circular dependencies that prevent startup; missing After= directives causing race conditions on socket or database availability; and incorrect target configuration causing services to start before their dependencies. The systemd-analyze verify command checks unit files for common configuration errors before deployment.

Systemd Journal Maintenance

The journald logging subsystem requires ongoing maintenance to prevent disk exhaustion. Journal files are stored in /var/log/journal/ for persistent storage, automatically rotated when SystemMaxUse is reached. The journalctl --rotate command forces log rotation, useful before backup operations. Journald’s compression reduces storage by 80-90% for text-heavy logs. The MaxRetentionSec directive automatically removes entries older than the specified duration. For debugging, journalctl --list-boots shows all boot entries with timestamps. The journalctl --output=export format produces machine-parseable output suitable for ingestion by centralized logging systems like Loki, Elasticsearch, or Graylog. The systemd-journal-upload service forwards logs to a remote journal server over HTTPS, with TLS mutual authentication for security. For high-volume logging, the RateLimitIntervalSec and RateLimitBurst directives prevent log flooding. The Storage=volatile setting in journald.conf runs a purely in-memory journal for read-only root filesystems or systems with limited flash write endurance. The journalctl --header command displays journal file metadata including size, number of entries, and sealing state.

Systemd Timer Units

Systemd timer units replace traditional cron jobs with more expressive scheduling and integrated logging. A timer unit pairs with a service unit of the same name: myjob.timer triggers myjob.service. The OnCalendar= directive accepts cron-like syntax: OnCalendar=*-*-* 02:00:00 runs daily at 2 AM, OnCalendar=Mon..Fri 09:00:00 runs weekdays at 9 AM. The OnBootSec= and OnUnitActiveSec= directives enable relative scheduling: OnBootSec=10min starts the timer 10 minutes after boot, OnUnitActiveSec=1h repeats every hour after the previous run completes. The RandomizedDelaySec= directive prevents thundering herd problems by randomizing startup within a window. The AccuracySec= directive controls scheduling precision: default is 1 minute, set lower for time-sensitive tasks.

Timer units support monotonic and realtime event types. Monotonic timers (OnBootSec, OnUnitActiveSec, OnStartupSec) are tied to system uptime, unaffected by clock changes. Realtime timers (OnCalendar) use wall-clock time and respect timezone changes. The Persistent=true directive runs missed timer events after boot if the system was powered off during the scheduled time, preventing skipped maintenance tasks. Timer status is checked with systemctl list-timers --all, which shows the next trigger time, elapsed since last trigger, and the associated service unit. Timers also support OnClockChange and OnTimezoneChange triggers, though these are rarely used. The systemd-analyze calendar "*-*-* 02:00:00" command validates calendar expressions and shows upcoming trigger times.

Systemd Service Dependencies and Ordering

Service dependencies in systemd combine requirement relationships with ordering constraints. Requires= specifies hard dependencies: the service fails to start if requirements fail. Wants= specifies soft dependencies: the service starts regardless of requirement status. BindsTo= ties service lifetime to a dependency: if the dependency stops, this service stops. PartOf= makes the service stop/start with a parent unit. Conflicts= ensures services cannot run simultaneously: starting one stops the other.

Ordering directives (Before=, After=) sequence service startup regardless of requirement status. After=network.target ensures networking is available before service start, but does not require it. Combine After= with Requires= or Wants= for both ordering and requirement semantics. The network-online.target waits for actual network connectivity, not just interface configuration. The syslog.target and time-sync.target ensure logging and time synchronization before dependent services. The systemd-analyze dot command generates dependency graphs. The systemd-analyze critical-chain identifies the longest startup path for boot optimization. For complex deployments, use target units to group services: multi-user.target is the standard multi-user boot target, and custom targets (myapp.target) can group application services.

Resource Limits with Systemd

Systemd imposes resource limits on services through directives that map to Linux rlimits and cgroup controllers. Core resource limits set with LimitCPU=, LimitNOFILE=, LimitNPROC=, LimitFSIZE=, and LimitDATA= correspond to the traditional setrlimit syscall values. The infinity value removes the limit. TasksMax= sets the maximum number of tasks (threads/processes) for the service cgroup, preventing fork bombs. MemoryMax= and MemoryHigh= set cgroup v2 memory limits.

CPU limits use CPUQuota= with percentage: CPUQuota=200% allocates two cores. CPUWeight= sets proportional CPU share during contention (100 default, range 1-10000). I/O limits: IOReadBandwidthMax=, IOWriteBandwidthMax=, IOReadIOPSMax=, IOWriteIOPSMax= control throughput. The systemd.resource-control(5) man page details all options. Resource limits should be configured for each service based on its performance requirements and the host’s capacity. The systemctl show myservice output includes effective resource limits after unit file parsing and defaults.

Frequently Asked Questions

What is the difference between systemctl enable and systemctl start? enable creates symlinks so the service starts at boot. start immediately activates the service.

How do I view logs from the previous boot? journalctl -b -1 shows logs from the previous boot.

How do I troubleshoot a service that fails to start? Check systemctl status myservice, journalctl -u myservice -n 50, and systemctl show myservice.

What is the difference between After and Requires? After controls startup ordering. Requires enforces that the depended unit must start successfully.

How do I run a script at boot with systemd? Create a service unit with Type=oneshot and RemainAfterExit=yes, then enable it.

Related: Linux Systemd Advanced | Linux Namespaces | Linux Text Processing