linux / systemd

updated September 02, 2026

I run long-lived processes on Linux VMs as systemd services so they survive SSH logout and restart on crash or reboot. systemd also captures each service's output, so journalctl reads it back.

A service unit

This unit runs the farmer from cmd / cibot:

# /etc/systemd/system/cibot-farmer.service
[Unit]
Description=cibot farmer
After=network-online.target
Wants=network-online.target

[Service]
User=croaky
EnvironmentFile=/etc/cibot-farmer.env
ExecStart=/usr/local/bin/cibot farmer
Restart=always
RestartSec=2

[Install]
WantedBy=multi-user.target

Install and start it:

sudo cp cibot-farmer.service /etc/systemd/system/
sudo systemctl enable --now cibot-farmer

enable starts it on every boot and --now starts it now. Restart=always restarts it after it exits.

Template units

A template unit, marked by the @ suffix, runs several copies of one process:

# /etc/systemd/system/[email protected]
[Service]
User=croaky
EnvironmentFile=/etc/cibot.env
ExecStart=/usr/local/bin/cibot worker
Restart=always

sudo systemctl enable --now cibot-worker@{1..8}

That starts cibot-worker@1 through @8.

Environment variables

systemd gives a process a minimal environment, so I keep config and secrets in an EnvironmentFile. The path /etc/cibot.env is my choice. Debian uses /etc/default/<name> and Red Hat uses /etc/sysconfig/<name>.

# /etc/cibot.env
FARMER_URL=http://10.50.96.89:1994
BOX_TOKEN=...
PATH=/usr/local/go/bin:/usr/local/bin:/usr/bin:/bin

The default PATH is minimal, so I set it when the process shells out to go or git. The file holds secrets, so I restrict it: sudo chown root:root /etc/cibot.env && sudo chmod 600 /etc/cibot.env.

Everyday commands

sudo systemctl restart cibot-farmer
sudo systemctl stop cibot-farmer
systemctl is-active cibot-farmer
systemctl status cibot-farmer
systemctl show cibot-farmer -p NRestarts --value

After editing a unit file, reload systemd's view before restarting:

sudo systemctl daemon-reload

Logs

systemd captures each service's stdout and stderr into its journal, so I read logs with journalctl rather than hunting for files. Every unit's output is addressable by name:

journalctl -u cibot-farmer         # everything for the unit
journalctl -u cibot-farmer -n 50   # last 50 lines
journalctl -u cibot-farmer -f      # follow live, like tail -f
journalctl -u cibot-farmer -e      # jump to the end

A template unit is addressable by instance:

journalctl -u 'cibot-worker@1'
journalctl -u 'cibot-worker@*'     # all instances

By time:

journalctl -u cibot-farmer --since "10 min ago"
journalctl -u cibot-farmer --since today
journalctl -u cibot-farmer --since "2026-07-19 17:00" --until "18:00"

By boot and severity:

journalctl -b        # since the current boot
journalctl -b -1     # the previous boot
journalctl -p err    # errors and worse, any unit

Filter with grep, or with the journal's own matching:

journalctl -u cibot-farmer | grep error
journalctl -u cibot-farmer -g connect   # journal-native regex

The service prints plain text and systemd stores it. There are no log file paths to remember and no logrotate to configure, and journalctl reads the same way for every service on the box.

← All articles