Lecture 4

Page ↑ / Page ↓ to move · scroll works too

Linux & DevOps — Lecture 4 of 8

The Commands You'll Live In

You can navigate the machine, and you know who’s allowed to touch what.
Today: run things, watch what they say, and move code onto the box.

All slides available at

slides.raafat.io

From static structure to the daily loop

Lecture 2 gave you the shell. Lecture 3 gave you the map and the lock. Both were about the machine sitting still. Today is about the loop you’ll run dozens of times a day for the rest of this course, and in any real DevOps job: start a service, check whether it’s actually working, read what it’s telling you when it isn’t, and ship a new version of the code it’s running.

systemctl journalctl ss dig curl git

systemctl

Units: the things systemd manages
Writing a custom .service unit
start/stop/enable/restart/status
Reading systemctl status output
Inventory: list-units, list-unit-files

A quick reminder of what systemd is doing

Service management

Starts, stops, and supervises services from declarative unit files — and restarts whatever crashes.

Parallel boot

Starts services concurrently based on a dependency graph, not a fixed sequence.

Centralized logging

journald collects every service's output into one structured store, read back with journalctl.

Targets

Named states — multi-user, graphical, rescue — the replacement for numbered runlevels.

This was Lecture 1’s theory. Everything after this slide is the hands-on side of it — systemctl is how you talk to systemd, unit files are what you write for it to manage.

Not every unit is a service

.service — a long-running or one-shot process
.target — a named grouping of units, a synchronization point
.timer — triggers another unit on a schedule, systemd's answer to cron
.socket — starts its paired service on first connection, not at boot

systemctl list-units --type=service filters to just one type when the full list is noisy.

Writing a custom .service unit

/etc/systemd/system/myapp.service

[Unit]

Description=Todo app placeholder service

After=network.target

[Service]

ExecStart=/opt/app/myapp.sh

Restart=on-failure

User=deploy

[Install]

WantedBy=multi-user.target

[Unit] describes ordering and dependencies, [Service] is what actually runs and how systemd should react if it dies, [Install] says which target pulls this unit in when enabled.

The verbs you’ll type constantly

$ sudo systemctl daemon-reload

$ sudo systemctl start myapp

$ sudo systemctl enable myapp

$ sudo systemctl restart myapp

$ sudo systemctl status myapp

$ systemctl is-active myapp

$ systemctl is-enabled myapp

$ sudo systemctl stop myapp

$ sudo systemctl disable myapp

daemon-reload after any edit to a unit file — systemd caches unit definitions and won't notice the change otherwise. enable only wires it into a target for next boot; it does not start it now, which is why you usually run both enable and start, or enable --now to do both at once.

Reading systemctl status output

$ systemctl status myapp

● myapp.service - Todo app placeholder service

Loaded: loaded (/etc/systemd/system/myapp.service; enabled)

Active: active (running) since Fri 2026-09-11 09:14:02 UTC; 2h ago

Main PID: 4821 (myapp.sh)

Tasks: 3 (limit: 4691)

Memory: 12.4M

CPU: 340ms

CGroup: /system.slice/myapp.service

└─4821 /opt/app/myapp.sh

Sep 11 09:14:02 host myapp.sh[4821]: listening on :8080

The dot is a traffic light: green ● running, red ● failed, white ○ inactive
Active line has both the state and how long it's held that state
The last few lines are a live tail of the journal — often enough to see the problem immediately

Taking inventory of what’s on a box

$ systemctl list-units --type=service --state=running

$ systemctl list-unit-files --type=service

$ systemctl --failed

systemctl --failed is the fastest "what's broken on this machine" you have — it lists only units that tried to start and gave up.

journalctl

journald as the log backend
Filtering: -u, -f, --since, -p
Persistent vs. volatile logs
The systemctl → journalctl workflow

One log store instead of scattered text files

Before systemd, each service wrote its own log file wherever it felt like, in whatever format it felt like. journald collects everything — kernel messages, every unit’s stdout/stderr, structured metadata like the unit name and PID — into one binary, indexed store. journalctl is the query tool in front of it.

Filtering: the flags you’ll use every time

$ journalctl -u myapp

$ journalctl -u myapp -f

$ journalctl -u myapp --since "10 min ago"

$ journalctl -u myapp --since today --until "1 hour ago"

$ journalctl -p err -b

$ journalctl -e

-u scopes to one unit — almost always your starting point
-f follows live, like tail -f
-p filters by priority (emerg…debug), -b scopes to the current boot
-e jumps straight to the end of the output

Persistent vs. volatile logs

$ journalctl --disk-usage

$ sudo mkdir -p /var/log/journal

$ sudo systemctl restart systemd-journald

$ sudo journalctl --vacuum-time=2weeks

$ sudo journalctl --vacuum-size=500M

By default the journal lives in /run/log/journal — memory, wiped on reboot. Creating /var/log/journal and restarting systemd-journald switches it to persistent, on-disk storage. —vacuum-* caps how much disk it’s allowed to hold onto.

The workflow: status tells you that, journalctl tells you why

systemctl status something is failed

→

journalctl -u <unit> --since "10 min ago"

→

Fix & restart systemctl restart

This is the loop for the rest of the course. Lecture 2’s journalctl -u sshd | tail -n 50 one-liner was your first taste of it — now you know the rest of the flags around it.

Network Diagnostics

The service is up according to systemctl. Nobody can reach it. Now what.

ss: what's actually listening
ping: is the host even there
dig / nslookup: is DNS the problem
curl -v: probing the handshake itself

ss: what’s actually listening

$ ss -tulpn

Netid State Local Address:Port Peer Address:Port Process

tcp LISTEN 0.0.0.0:8080 0.0.0.0:* users:(("myapp.sh",pid=4821))

tcp LISTEN 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=812))

-t TCP, -u UDP, -l listening sockets only, -p show the owning process, -n numeric ports instead of resolving names. ss is the modern replacement for netstat, which isn't installed by default on RHEL 9 or AlmaLinux — ss comes from iproute, a core package on both.

ping — is anyone home

$ ping -c 4 10.0.0.5

$ ping -c 4 example.com

dig / nslookup — is DNS the problem

$ dig example.com

$ dig +short example.com

$ nslookup example.com

-c 4 sends four pings and stops, instead of running forever. dig/nslookup come from bind-utils, which — like git — is not installed by default: sudo dnf install bind-utils.

curl -v: one level below “does it work”

$ curl -v http://localhost:8080/health

* Connected to localhost (127.0.0.1) port 8080

> GET /health HTTP/1.1

> Host: localhost:8080

< HTTP/1.1 200 OK

< Content-Length: 2

ping tells you the host is reachable at all, ss tells you something's listening on that port, curl -v tells you whether the actual HTTP conversation succeeds — three different layers, three different tools, in the order you'd reach for them.

curl

Enough to sanity-check a web service from the command line.

GET requests and following redirects
Headers only, and saving output
Checking a status code in a script

The handful of flags you need day to day

$ curl https://example.com

$ curl -L https://example.com

$ curl -I https://example.com

$ curl -o page.html https://example.com

$ curl -O https://example.com/file.tar.gz

No flags: prints the response body to stdout
-L follows redirects — plain curl stops at the first 3xx
-I fetches headers only, via a HEAD request
-o saves to a chosen filename, -O keeps the remote filename

This is deliberately minimal — using curl to POST data and drive an API is a later lecture’s job, once we’re building the webhook listener the capstone needs.

Checking a status code from a script

$ curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080/health

200

-s silences the progress meter, -o /dev/null throws away the body since you only care about the code, -w prints just the format you ask for. This one line is the core of a health check — the same idea a future deployment script will use to decide whether to switch traffic to a new instance.

Git

How code gets onto the machines you now know how to run services on.

clone, fetch, pull, push, remotes
The everyday loop: status, diff, add, commit, log
Branches and merging
.gitignore
Resolving a merge conflict

First: it’s not there by default

$ git --version

bash: git: command not found...

$ sudo dnf install git

$ git --version

git version 2.43.5

Same story as bind-utils in the last part — RHEL 9 and AlmaLinux 9 don’t ship git on a minimal install. Straight out of Lecture 3: it’s just another package, one dnf install away.

Getting a copy, and where it comes from

$ git clone https://github.com/example/todo-app.git

$ cd todo-app

$ git remote -v

origin https://github.com/example/todo-app.git (fetch)

origin https://github.com/example/todo-app.git (push)

$ git fetch

$ git pull

A remote is just a name for a URL. fetch downloads new commits without touching your working files; pull is fetch immediately followed by a merge into your current branch — which is why a pull can produce a conflict and a bare fetch never can.

The loop you’ll run dozens of times a day

$ git status

$ git diff

$ git add app.py

$ git commit -m "Fix health check endpoint"

$ git log --oneline --graph

status — what's changed, what's staged
diff — the actual line-by-line change, before you commit to it
add stages a change, commit records it with a message
log --oneline --graph — the history, compactly

.gitignore: telling git what isn’t source code

.gitignore

*.log

__pycache__/

.env

node_modules/

Logs, caches, dependency directories, and secrets don’t belong in the repository — a stray .env committed once stays in the history forever, even if you delete it in a later commit.

Branches: working on something without breaking main

$ git switch -c feature/health-endpoint

$ git add app.py

$ git commit -m "Add /health endpoint"

$ git switch main

$ git merge feature/health-endpoint

$ git branch -d feature/health-endpoint

switch -c creates and moves onto a new branch in one step (the older, still-common spelling is checkout -b). A merge into a branch that hasn't moved since you branched off is a fast-forward — no new commit needed, just the pointer moving.

When two branches touch the same lines

app.py

<<<<<<< HEAD

PORT = 8080

=======

PORT = int(os.environ.get("PORT", 5000))

>>>>>>> feature/health-endpoint

$ git merge feature/health-endpoint

Auto-merging app.py

CONFLICT (content): Merge conflict in app.py

$ vim app.py

$ git add app.py

$ git commit

Git marks both versions in the file itself. Edit it down to what it should actually say, delete the <<<<<<</=======/>>>>>>> markers, then add and commit like normal — the commit is what tells git the conflict is resolved.

The one-liner from Lecture 2, fully explained now

rollout.sh

ssh "$host" "cd /opt/app && git pull && systemctl restart app"

ssh — Lecture 2
git pull — fetch and merge the latest commit, today
systemctl restart — reload the service with the new code, today

Every piece of this line is now something you’ve actually learned, not just typed.

The Commands You'll Live In

From static structure to the daily loop

systemctl

A quick reminder of what systemd is doing

Service management

Parallel boot

Centralized logging

Targets

Not every unit is a service

Writing a custom .service unit

The verbs you’ll type constantly

Reading systemctl status output

Taking inventory of what’s on a box

journalctl

One log store instead of scattered text files

Filtering: the flags you’ll use every time

Persistent vs. volatile logs

The workflow: status tells you that, journalctl tells you why

Network Diagnostics

ss: what’s actually listening

ping — is anyone home

dig / nslookup — is DNS the problem

curl -v: one level below “does it work”

curl

The handful of flags you need day to day

Checking a status code from a script

Git

First: it’s not there by default

Getting a copy, and where it comes from

The loop you’ll run dozens of times a day

.gitignore: telling git what isn’t source code

Branches: working on something without breaking main

When two branches touch the same lines

The one-liner from Lecture 2, fully explained now

Lab & Logistics

Fin