Page ↑ / Page ↓ to move · scroll works too
Linux & DevOps — Lecture 6 of 8
Your VM deploys itself on every push.
Today: find out how it’s doing — before someone tells you it’s down.
So far, the way you learn something is wrong has been the hard way: the page doesn’t load, or a deploy silently didn’t happen, and then you SSH in and go digging with systemctl and journalctl. Monitoring flips that around. Small programs on the VM report numbers about it — CPU, memory, disk, requests served, whether the chess site answers — a collector stores those numbers over time, a dashboard draws them, and rules watch them so a problem shows up on a screen instead of in a message from someone else.
Three jobs, three tools — and the collector goes and gets the data itself.
Numbers sampled over time: CPU at 12%, 340 requests served, disk at 61%. Cheap to store, easy to graph, great for trends and is it healthy right now. Today's focus.
Text events with detail: why a request failed, what a service printed before it died. You already have this — journalctl from Lecture 4.
A rule that watches a metric and changes state when it crosses a line, so you don't have to be staring at the dashboard when it happens.
Metrics tell you that something is wrong and roughly when; logs tell you why. In practice you spot the problem on a graph, then go read journalctl -u for the culprit.
Nothing pushes data anywhere. Each exporter just serves its current numbers as plain text at /metrics over HTTP; Prometheus visits each one on a schedule — a scrape — and appends the numbers to its database. Grafana stores nothing itself: it asks Prometheus questions and draws the answers. If an exporter is down, Prometheus notices right away — the scrape fails — which is why “is the target up” comes for free.
CPU, memory, swap, disk, load, network — from node_exporter, a small program that reads /proc and /sys for you.
Requests per second, status codes, latency — from Caddy, the proxy every request to chess already passes through.
A blackbox probe: a program that requests the site the way a visitor would and reports whether it worked.
Are cicd.service and caddy.service still active? Read from systemd by node_exporter.
The chess app is a static site with no server-side code of its own, so there is nothing inside it to instrument. Everything about it is measured from the outside — at the proxy in front of it, and by a visitor’s-eye probe.
The whole stack costs about 250 MB on top of what the VM already runs — that’s why the config later keeps scrapes at 30 seconds and history at 3 days. Monitoring software that eats the machine it’s watching is worse than none. Disk is the tighter budget: 7 GB total, and every image you pull and every rebuild of chess leaves layers behind.
The VM's own numbers, as plain text over HTTP.
node-exporter is in EPEL, the extra repository from Lecture 3 — so it arrives as an RPM with its own systemd unit, its own unprivileged prometheus user, and SELinux is already happy with it. No downloading tarballs from the internet, no writing a unit by hand.The unit’s EnvironmentFile points here, and ARGS is appended to the command. —web.listen-address=127.0.0.1:9100 keeps the metrics on loopback only — nothing outside the VM needs to read them, and the firewall never has to know. The unit-include regex narrows the systemd collector to the handful of units we care about; the file demands the backslashes doubled twice (once for the shell, once for systemd), which is the only ugly part. Restart the service after editing.
{ … } holds labels — the same metric split by CPU, mode, mountpoint. Labels are how you slice a graph later.Every unit gets one line per possible state, and exactly one of them is 1 — the same information systemctl is-active cicd gives you, but sampled every 30 seconds and kept as history. That’s how you’ll know the webhook listener from Lecture 5 died at 3 a.m. instead of finding out when a push doesn’t deploy.
Collect the numbers, then ask questions of them.
A job is a group of targets that do the same thing; a target is a host:port serving /metrics. Caddy’s admin API on :2019 is already listening on your VM from Lecture 5 — it just wasn’t being read. The full file adds the probe job and Prometheus’s own metrics; it ships in the lab bundle.
network_mode: host — the container shares the VM's network, so 127.0.0.1:9100 inside it is the same 127.0.0.1:9100 node_exporter is listening on. Simplest way for a container to reach services that only listen on loopback.--web.listen-address=127.0.0.1:9090 — with host networking, "publishing a port" no longer applies; bind to loopback yourself so Prometheus isn't exposed to the network.:ro,Z — read-only, and Z tells Podman to relabel the folder for SELinux so the container is allowed to read it. Leave it off and the container cannot read its own config file — "Permission denied", even though the file is world-readable. The next lecture explains why.Then open http://localhost:9090 on your laptop. This is the port-forwarding from Lecture 2 doing something useful: Prometheus stays bound to the VM’s loopback, invisible to the internet, while you reach it through your existing SSH session. Under Status → Targets, every job should read UP. Anything else, and the error column tells you why.
A bare metric name is a query: it returns the latest value of every series with that name. Curly braces after the name (mountpoint=”/” in the third query) filter by label, and ordinary arithmetic works between series. The last line is exactly how “disk used %” on the dashboard is computed — free space as a share of the total, subtracted from 100.
rate(x[2m]) — how fast it grew per second, averaged over the last two minutes. This is the query you want for almost every counter.avg, sum by (label) — collapse many series into one, or one per label value. Line three: CPU used across all cores. Line four: CPU broken down by what it's doing.Nothing inside the app to instrument — so watch it from the two places that see everything.
metrics switches on per-request counting. name matters more than it looks: without it Caddy calls its servers srv0, srv1 and which is which depends on config order — so a metric labelled server="chess" is only possible if you name it. The block matches a server by the port it listens on, which is why chess (:3000) and the webhook listener (:9000) show up as separate series. Apply it with sudo caddy validate then sudo systemctl reload caddy — a reload, not a restart, so chess keeps serving while it happens.Three questions, three queries: how many requests a second, split by status code (a spike of 5xx means the proxy is failing to reach the container), and how slow the slowest 5% of requests are — histogram_quantile reads Caddy’s latency histogram. Everything here also passes through the webhook server, but server=“chess” keeps them apart.
probe_success (1 or 0), probe_duration_seconds, and the TLS certificate's expiry.fail_if_body_not_matches_regexp — "200 OK" isn't enough. A page that loads empty is still an outage, so the probe also demands a <title>.insecure_skip_verify — chess uses Caddy's tls internal, a certificate no browser or tool trusts by default. Skipping verification is fine for a probe hitting its own VM.Host: 178.18.249.160 — Caddy routes by hostname, and the probe connects to 127.0.0.1. With the wrong Host header Caddy has no matching site and answers with an empty 200 — a probe that reports success on a broken site. Replace it with your own VM's IP.Different from the other jobs: Prometheus doesn’t scrape chess. It scrapes the blackbox exporter and tells it, through the target parameter, what to visit. The relabel_configs shuffle the address around — the real target goes into the target parameter, and the scrape itself goes to the exporter on :9115. It reads like plumbing because it is; you write it once.
The dashboard is configuration too — no clicking it together by hand.
GF_… environment variables override Grafana's settings file — the usual way to configure a container.GRAFANA_PASSWORD comes from a .env file next to compose.yml, which compose reads automatically. Never commit it — this dashboard will be reachable from the internet, and the default admin/admin login on a public IP gets found within hours.chmod 600 so only your user can read it. The repo ships .env.example with a placeholder and a .gitignore listing .env — the pattern for every secret you'll handle from here on, the same way the webhook secret in Lecture 5 lived on the server and not in the code. Note the password only applies the first time Grafana creates its database; change it later from inside Grafana.On startup Grafana reads /etc/grafana/provisioning and creates whatever it finds: this file adds the Prometheus data source, and a sibling file points at a folder of dashboard JSON — that’s where VM & Chess comes from, already built. It’s the same idea as everything else this course: configuration lives in files you can version, so a fresh VM comes up identical instead of depending on someone having clicked the right buttons once. uid is the stable name dashboards refer to.
Same recipe as chess, a new port: a tls internal site that reverse-proxies to the local service, plus a hole in firewalld for it (the firewall was never disabled — it’s still on, with exactly the ports you opened). Why a separate port rather than /grafana on the chess site? Because then Grafana’s own page loads would count as “chess traffic” in the request graphs. Giving it its own server keeps each dashboard number honest. That covers the VM’s side; the next slide covers how a browser gets to it.
Same trick as the webhook in Lecture 5. Every VM’s Caddy listens on the same port, 6000, so your Caddyfile and firewall rule are identical to everyone else’s. The gateway maps a per-VM public port onto it: VM 07 is reached on 6007, VM 21 on 6021. In the browser use 60XX, not 6000 — 6000 only exists on the VM’s side of the gateway. Expect a warning for the internal certificate; accept it, and sign in as admin with the password from your .env.
CPU, memory, disk and load as four big numbers that turn orange, then red, as they climb — and below them, the history behind each one.
Probe UP/DOWN, probe latency, requests per second, 5xx count, and certificate days remaining; then requests by status code and p50/p95 latency.
cicd.service and caddy.service up or down, and how many webhook POSTs — deploys triggered — arrived in the last minute.
Open a panel and choose Edit — every panel is one or two PromQL queries, most of them the very ones you typed into the Prometheus UI. You are allowed to break the dashboard: it reloads from the file, and allowUiUpdates lets you experiment without it fighting you.
A dashboard only helps if someone is looking at it.
Grafana runs the query every minute. When the condition is true, the alert goes Pending; only if it stays true for the whole for period does it become Firing. That delay is deliberate: a single failed probe during a deploy is not an incident, five in a row is. Too short a for and you get noise you learn to ignore.
probe_success < 1 for 1 minute. The first alert every website should have.
sum(increase(caddy_http_request_duration_seconds_count{server="chess",code=~"5.."}[5m])) > 0. The proxy is up but the container behind it isn't answering.
node_systemd_unit_state{name="cicd.service",state="active"} < 1. Pushes will quietly stop deploying.
Root filesystem over 85% for 5 minutes. Every rebuild leaves image layers behind — on a 7 GB disk this one is not hypothetical.
for duration.For a rule that uses a “no data” state: if the series doesn’t exist at all (the probe job was deleted), the query returns nothing rather than 0. Set No data handling to Alerting for “is it up” rules so silence counts as a failure, not a pass.
An alert you’ve never seen fire is an alert you’re guessing about. Stop the container, watch the probe go DOWN, the 5xx graph light up, and the rule move through its states — then start it and watch everything go back to Normal. Do it on your own VM, not a classmate’s.