Lab 21: I Finally Closed the Detection Gap
Eight labs ago, I ran a brute force attack against my own system and watched zero alerts fire.
I knew the attack happened. The logs confirmed it. But the SIEM never saw it — because the logs that mattered weren't making it into Elastic. I documented the gap, flagged it for remediation, and kept building.
That gap followed me through Labs 14, 15, 16, 17, 18, 19, and 20.
Lab 21 is where it finally gets closed.
What Was Actually Broken
The root cause turned out to be two configuration issues working together to create a complete blind spot.
Kali Linux doesn't install rsyslog by default. Without it, systemd-journald — the logging backend — has no syslog consumer to forward logs to. And without a syslog consumer, no /var/log/auth.log file gets created.
That's the first problem.
The second problem: ForwardToSyslog in /etc/systemd/journald.conf was set to no by default. Even if rsyslog had been installed, forwarding still wouldn't have happened without explicitly enabling it.
The combined effect: SSH authentication failures were being captured in the systemd journal — they existed, they were real, they were verifiable — but they never made it to a file that Elastic Agent could read. Elastic Agent's System integration monitors file-based log sources. No file, no ingestion. No ingestion, no alerts.
The detection rule wasn't broken. The pipeline was.
The Remediation
The fix required two steps: enable syslog forwarding in journald, then install rsyslog to receive it.
Step 1 — Enable ForwardToSyslog
sudo nano /etc/systemd/journald.conf
Changed: ForwardToSyslog=yes
sudo systemctl restart systemd-journald
Step 2 — Install rsyslog
sudo apt install rsyslog -y
Step 3 — Confirm auth.log exists
ls /var/log/auth.log
Output: /var/log/auth.log
That output — /var/log/auth.log — was the moment I knew the fix had worked. The file existed. The pipeline had somewhere to write.
Validating End-to-End Detection
A fix that hasn't been tested is a fix you can't trust.
I ran the same brute force simulation that had produced zero alerts in Lab 13 — ten consecutive failed SSH authentication attempts against the local SSH service using an invalid username:
for i in {1..10}; do ssh -o ConnectTimeout=5 invalid_user@127.0.0.1; done
Then I verified the attack was writing to auth.log:
sudo grep "invalid_user" /var/log/auth.log
Confirmed. Then I went to Kibana Discover and queried for ingested events from the Kali host.
61 documents returned.
Then I checked the Alerts dashboard.
31 alerts fired.
Rule: SSH Authentication Failure Detected. Host: kali. Severity: Medium. Risk Score: 47.
The full pipeline — from attack activity on the host, through rsyslog, into /var/log/auth.log, picked up by Elastic Agent, ingested into logs-system.auth-*, evaluated against the detection rule, and surfaced as an alert in Kibana — was working end-to-end.
Eight labs. One misconfiguration. Thirty-one alerts.
What This Actually Means
There's a lesson here that goes beyond rsyslog configuration.
The detection gap was invisible. The rules didn't error. The agent didn't report a problem. The SIEM dashboard looked normal. Nothing screamed that something was wrong — because a missing log source doesn't generate noise. It generates silence.
And silence in a SOC is dangerous, because silence can look like safety.
In a real environment, this kind of gap could persist for months without anyone noticing. Attackers targeting SSH on Linux hosts would have gone completely undetected. No alerts, no investigation, no response — not because the security team wasn't watching, but because the SIEM was watching the wrong data stream.
The analysts who catch gaps like this aren't the ones who trust the dashboard. They're the ones who run simulations, verify log sources, and treat silence as a signal worth investigating.
That's the work.
The Detection Chain — Fully Validated Step Status SSH brute force executed ✅ Confirmed Events written to auth.log ✅ Confirmed Logs ingested to Elastic ✅ 61 documents Detection rule triggered ✅ Yes Alerts generated ✅ 31 alerts What's Next
The Elastic SIEM phase of the lab portfolio is complete.
Next up: Splunk. Same detection logic, different platform — because real detection engineering isn't about memorizing tool syntax. It's about understanding the pattern underneath it. Platform-agnostic. Threat-focused.
Labs 22–25 build out Splunk detection coverage from scratch. The gap that took eight labs to close in Elastic? We're not making the same mistake twice.
Full lab report and documentation on GitHub: github.com/RouteToRoot
Follow the journey: GitHub: github.com/RouteToRoot YouTube: youtube.com/@RouteToRoot_Sec Portfolio: routetoroot.io
