What only goes wrong the first time

Onboarding a client is the first thing Wegweiser promises, and the first day of a new machine was the part of it nobody was testing. Every agent test ran on a machine that had had the agent before, so anything that only goes wrong once, on the very first run, went wrong quietly and stayed wrong.

There is now a test that installs the agent on a machine that has never had it: a disposable clean Windows Server 2022, Ubuntu 24.04 or Rocky 9, enrolled with the exact command a group page hands a customer and watched until every kind of reading has arrived. Its first morning found five faults, one of them five months old.

A first day of event logs, scored on nothing

The Windows collector writes each log to a folder that only got created after a successful collection, which on a fresh install never happened. So the first run failed on all three channels, posted three empty summaries anyway, uploaded no files, and logged itself as done. The analyses then scored the silence: Application 100 and "nothing to assess", System 95, Security 65, on logs nobody had read. Eleven of the thirteen Windows machines enrolled since 14 May started that way.

Worse than one wrong day: the cadence gate, which skips a reading when a recent one of the same type exists, then held the real summary back behind the empty one, and on a monthly tenant held every summary after it too. One machine read "nothing to assess" for all three logs from 4 September to today, with 22 real summaries per log queued behind a verdict about nothing.

The agent creates the folder, posts nothing for a channel it collected nothing from, and treats an unreadable channel as an error rather than as an empty log. The server drops an empty summary for a log a machine has never uploaded, which covers every agent already in the field, and a summary with events in it after an analysis of an empty one is now a material change, so the real reading goes through.

The first few minutes

• The Windows install command printed wgT. A translation change on 9 September left a JavaScript call inside the PowerShell text, so every command copied from a group page or the dashboard carried it, and the elevation check failed on it instead of telling the operator to run as administrator.
• A clock more than a minute out meant silence. Enrolment is not time checked, so such a machine enrolled and then had every request refused, for ever, saying so nowhere. The agent now learns the server's time from its responses over TLS and signs with that when the two disagree.
• A first failed fetch left the service idle. If the network was not up at boot, or a deploy was restarting the server, the agent stayed "Running" and did nothing until something restarted it. It now retries.
• Plans arrived without a platform. A new machine asks for its collection plans before its first heartbeat, and the platform came only from the heartbeat, so a new Windows machine got the cross-platform plans and waited five minutes for its event logs, security posture and the rest. The platform now comes from what the machine registered as.

Threat scans that reach the end of the file list

The scanner was passing its rule engine a 60 second timeout that our own comments called a per-file cap. It is a budget for the whole run: any shard with more than a minute of work stopped there and the rest of its file list was never looked at. About half of all deep scans since 19 September lost at least one shard, and 36 of 256 daily scans in a week ran into the cut, on servers and desktops alike.

Removing the cut showed what it had been hiding: uncut, the scanner prints every occurrence of every match, and on a 1 GB machine that got the agent killed by the kernel. Matches per pattern are now capped where no rule in the catalogue counts (peak memory 633 MB down to 162 MB, with identical detections), and a deep shard gets an hour to finish rather than ten minutes while staying at a quarter of the machine, one shard at a time. Agents from 0.3.107 scan uncut; until a machine has that build it behaves as it did before. If an onboarding scan of yours lost shards, it did not see those files, and the next full scan covers them.

Backup health, with something to read

Backup health was declared, billed, on by default and mapped to eleven compliance controls, and had never produced a single row, because nothing collected its input: every settings page was offering a check that did not exist. It now reads what the machine itself recorded: Windows Server Backup's own event channel and summary, Veeam Agent and Veeam B&R rated by event level rather than message text so a German Windows reads the same, and on Linux every backup run by a systemd timer, whatever the tool, with the tool's own last lines as the reason a run failed. Products that cannot be read this way, Datto, Acronis, Macrium, an rsnapshot line in cron, are named and never scored, and a host with no readable job gets no row at all: unseen is not unprotected. Each job is rated against its own schedule, and nothing is rated that an endpoint cannot show.

Reports, and the numbers in them

A group is how you say "site" or "department", it is the level a client asks about, and it was the one level with no report. It has one now, on the same kit as the others, with the weekly fleet synthesis that no other report carries, plus a device inventory as CSV.

Reading the first one turned up three faults that were in the device, organisation, tenant and QBR reports as well: every device said "Last seen: Never", every week of the health trend said "0 healthy, 0 at risk", and the first week printed a dash instead of an opening reading. All three are fixed, the missing times are backfilled, and a long value no longer pushes Health and Last seen off the edge of an inventory. The website security report now opens on one table of every open finding, and the duplicated CVE lists that inflated a low finding's severity are gone, which takes it from thirteen pages to ten with no blank ones.

A website that only looks fixed

A completed scan that looked at less than the last one reads exactly like a client's website getting better. That happened on 24 September, and four of one site's six findings were recorded as fixed. One was a private IP address still published on a blog.

When at least three findings disappear at once and they are at least half of what was open, they are now held open and shown as "Not seen, unconfirmed" on the card, in the chat and in the PDF, and the next completed scan decides. One or two going still closes them straight away, so an ordinary fix is not made to wait.

A scan can now also say for itself that it saw less. Each one reports what it covered: pages mapped, active tests run, how long it took, whether it ran logged out, and the scanner's own caveats, such as a firewall refusing four in ten of its requests. When the scanner says coverage was reduced, or a scan mapped under half the pages the one before it did, nothing closes on its word, not even what the previous scan held, and the reason is printed beside every "not seen". The card and the client report show that coverage next to the grade, so a grade is read against what was actually tested.

A finding shows where it was found

Every website finding now carries what the scanner knows about it beyond its name. Filed under gives its CWE and the OWASP Top 10 category and testing guide entry the scanner's rule was filed under by its own author. How it was found says whether a test attack proved it or it was only observed, with the scanner's confidence, and marks it when the problem is site-wide. Where, exactly lists the pages it turned up on, with the method, the parameter, what was sent and what came back, and Read more links the rule's references. The client report prints the same for each finding, and the chat can say where a finding is and how it is classified.

What was sent and what came back are copied from the client's website, so they are treated as its text rather than ours: capped, escaped wherever they are shown, and kept out of the chat. The detail fills in on each website's next scan.

Also

Cases gained three read-only diagnostics, all three asked for by the crash case in yesterday's post: per-process GDI and USER object counts with their peaks, the DLLs loaded in a running process, and UTF-16 text files, which is how Windows writes its crash reports.

And deleting a machine whose agent was still running could stop every other machine's liveness from being recorded for two and a half minutes, long enough for a client's whole fleet to show Connection Lost for the last half minute of it. A heartbeat for a machine that no longer exists is now dropped before the rest are written.