Wegweiser looks at the machine when your monitoring raises an alarm
Prometheus Alertmanager and Grafana can now send their alarms to Wegweiser. On a machine our agent runs on, Wegweiser records what the machine was doing within seconds, and a critical alarm offers a case that finds out why. Zabbix, Checkmk and PRTG are next.
Prometheus Alertmanager and Grafana can now send their alarms to Wegweiser, through a new integration, Monitoring alarms. When one fires about a machine our agent runs on, Wegweiser records the machine's state within seconds. A critical alarm offers a case straight away.
Your monitoring tells you that something is wrong: "MemoryCritical on SRV-DB01, 03:12". The cause is on the machine, in the processes that were running, the event log around 03:12 and the update that ran at 03:00. By the time somebody looks, the spike has usually passed and the process behind it has exited.
What Wegweiser takes from the machine
Within seconds of the alarm, on any machine whose agent is reporting:
- a 10-second live sample of CPU, memory, disk queue and network, with the processes behind each, the same sample the help button takes;
- on Windows, the System and Application events at Warning and worse from the 15 minutes before; on Linux, the journal's last hour;
- the volumes' free space, when the alarm is about a disk;
- the service's configuration and recovery settings, when the alarm names a service.
No model is called for any of it, and nothing on the machine changes. Captures cost no Wegcoins, up to 200 a day across all your sources.
When your monitoring says a host is unreachable and your rules capture that alarm, the row tells you which side failed. If our agent was reporting, the row says "captured" and gives the machine's CPU and memory: the machine is up, and the path from your monitoring to it is what failed. If it wasn't, the row says when our agent last reported.
A case from the alarm
Press Investigate on the alarm, on the machine's page or the client's, and a case starts from the capture. It sees the alarm's own text, quoted as evidence, and the half hour before the alarm. By default a case from an alarm runs at the short Alarm profile, at most 60 Wegcoins, and its conclusion lands in the machine's chat like any case's.
Which machine an alarm is about
An alarm acts only on a machine you've named: one whose ID
you've put in a wegweiser_device label, or a host you've linked once on
the source's page. A host name that happens to match is shown as a
suggestion, and it starts nothing until you link it.
What each alarm does
That is yours to set. Out of the box, a critical alarm captures the machine and offers a case, a high one captures, and warnings and info are recorded. Rules can pick out a source, a client, an alarm by name or a label value, and the first rule that matches decides.
What it leaves alone
Wegweiser writes nothing into your monitoring: no alarm is acknowledged, silenced or closed there. It doesn't call your monitoring server either. The alarms come to Wegweiser, each source with its own token, and Wegweiser keeps only a fingerprint of that token.
Setting it up
Open Integrations > Monitoring alarms, choose the tool and press Add the source. The source's page shows its address, its token once, and how to point the tool at it: a receiver to paste into alertmanager.yml, or the steps for a Webhook contact point in Grafana. Then save the settings under What an alarm does: captures read machines under the authority of whoever saved them last, and until someone has, nothing is taken.
Next come cases that start without a click, with Zabbix and VSA 10's own notification webhook; then Checkmk and PRTG; then a daily report on the machines you choose.