Send Yourself a Telegram when a Systemd-Unit Fails

For CurrySearch we needed a reliable way to tell if everything is running as expected. Along with collectd and central logging, notifications on unit failure help us ensure the reliability of the system and act swiftly in case of disaster.

You may like or dislike systemd but on way or the other, someday, you probably have to deal with it. This blogpost describes a way to get notifications through telegram when a systemd-unit fails.

Telegram is a modern messenger app with some unique features. Among others, it is possible to create bots that interact with users.

Based on an article on the archlinux wiki, which describes notifications via e-mail, we will show you how to make systemd units send their status to you through telegram.

You can find the results on GitHub.

When done everything right, you will be able to:

  1. telegram "message" in any shell script: Your bot will send you “message”
  2. unit-status-telegram systemd-unit-name in any shell script: Your bot will send you the status message of systemd-unit
  3. Place OnFailure=unit-status-telegram@%n.service in the Unit section if any systemd-unit and be notified on its failure in telegram

Getting the ApiKey and Chat-Id

First you need a bot api-key and your chat id: Use @botfather to get an api-key. Next you can message your bot using its username and get the chat-id using from url:

https://api.telegram.org/bot$API_KEY/getUpdates

With the api-key and the chat-id we can now send us a telegram message from commandline:

curl  -s -d "chat_id=$CHAT_ID&disable_web_page_preview=1&text=MESSAGE" https://api.telegram.org/bot$TELEGRAM_KEY/sendMessage

Talk to yourself

A bit of abstraction and cleanup results in the following bash-script.

telegram.sh:

#!/bin/bash
source /etc/telegram/key.sh

URL="https://api.telegram.org/bot$TELEGRAM_KEY/sendMessage"
curl -s -d "chat_id=$CHAT_ID&disable_web_page_preview=1&text=$1" $URL > /dev/null

With key.sh beeing:

export TELEGREM_KEY="API-KEY"
export CHAT_ID=CHAT_ID

Move the telegram.sh into /usr/bin and set the x flag. Now you can simply run

telegram "Message"

to message yourself.

Systemd Unit Status

To message yourself with a systemd unit status use. unit-status-telegram.sh:

#!/bin/bash

UNIT=$1

UNITSTATUS=$(systemctl status $UNIT)
ALERT=$(echo -e "\u26A0")

telegram "$ALERT Unit failed $UNIT $ALERT
Status:
$UNITSTATUS"

Place the script into /usr/bin/, make sure it is executable and test it:

unit-status-telegram systemd-journald

Resulting in something like: Result

The systemd unit

Now we only need a systemd unit, that we can trigger on failure.

unit-status-telegram@.service:

[Unit]
Description=Unit Status Telegram Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/unit-status-telegram %I

Place it under /etc/systemd/system/unit-status-telegram@.serivce.

Now we can finally add

OnFailure=unit-status-telegram@%n.service

to the [Unit] section of any service.