Send Yourself a Telegram when a Systemd-Unit Fails
For CurrySearch we needed a reliable way to tell if everything is running as expected. Along with collectd and central logging, notifications on unit failure help us ensure the reliability of the system and act swiftly in case of disaster.
You may like or dislike systemd but on way or the other, someday, you probably have to deal with it. This blogpost describes a way to get notifications through telegram when a systemd-unit fails.
Telegram is a modern messenger app with some unique features. Among others, it is possible to create bots that interact with users.
Based on an article on the archlinux wiki, which describes notifications via e-mail, we will show you how to make systemd units send their status to you through telegram.
You can find the results on GitHub.
When done everything right, you will be able to:
telegram "message"in any shell script: Your bot will send you “message”
unit-status-telegram systemd-unit-namein any shell script: Your bot will send you the status message of systemd-unit
Unitsection if any systemd-unit and be notified on its failure in telegram
Getting the ApiKey and Chat-Id
First you need a bot api-key and your chat id: Use @botfather to get an api-key. Next you can message your bot using its username and get the chat-id using from url:
With the api-key and the chat-id we can now send us a telegram message from commandline:
curl -s -d "chat_id=$CHAT_ID&disable_web_page_preview=1&text=MESSAGE" https://api.telegram.org/bot$TELEGRAM_KEY/sendMessage
Talk to yourself
A bit of abstraction and cleanup results in the following bash-script.
#!/bin/bash source /etc/telegram/key.sh URL="https://api.telegram.org/bot$TELEGRAM_KEY/sendMessage" curl -s -d "chat_id=$CHAT_ID&disable_web_page_preview=1&text=$1" $URL > /dev/null
With key.sh beeing:
export TELEGREM_KEY="API-KEY" export CHAT_ID=CHAT_ID
Move the telegram.sh into /usr/bin and set the x flag. Now you can simply run
to message yourself.
Systemd Unit Status
To message yourself with a systemd unit status use. unit-status-telegram.sh:
#!/bin/bash UNIT=$1 UNITSTATUS=$(systemctl status $UNIT) ALERT=$(echo -e "\u26A0") telegram "$ALERT Unit failed $UNIT $ALERT Status: $UNITSTATUS"
Place the script into /usr/bin/, make sure it is executable and test it:
Resulting in something like:
The systemd unit
Now we only need a systemd unit, that we can trigger on failure.
[Unit] Description=Unit Status Telegram Service After=network.target [Service] Type=simple ExecStart=/usr/bin/unit-status-telegram %I
Place it under /etc/systemd/system/unit-status-telegram@.serivce.
Now we can finally add
to the [Unit] section of any service.