Setting up a Zabbix 7 + TimescaleDB monitoring server in docker

• docker-compose.yml — a file describing the services launched in docker containers:
  • NETWORK section — defines two virtual docker networks:
    • zbx_net_backend — an isolated network for interaction between the postgres-server, zabbix-server, zabbix-web, and zabbix-agent containers;
    • zbx_net_frontend — a network with access to containers from the host OS: for the zabbix-server and zabbix-web containers.
  • SERVICES section — defines the parameters of the running services/containers:
    • postgres-server — the TimescaleDB DBMS service;
    • zabbix-server — the Zabbix server service;
    • zabbix-web — the web interface service for the Zabbix server (under the hood nginx + php-fpm);
    • zabbix-agent — the Zabbix agent service for collecting metrics from the Zabbix server itself.
  • SECRETS section — defines files containing sensitive information that are passed into the postgres-server, zabbix-server, and zabbix-web containers:
    • POSTGRES_USER (file ./env/.POSTGRES_USER) — the username for the Zabbix database in the TimescaleDB DBMS;
    • POSTGRES_PASSWORD (file ./env/.POSTGRES_PASSWORD) — accordingly, the user’s password.
• env — a directory with files containing environment variables whose values are passed into the container at startup:
  • .env_db — parameters for postgres-server;
  • .env_srv — parameters for zabbix-server;
  • .env_web — parameters for zabbix-web;
  • .env_agent — parameters for the local zabbix-agent;
  • .POSTGRES_PASSWORD — the user’s password in the zabbix DB;
  • .POSTGRES_USER — the username in the zabbix DB.
• src — a directory with source files for building the custom zabbix-server container with extended functionality:
  • alertscripts — a directory with files for sending notifications to Telegram:
  • docker-entrypoint.sh — a script for preparing the environment inside the container (unchanged);
  • Dockerfile — a customized image build description file;
  • externalscripts — a directory with a domain delegation monitoring script.

adgroup — the command to create a group;

• --system — marks the group as a system group and applies the established policies to it;
• --gid 1995 — specifies an explicit group identifier (GID) — in this case 1995 (this is the exact GID that Zabbix developers set in their containers);
• zabbix — the name of the group being created;

adduser — the command to create a user

• --system — marks the group as a system group and applies the established policies to it;
• --gecos — allows setting a user description;
• --disabled-password — disables the user’s password (in this case you cannot log into the system with a password under this user);
• --uid 1997 — specifies an explicit user identifier (UID) — in this case 1997 (this is the exact UID that Zabbix developers set in their containers);
• --ingroup zabbix — adds the user to the previously created zabbix group;
• --shell /sbin/nologin — sets nologin as the user’s shell — it’s impossible to log into the system with it;
• --home /opt/zabbix/zabbix_data — sets the user’s home directory, in our case this is the directory with the zabbix service files;
• zabbix — the name of the user being created.

The tr -cd command, using the standard stream redirection mechanism, takes as input Latin letters and digits (the [:alnum:] character class) generated by the /dev/urandom pseudo-device. Then the head command, receiving data via the pipe, uses the -c flag to pick the first 30 characters (bytes). And finally, the resulting pseudo-random string is passed as input to the tee command, run as superuser. tee writes the received data to the specified file (/opt/zabbix/env/.POSTGRES_PASSWORD) and duplicates the output to the standard stream (stdout).

1. If the Zabbix server is deployed on the local machine

Just type the URL in your browser 😁😁😁:

http://127.0.0.1:8080/

2. If the Zabbix server is deployed on a remote server that is on a local or VPN network

In this case, edit the network settings for the zabbix-server and zabbix-web services in the docker-compose.yml file.

For example, my server’s internal address on the VPN network is: 192.168.122.24. Let’s edit docker-compose.yml:

sudo vim /opt/zabbix/docker-compose.yml

zabbix-server:

zabbix-web:

Let’s restart all containers:

sudo docker compose -f /opt/zabbix/docker-compose.yml down

sudo docker compose -f /opt/zabbix/docker-compose.yml up -d

sudo docker ps

On a client connected to the VPN network, I open the following URL in the browser:

http://192.168.122.24:8080/

3. If the Zabbix server is deployed on a remote server and is not connected to a local or VPN network

In this case you can forward an SSH port or open direct access, or access via a reverse proxy such as nginx. I’ll only cover the first case, as the simplest. Also because the other two are beyond the scope of this article 😜.

On the client, open a terminal and run:

ssh -L 8080:127.0.0.1:8080 user@example.com

Where (in order):

• -L — the local port forwarding flag;
• 8080 — the port that the client machine will listen on and forward to the server’s port 8080;
• 127.0.0.1 — the address of the server on which it listens on the port;
• 8080 — accordingly, the port we’re forwarding to;
• user@example.com — the user and address of the SSH server.

In my example the command is:

Now open a browser and go to: http://127.0.0.1:8080/

Access to the web interface will be available while the SSH session is open.

I won’t go into much detail on Zabbix configuration, since that’s a topic for more than one article 🤯. I’ll just show how to change the interface language, create a new user with admin rights, disable the default Admin, and configure a local Zabbix agent to monitor the Zabbix server itself.

All the steps are shown in images and don’t really need comments 😏.

Changing the language

Creating a new user

And log back into the interface as the new user:

Disabling the default Admin user

Connecting the local Zabbix agent

This agent was set up by us at the moment we launched the whole installation and is available under the domain name zabbix-agent inside the container network.

Here, make sure to select “Connect to” — DNS:

Now go to the “Monitoring” — “Problems” page. After a few minutes, the issue with the local zabbix-agent availability should resolve:

At the beginning of the article we created a zabbix service user. Let’s use it to safely run the Zabbix stack services.

Configuring sudoers to run Zabbix as the service user

I recently examined the intricacies of the privilege escalation mechanism in Linux. For a better understanding of the following steps, I recommend reading: Linux command line, privilege escalation: su, sudo commands 😌.

Let’s create a new file describing restricted permissions for the zabbix user:

sudo visudo -f /etc/sudoers.d/zabbix

And fill it with:

Cmnd_Alias ZBX = \
    /usr/bin/docker compose -f /opt/zabbix/docker-compose.yml up, \
    /usr/bin/docker compose -f /opt/zabbix/docker-compose.yml down

zabbix ALL = (:docker) NOPASSWD: ZBX

We allow the zabbix user to start and stop the services described in our docker-compose.yml, as the privileged docker group, via sudo.

Save the file and close the editor.

Creating the systemd service file

Open it for editing:

sudo vim /etc/systemd/system/zabbix.service

And fill it in:

[Unit]
Description=Zabbix service
Requires=docker.service
After=docker.service

[Service]
Restart=always
RestartSec=5
User=zabbix
ExecStart=/usr/bin/sudo --group=docker /usr/bin/docker compose -f /opt/zabbix/docker-compose.yml up
ExecStop=/usr/bin/sudo --group=docker /usr/bin/docker compose -f /opt/zabbix/docker-compose.yml down

[Install]
WantedBy=multi-user.target

Save, close. After that, stop the running containers, reload the systemd configuration, enable autostart of the zabbix service, and start it:

sudo docker compose -f /opt/zabbix/docker-compose.yml down

sudo systemctl enable --now zabbix

sudo systemctl status zabbix

Let’s check the containers’ status:

sudo docker ps

Everything is running. Good stuff 🙂.

Since we deployed Zabbix in docker, it’s logical to set up the Zabbix agents the same way.

Obviously, this requires an installed and running Docker engine.

Downloading the project files and creating a service user

Let’s clone the repository with the files:

sudo git clone https://github.com/r4ven-me/zabbix-agent /opt/zabbix-agent

As with the server, we create a service user to run the Zabbix agent:

sudo addgroup --system --gid 1995 zabbix

sudo adduser --system --gecos "Zabbix agent" \
    --disabled-password --uid 1997 --ingroup zabbix  \
    --shell /sbin/nologin --home /opt/zabbix-agent/data zabbix

Let’s edit the .env file — the file with the Zabbix agent’s parameters.

sudo vim /opt/zabbix-agent/.env

Here you must set the existing IP address of the host on which the Zabbix agent service will wait for connections. Also specify the agent’s hostname and the Zabbix server’s address.

In my example, the data is:

• Zabbix server address: 192.168.122.24
• Zabbix agent address: 192.168.122.105
• Zabbix agent hostname: zabbix-agent.example.com

See here for a description of all the parameters.

Now, just in case, let’s check the finished sudoers file and if everything is fine — copy it to /etc/sudoers.d/:

sudo visudo -c -f /opt/zabbix-agent/zabbix_agent

sudo cp /opt/zabbix-agent/zabbix_agent /etc/sudoers.d/

Next, copy the systemd service file, reload the configuration, and start zabbix-agent:

sudo cp /opt/zabbix-agent/zabbix-agent.service /etc/systemd/system/

sudo systemctl daemon-reload

sudo systemctl enable --now zabbix-agent

Keep in mind that on the first start, docker will pull the image from the registry.

You can check the agent’s status with these commands:

sudo systemctl status zabbix-agent

sudo journalctl -fu zabbix-agent

sudo docker ps

The Zabbix agent is running.

A nuance. If you have many external partitions mounted on your server, you need to add them to the volume block in docker-compose.yml, so the Zabbix agent detects them.

Adding the Zabbix agent to the Zabbix server

Here it’s almost all in images 😇.

Go to “Monitoring” — “Hosts” and after some time:

Hooray! 🥳🥳🥳

And now for something interesting 🧐. To implement the ability to send alert notifications to Telegram with graphs, I rebuilt the zabbix-server container, adding a Python interpreter, a special venv, and files from the GitHub repository of developer xxsokolov — this method is marked on the official Zabbix website as valid 👌.

In the instructions below, the option of using my image is covered: r4venme/zabbix-server-pgsql:alpine-7.0.3, but you can also build your own. You’ll find all the build files in the repository downloaded at the start of the article, in the src folder.

Let’s start the configuration.

Creating the alert type (web interface)

• Name (arbitrary): Alerter
• Type: Script
• Script name: zbxTelegram.py
• Script parameters:
  • {ALERT.SENDTO}
  • {ALERT.SUBJECT}
  • {ALERT.MESSAGE}

Next, go to the templates tab and add 2: for problems and for recovery:

Problem template:

• Message type: Problem
• Subject: {Problem} Problem ({TRIGGER.SEVERITY})
• Message:

<?xml version="1.0" encoding="UTF-8" ?>
<root>
    <body>
        <messages>
<![CDATA[
Problem: {EVENT.NAME}

Host: {HOST.NAME} ({HOST.IP})
Start: {EVENT.TIME} {EVENT.DATE}

Last value: {ITEM.LASTVALUE1}

###########################
]]>
        </messages>
    </body>
    <settings>
        <graphs>True</graphs>
        <hostlinks>True</hostlinks>
        <graphlinks>False</graphlinks>
        <acklinks>True</acklinks>
        <eventlinks>True</eventlinks>
        <triggerlinks>True</triggerlinks>
        <eventtag>False</eventtag>
        <eventidtag>False</eventidtag>
        <itemidtag>False</itemidtag>
        <triggeridtag>False</triggeridtag>
        <actionidtag>False</actionidtag>
        <hostidtag>False</hostidtag>
        <zntsettingstag>True</zntsettingstag>
        <zntmentions>True</zntmentions>
        <keyboard>True</keyboard>
        <graphs_period>default</graphs_period>
        <host>{HOST.HOST}</host>
        <itemid>{ITEM.ID1} {ITEM.ID2} {ITEM.ID3} {ITEM.ID4}</itemid>
        <triggerid>{TRIGGER.ID}</triggerid>
        <eventid>{EVENT.ID}</eventid>
        <actionid>{ACTION.ID}</actionid>
        <hostid>{HOST.ID}</hostid>
        <title><![CDATA[{HOST.NAME} - {EVENT.NAME}]]></title>
        <triggerurl><![CDATA[{TRIGGER.URL}]]></triggerurl>
        <eventtags><![CDATA[{EVENT.TAGS}]]></eventtags>
    </settings>
</root>

After filling it in, click “Add”:

Then “Add” again for the next one:

Recovery template:

• Message type: Problem recovery
• Subject: {Resolved} Recovery
• Message:

<?xml version="1.0" encoding="UTF-8" ?>
<root>
    <body>
        <messages>
<![CDATA[
Problem: {EVENT.RECOVERY.NAME}

Host: {HOST.NAME} ({HOST.IP})
Duration: {EVENT.DURATION} 
Recovery time: {EVENT.RECOVERY.TIME} {EVENT.RECOVERY.DATE}

Last value: {ITEM.LASTVALUE1}

###########################
]]>
        </messages>
    </body>
    <settings>
        <graphs>True</graphs>
        <hostlinks>True</hostlinks>
        <graphlinks>False</graphlinks>
        <acklinks>True</acklinks>
        <eventlinks>True</eventlinks>
        <triggerlinks>True</triggerlinks>
        <eventtag>False</eventtag>
        <eventidtag>False</eventidtag>
        <itemidtag>False</itemidtag>
        <triggeridtag>False</triggeridtag>
        <actionidtag>False</actionidtag>
        <hostidtag>False</hostidtag>
        <zntsettingstag>True</zntsettingstag>
        <zntmentions>True</zntmentions>
        <keyboard>True</keyboard>
        <graphs_period>default</graphs_period>
        <host>{HOST.HOST}</host>
        <itemid>{ITEM.ID1} {ITEM.ID2} {ITEM.ID3} {ITEM.ID4}</itemid>
        <triggerid>{TRIGGER.ID}</triggerid>
        <eventid>{EVENT.ID}</eventid>
        <actionid>{ACTION.ID}</actionid>
        <hostid>{HOST.ID}</hostid>
        <title><![CDATA[{HOST.NAME} - {EVENT.NAME}]]></title>
        <triggerurl><![CDATA[{TRIGGER.URL}]]></triggerurl>
        <eventtags><![CDATA[{EVENT.TAGS}]]></eventtags>
    </settings>
</root>

Next, “Add”:

And at the end, once more here:

Should turn out like this:

Creating a service user (web interface)

Now let’s create a service user for sending notifications:

Fill in all the necessary fields:

The theme you choose determines the background of the graphs in notifications.

After that, go to the “Media” tab and in the “Send to” field specify the Telegram chat or group ID (instructions from the official Zabbix site on how to get the ID):

Next, go to “Permissions”. Ideally, here you should first set up restricted user groups and roles. But this is a delicate process, and it’s also beyond the scope of the article. As a quick solution, I grant the service user “super admin” rights, since the script needs read permissions for hosts and templates. It’s up to you.

And at the end, click the main “Add”:

Configuring alert conditions (web interface)

Go to “Trigger actions” and create a new one:

Set a name and add a condition. In the condition, specify 2 templates, as in the screenshot:

Next, go to the “Operations” tab and there add 2 actions: for “Operations” and “Recovery operations”:

Specify the recipient — our service user, for whom we set the group/chat ID in Telegram. Here we also specify the sending method: the one we created earlier:

Roughly like this:

Editing docker-compose.yml (server console)

Let’s go back to the server and open for editing:

sudo vim /opt/zabbix/docker-compose.yml

Let’s edit the zabbix-server service, the image block. Comment out the current line, and uncomment the line with the address of the image I built:

# image: zabbix/zabbix-server-pgsql:alpine-7.0.3
image: r4venme/zabbix-server-pgsql:alpine-7.0.3

Save and restart zabbix-server:

sudo systemctl restart zabbix

sudo journalctl -fu zabbix

sudo docker ps

After starting, create the alert scripts folder, copy the config file for the Telegram notification script from the zabbix-server docker container into it, and change its owner:

sudo mkdir -p /opt/zabbix/zabbix_data/server/alertscripts/

sudo docker cp zabbix-server:/usr/lib/zabbix/alertscripts/zbxTelegram_config.py \
    /opt/zabbix/zabbix_data/server/alertscripts/

sudo chown -R zabbix:zabbix /opt/zabbix/zabbix_data/server/alertscripts/

Open docker-compose.yml again:

sudo vim /opt/zabbix/docker-compose.yml

For the zabbix-server service, in the volume block, uncomment this line:

- ./zabbix_data/server/alertscripts/zbxTelegram_config.py:/usr/lib/zabbix/alertscripts/zbxTelegram_config.py

Don’t forget to save.

Configuring the sending (server console)

Just a little bit left 🥺.

Open the copied config file for editing:

sudo vim /opt/zabbix/zabbix_data/server/alertscripts/zbxTelegram_config.py

And specify your bot’s token (how to get it) in the tg_token line inside single quotes:

Scroll the file down and also specify the service user and password we created earlier:

zabbix_api_url is the internal network address of the container. Don’t touch it if you haven’t changed it.

Recreate the zabbix-server container:

sudo docker compose -f /opt/zabbix/docker-compose.yml down zabbix-server

sudo docker compose -f /opt/zabbix/docker-compose.yml up -d zabbix-server

Verification (server console)

Let’s check message sending from the command line:

sudo docker exec -it zabbix-server \
    /usr/lib/zabbix/alertscripts/zbxTelegram.py 1234567890 test test

Where 1234567890 is the chat or group ID in Telegram.

It works 👍.

Testing in practice

If you connected the local zabbix-agent in the initial setup step, then to test it just stop it with the command:

sudo docker compose -f /opt/zabbix/docker-compose.yml stop zabbix-agent

Go to the web interface: “Monitoring” — “Problems”. After a few minutes, the agent will get the “Unavailable” status, a problem will appear on the corresponding web panel, and you should get a notification in Telegram.

The message arrived, great. Now let’s start the agent:

sudo docker compose -f /opt/zabbix/docker-compose.yml start zabbix-agent

After a few minutes it will recover in the web panel and a corresponding notification will arrive in Telegram:

Lovely 🤤.