Raven's Blog | Linux and IT
Arkeep - Centralized Backup Server with Agents, Based on Restic and Rclone

Arkeep - Centralized Backup Server with Agents, Based on Restic and Rclone

20 minute
4239 words
Categories:
Data storage
Tags:
#arkeep #restic #rclone #backup #docker #own-infrastructure #open-source #linux #guide
Level: Medium
Author: Ivan Cherniy
Greetings!

In this article, we will look at installing and configuring the Arkeep backup system using Docker Compose.

Preface

Arkeep is an open source centralized backup system built on top of such excellent tools as Restic and Rclone.

There are many different backup tools that use these utilities. But none of them are “fully centralized”. This is quite a relevant problem in the open source world.

And as programmer philosophy says: if the tool you need does not exist, write it yourself. Which is exactly what the Arkeep developer did. Screenshot from Reddit:

Arkeep has an agent-server architecture. That is, you configure the Arkeep server, then install arkeep-agent on the hosts you need and connect them to the server. After that, you can manage backup configuration for these hosts. As storage, you can use anything that Restic and Rclone can work with.

Backup scheme with Arkeep

In this article, we will look at configuring backups for Linux hosts running Debian 13 with Arkeep (Restic) to an S3 object storage based on RustFS.

We will organize access to all resources through the Angie reverse proxy server. We will connect to Web GUI panels over HTTPS, so we will configure automatic TLS certificate issuance from Let’s Encrypt.

Below is a diagram of the future infrastructure:

There is one important thing to understand here:

We will configure interaction between agents and the Arkeep server through gRPC proxying using the grpc_proxy directive. Agent connections to the proxy server will use HTTPS, while from Angie to the Arkeep server it will already be plain HTTP+gRPC.

Input data

Software used in the article:

SoftwareVersion
Debian13
Docker Engine28
Angie1.9.1
Arkeep0.4.5
RustFS1.0.0-beta.8

Domain scheme:

Infrastructure preparation

Installing the Arkeep backup server

Connect to the arkeep.r4ven.me server using SSH and switch to the root account using sudo:

BASH
ssh ivan@arkeep.r4ven.me

sudo -i
Click to expand and view more

Create a shared Docker network:

BASH
docker network create --opt com.docker.network.bridge.name=br-arkeep --opt com.docker.network.enable_ipv6=false --driver bridge --subnet 172.20.20.0/24 --gateway 172.20.20.1 arkeep_net
Click to expand and view more

Create the Arkeep project directory:

BASH
install -m 700 -d /etc/compose

mkdir -vp /etc/compose/arkeep && cd /etc/compose/arkeep
Click to expand and view more

Now create a compose file, for example, using the vim editor:

BASH
vim ./compose.yaml
Click to expand and view more

Fill it in:

YAML
---
# https://github.com/arkeep-io/arkeep/tree/main/deploy/docker

networks:
  arkeep_net:
    external: true

volumes:
  arkeep_server:
    name: arkeep_server
  arkeep_agent:
    name: arkeep_agent

services:
  arkeep_server:
    image: ghcr.io/arkeep-io/arkeep-server:v0.4.5
    container_name: arkeep-server
    restart: unless-stopped
    stop_grace_period: 1m
    cpus: 1
    mem_limit: 1G
    hostname: arkeep-server
    env_file: ./.env
    healthcheck:
      test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:8080/health/ready"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s
    expose:
      - "8080"   # REST API + GUI
      - "9090"   # gRPC (agents)
    volumes:
      - arkeep_server:/var/lib/arkeep/
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    networks:
      - arkeep_net

  arkeep_agent:
    image: ghcr.io/arkeep-io/arkeep-agent:v0.4.5
    depends_on:
      arkeep_server:
        condition: service_healthy
    container_name: arkeep-agent
    restart: unless-stopped
    stop_grace_period: 1m
    cpus: 1
    mem_limit: 1G
    hostname: arkeep-agent
    environment:
      TZ: "${TZ:-Europe/Moscow}"
      ARKEEP_SERVER_ADDR: "arkeep-server:9090"
      ARKEEP_AGENT_SECRET: "${ARKEEP_AGENT_SECRET}"
      ARKEEP_STATE_DIR: "/var/lib/arkeep-agent"
      ARKEEP_LOG_LEVEL: "${ARKEEP_LOG_LEVEL:-info}"
      ARKEEP_GRPC_INSECURE: "${ARKEEP_GRPC_INSECURE:-false}"
    volumes:
      - arkeep_agent:/var/lib/arkeep-agent/
      - /:/hostfs/:ro # :ro for backup-only; change to :rw to also restore to local paths
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/lib/docker/volumes/:/var/lib/docker/volumes/:ro
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    networks:
      - arkeep_net
Click to expand and view more

It is worth separately mentioning mounting the root of the host filesystem: /:/hostfs/:ro. Mounting in read-only mode (ro) allows backing up any point in the system. If you also want to be able to restore files to their original directories, in that case you need to mount in rw mode (just remove ro). And since the Arkeep agent in our case will run as the root user, this implies full agent access to the system files. Keep that in mind. Let’s continue.

Create the env file:

BASH
vim ./.env
Click to expand and view more

Fill it in:

INI
# https://github.com/arkeep-io/arkeep/tree/main/deploy/docker

TZ=Europe/Moscow
ARKEEP_SECRET_KEY=
ARKEEP_AGENT_SECRET=
ARKEEP_DATA_DIR=/var/lib/arkeep/data
ARKEEP_DB_DRIVER=sqlite
ARKEEP_DB_DSN=/var/lib/arkeep/arkeep.db
ARKEEP_LOG_LEVEL=info
ARKEEP_SERVER_ADDR=arkeep-grpc.r4ven.me:443
ARKEEP_SERVER_HTTP_ADDR=https://arkeep.r4ven.me
ARKEEP_GRPC_INSECURE=true
ARKEEP_TELEMETRY=true
Click to expand and view more

Parameter description:

Generate secrets of the recommended length:

BASH
sed -iE "s/^ARKEEP_SECRET_KEY=.*/ARKEEP_SECRET_KEY=$(openssl rand -hex 32)/" ./.env

sed -iE "s/^ARKEEP_AGENT_SECRET=.*/ARKEEP_AGENT_SECRET=$(openssl rand -hex 24)/" ./.env
Click to expand and view more

Start the stack:

BASH
docker compose up -d && docker compose logs -f
Click to expand and view more

Check:

BASH
docker compose ps

docker compose volumes
Click to expand and view more

If everything is fine, we will see roughly this output:

Let’s configure autostart with Systemd right away:

BASH
cat > ./arkeep.service << EOF
[Unit]
Description=Arkeep service
Requires=docker.service
After=docker.service

[Service]
Restart=always
RestartSec=5
User=root
Group=root
WorkingDirectory=$(realpath ./)
ExecStart=/usr/bin/docker compose up --remove-orphans
ExecStop=/usr/bin/docker compose down

[Install]
WantedBy=multi-user.target
EOF
Click to expand and view more
BASH
chmod 600 ./arkeep.service

ln -s $(realpath ./arkeep.service) /etc/systemd/system

systemctl daemon-reload

systemctl enable --now arkeep

systemctl status arkeep
Click to expand and view more

Now the arkeep service can be managed with the systemctl utility:

BASH
systemctl stop arkeep
systemctl start arkeep
systemctl restart arkeep
systemctl status arkeep
Click to expand and view more

Installing the RustFS S3 storage

RustFS is an S3-compatible storage, whose installation, configuration, and usage we covered in the previous article.

Create the project directory:

BASH
mkdir -vp /etc/compose/rustfs && cd /etc/compose/rustfs
Click to expand and view more

Now create the compose file:

BASH
vim ./compose.yaml
Click to expand and view more

Fill it in:

YAML
---
# https://github.com/rustfs/rustfs

networks:
  arkeep_net:
    external: true

volumes:
  rustfs:
    name: rustfs

services:
  rustfs:
    image: rustfs/rustfs:1.0.0-beta.8
    container_name: rustfs
    restart: unless-stopped
    stop_grace_period: 30s
    cpus: 1
    mem_limit: 1G
    hostname: rustfs
    env_file: ./.env
    healthcheck:
      test:
        ["CMD", "sh", "-c", "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    expose:
      - "9000" # S3 API port
      - "9001" # Console port
    volumes:
      - rustfs:/data/
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    networks:
      - arkeep_net
Click to expand and view more

Create the env file:

BASH
vim ./.env
Click to expand and view more

Fill it in:

INI
# https://docs.rustfs.com/installation/docker/#complete-parameter-configuration-example

TZ=Europe/Moscow
# RUSTFS_VOLUMES=/data/rustfs{0..3} # Define 4 storage volumes
RUSTFS_ADDRESS=0.0.0.0:9000
RUSTFS_CONSOLE_ENABLE=true
RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
RUSTFS_CORS_ALLOWED_ORIGINS=https://rustfs.r4ven.me
RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=https://rustfs.r4ven.me
RUSTFS_ACCESS_KEY=
RUSTFS_SECRET_KEY=
RUSTFS_OBS_LOGGER_LEVEL=info
Click to expand and view more

Where:

Set the “login” (replace it with your own) and generate the “password”:

BASH
sed -iE "s/^RUSTFS_ACCESS_KEY=.*/RUSTFS_ACCESS_KEY=ivan/" ./.env

sed -iE "s/^RUSTFS_SECRET_KEY=.*/RUSTFS_SECRET_KEY=$(openssl rand -hex 32)/" ./.env
Click to expand and view more

Start the stack:

BASH
docker compose up -d && docker compose logs -f
Click to expand and view more

Check:

BASH
docker compose ps

docker compose volumes
Click to expand and view more

If everything is fine, we will see roughly this output:

Let’s configure autostart with Systemd right away:

BASH
cat > ./rustfs.service << EOF
[Unit]
Description=RustFS S3 service
Requires=docker.service
After=docker.service

[Service]
Restart=always
RestartSec=5
User=root
Group=root
WorkingDirectory=$(realpath ./)
ExecStart=/usr/bin/docker compose up --remove-orphans
ExecStop=/usr/bin/docker compose down

[Install]
WantedBy=multi-user.target
EOF
Click to expand and view more
BASH
chmod 600 ./rustfs.service

ln -s $(realpath ./rustfs.service) /etc/systemd/system

systemctl daemon-reload

systemctl enable --now rustfs

systemctl status rustfs
Click to expand and view more

Now the rustfs service can be managed with the systemctl utility:

BASH
systemctl stop rustfs
systemctl start rustfs
systemctl restart rustfs
systemctl status rustfs
Click to expand and view more

Installing the Angie reverse proxy server

Angie is a fork of the well-known Nginx (only better!) by our compatriots who previously worked on Nginx.

Create the project directory:

BASH
mkdir -vp /etc/compose/angie && cd /etc/compose/angie
Click to expand and view more

Create the Docker service description file:

BASH
vim ./compose.yaml
Click to expand and view more

Fill it in:

YAML
---
# https://angie.software/angie/docs/installation/docker/

networks:
  arkeep_net:
    external: true

services:
  angie:
    image: docker.angie.software/angie:1.9.1-alpine
    container_name: angie
    restart: unless-stopped
    stop_grace_period: 1m
    cpus: 1
    mem_limit: 1G
    hostname: angie
    environment:
      TZ: Europe/Moscow
    volumes:
      - ./config/angie.conf:/etc/angie/angie.conf:ro
      - ./config/http.d/:/etc/angie/http.d/:ro
      - ./certs/:/var/lib/angie/acme/
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "80:80/tcp"
      - "443:443/tcp"
      - "443:443/udp"
    networks:
      - arkeep_net
Click to expand and view more

Create the config:

BASH
mkdir -vp ./config

vim ./config/angie.conf
Click to expand and view more

Fill it in:

BASH
user angie;
worker_processes auto;
worker_rlimit_nofile 65536;

error_log /var/log/angie/error.log notice;
pid /run/angie/angie.pid;

events {
    worker_connections 65536;
}

http {
    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }

    include /etc/angie/mime.types;
    default_type application/octet-stream;

    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"';

    log_format extended '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" rt="$request_time" '
                        '"$http_user_agent" "$http_x_forwarded_for" '
                        'h="$host" sn="$server_name" ru="$request_uri" u="$uri" '
                        'ucs="$upstream_cache_status" ua="$upstream_addr" us="$upstream_status" '
                        'uct="$upstream_connect_time" urt="$upstream_response_time"';

    access_log /var/log/angie/access.log main;

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    keepalive_timeout 20;
    keepalive_requests 1000;

    server_tokens off;

    # DNS resolver for ACME.
    resolver 8.8.8.8 8.8.4.4 ipv6=off valid=300s;

    acme_client le https://acme-v02.api.letsencrypt.org/directory
        email=kar-kar@r4ven.me
        key_type=ecdsa
        renew_before_expiry=30d
        # renew_on_load=on
    ;

    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 1d;
    ssl_session_tickets off;

    upstream arkeep_server {
        server arkeep-server:9090;
    }

    upstream arkeep_console {
        server arkeep-server:8080;
    }

    upstream rustfs_server {
        server rustfs:9000;
    }

    upstream rustfs_console {
        server rustfs:9001;
    }

    server {
        listen 80;

        server_name
            arkeep-grpc.r4ven.me
            arkeep.r4ven.me
            rustfs-s3.r4ven.me
            rustfs.r4ven.me
        ;

        location /.well-known/acme-challenge/ {
            root /usr/share/angie/html/;
        }

        location / {
            return 301 https://$host$request_uri;
        }
    }

    server {
        listen 443 ssl;

        http2 on;
        http3 on;

        server_name arkeep-grpc.r4ven.me;

        acme le;

        ssl_certificate $acme_cert_le;
        ssl_certificate_key $acme_cert_key_le;

        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_prefer_server_ciphers on;

        location / {
            grpc_pass grpc://arkeep_server;

            grpc_set_header Host $host;
            grpc_set_header X-Real-IP $remote_addr;
            grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            grpc_connect_timeout 10s;
            grpc_send_timeout 1h;
            grpc_read_timeout 1h;

            grpc_socket_keepalive on;
        }
    }

    server {
        listen 443 ssl;
    
        http2 on;
        http3 on;
    
        server_name arkeep.r4ven.me;
    
        acme le;
    
        ssl_certificate $acme_cert_le;
        ssl_certificate_key $acme_cert_key_le;
    
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_prefer_server_ciphers on;
    
        client_max_body_size 0;
    
        location / {
            proxy_pass http://arkeep_console;
    
            proxy_http_version 1.1;
            proxy_request_buffering off;
            proxy_buffering off;
    
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-Host $host;
            proxy_set_header X-Forwarded-Port 443;
            proxy_set_header X-Forwarded-Ssl on;
    
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
    
            proxy_connect_timeout 10s;
            proxy_send_timeout 1h;
            proxy_read_timeout 1h;
        }
    }

    server {
        listen 443 ssl;

        http2 on;
        http3 on;

        server_name rustfs-s3.r4ven.me;

        acme le;

        ssl_certificate $acme_cert_le;
        ssl_certificate_key $acme_cert_key_le;

        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_prefer_server_ciphers on;

        client_max_body_size 0;

        location / {
            proxy_pass http://rustfs_server;

            proxy_http_version 1.1;
            proxy_request_buffering off;
            proxy_buffering off;
            proxy_cache off;
            proxy_max_temp_file_size 0;

            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-Host $host;
            proxy_set_header X-Forwarded-Port 443;
            proxy_set_header X-Forwarded-Ssl on;

            proxy_connect_timeout 10s;
            proxy_send_timeout 1h;
            proxy_read_timeout 1h;
        }
    }

    server {
        listen 443 ssl;

        http2 on;
        http3 on;

        server_name rustfs.r4ven.me;

        acme le;

        ssl_certificate $acme_cert_le;
        ssl_certificate_key $acme_cert_key_le;

        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_prefer_server_ciphers on;

        location / {
            proxy_pass http://rustfs_console;

            proxy_http_version 1.1;

            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
        }
    }

    include /etc/angie/http.d/*.conf;
}
Click to expand and view more

Start the stack:

BASH
docker compose up -d && docker compose logs -f
Click to expand and view more

Check Angie and all other services:

BASH
docker compose ps

docker ps
Click to expand and view more

If everything is fine, we will see roughly this output:

Let’s configure autostart with Systemd right away:

BASH
cat > ./angie.service << EOF
[Unit]
Description=Angie proxy service
Requires=docker.service
After=docker.service

[Service]
Restart=always
RestartSec=5
User=root
Group=root
WorkingDirectory=$(realpath ./)
ExecStart=/usr/bin/docker compose up --remove-orphans
ExecStop=/usr/bin/docker compose down

[Install]
WantedBy=multi-user.target
EOF
Click to expand and view more
BASH
chmod 600 ./angie.service

ln -s $(realpath ./angie.service) /etc/systemd/system

systemctl daemon-reload

systemctl enable --now angie

systemctl status angie
Click to expand and view more

Now the angie service can be managed with the systemctl utility:

BASH
systemctl stop angie
systemctl start angie
systemctl restart angie
systemctl status angie
Click to expand and view more

The first thing after starting the reverse proxy is to create an admin account in Arkeep:

Open a web browser, go to the address specified for the GUI panel, and set the administrator login credentials:

TXT
https://arkeep.r4ven.me/
Click to expand and view more

After logging in, we will get to the Arkeep dashboard:

In the “Agents” section, we should see our local agent, which starts together with the server:

Installing and configuring Arkeep agents on app servers

Now move to the hosts that need to be backed up:

Create the directory:

BASH
install -m 700 -d /etc/compose

mkdir -vp /etc/compose/arkeep_agent && cd /etc/compose/arkeep_agent
Click to expand and view more

Create the agent compose file:

BASH
vim ./compose.yaml
Click to expand and view more

Fill it in:

YAML
---
# https://github.com/arkeep-io/arkeep/tree/main/deploy/docker

volumes:
  arkeep_agent:
    name: arkeep_agent

services:
  arkeep_agent:
    image: ghcr.io/arkeep-io/arkeep-agent:v0.4.5
    container_name: arkeep-agent-app1
    restart: unless-stopped
    stop_grace_period: 1m
    cpus: 1
    mem_limit: 1G
    hostname: arkeep-agent-app1
    env_file: ./.env
    volumes:
      - arkeep_agent:/var/lib/arkeep-agent/
      - /:/hostfs/:ro # :ro for backup-only; change to :rw to also restore to local paths
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/lib/docker/volumes/:/var/lib/docker/volumes/:ro
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
Click to expand and view more

Here it is worth separately explaining the /hostfs bind mount: this is how the agent gets access to the host filesystem for backup. The ro mode is sufficient only for creating backups. To restore files to their original location, access in rw mode is required.

Create the env file:

BASH
vim ./.env
Click to expand and view more

Fill it in:

INI
# https://github.com/arkeep-io/arkeep/tree/main/deploy/docker

TZ=Europe/Moscow
ARKEEP_SERVER_ADDR=arkeep-grpc.r4ven.me:443
ARKEEP_SERVER_HTTP_ADDR=https://arkeep.r4ven.me
ARKEEP_AGENT_SECRET=
ARKEEP_STATE_DIR=/var/lib/arkeep-agent
ARKEEP_LOG_LEVEL=info
ARKEEP_GRPC_INSECURE=false
Click to expand and view more

Parameter description:

Start the container:

BASH
docker compose up -d && docker compose logs -f
Click to expand and view more

If the agent successfully connects to the server, we should see these two lines:

BASH
arkeep-agent-app1  | {"level":"info","ts":1781279912.8271096,"logger":"connection","caller":"connection/manager.go:427","msg":"registered with server","agent_id":"019ebc8e-7375-7ec1-b345-9d309cc983e8","agent_name":"arkeep-agent-app1"}
arkeep-agent-app1  | {"level":"info","ts":1781279912.827225,"logger":"connection","caller":"connection/manager.go:527","msg":"job stream open","agent_id":"019ebc8e-7375-7ec1-b345-9d309cc983e8"}
Click to expand and view more

And a new agent will appear in the Arkeep management console:

Let’s configure agent autostart with Systemd right away:

BASH
cat > ./arkeep-agent.service << EOF
[Unit]
Description=Arkeep agent service
Requires=docker.service
After=docker.service

[Service]
Restart=always
RestartSec=5
User=root
Group=root
WorkingDirectory=$(realpath ./)
ExecStart=/usr/bin/docker compose up --remove-orphans
ExecStop=/usr/bin/docker compose down

[Install]
WantedBy=multi-user.target
EOF
Click to expand and view more
BASH
chmod 600 ./arkeep-agent.service

ln -s $(realpath ./arkeep-agent.service) /etc/systemd/system

systemctl daemon-reload

systemctl enable --now arkeep-agent

systemctl status arkeep-agent
Click to expand and view more

Now the arkeep-agent service can also be managed with the systemctl utility:

BASH
systemctl stop arkeep-agent
systemctl start arkeep-agent
systemctl restart arkeep-agent
systemctl status arkeep-agent
Click to expand and view more

Configuring RustFS

To prepare and configure the S3 storage, please read my separate article:

RustFS - Installing and Configuring S3-Compatible Storage

RustFS - Installing and Configuring S3-Compatible Storage

In this article, we will look at installing and basic configuration of RustFS - an S3-compatible object storage that we will deploy on a Linux server using Docker Compose.

r4ven.me/en/storage/rustfs-ustanovka-i-nastroyka-s3-sovmestimogo-hranilishcha/

For further configuration, we will need the following data:

Configuring Arkeep

First, configure the remote storage where we will put backups.

Configuring Destination - S3

Go to the “Destinations” section:

Click “New Destination”, in the “Type” parameter specify the desired storage type, in our case it is “S3/Object Storage”, the bucket name, and set the connection credentials:

The new storage will appear in the list:

A few words about other Destination types

Besides S3, backups can be stored in a local directory, in a directory accessible via SFTP, on a REST Server, or in any other remote storage whose access can be configured with Rclone (and that is 40+ providers):

To use rclone in the case of arkep-agent running in Docker, you need to install rclone additionally and mount a config with the required remote storage to it. Example for WebDav:

BASH
cat > ./rclone.conf << EOF
[webdav-backup]
type = webdav
url = https://cloud.example.com/remote.php/dav/files/USER
vendor = nextcloud
user = USER
pass = OBSCURED_PASSWORD
EOF
Click to expand and view more

And the compose file will look like this:

YAML
---

services:
  arkeep_agent:
    image: ghcr.io/arkeep-io/arkeep-agent:v0.4.5
    container_name: arkeep-agent-rclone
    restart: unless-stopped
    hostname: arkeep-agent-rclone
    env_file: ./.env
    command: sh -c 'apk add rclone; exec arkeep-agent'
    volumes:
      - ./data/:/var/lib/arkeep-agent/
      - /:/hostfs/:rw
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/lib/docker/volumes/:/var/lib/docker/volumes/:ro
      - ./rclone.conf:/root/.config/rclone/rclone.conf:ro
Click to expand and view more

In the “Destination” settings in the Arkeep panel, you will need to specify the backup type - “Rclone” and, optionally, the storage directory path:

Adding a Policy

Next, configure a policy, or more precisely a backup job. Go to “Policies” and click “New Policy”:

Here, specify all required parameters: job name, agent, and password for encrypting files in the remote storage:

Next, specify the backup sources: in our example, we will back up the /etc directory and the Docker volume of the Arkeep agent on the app1.r4ven.me host:

Next, configure the job execution schedule in crontab format, as well as the retention time and the number of backups:

At the very end, select the remote storage (our S3) and set exclusions if needed:

Starting a backup

To check functionality, try running the backup job we created manually:

Success:

A new backup will appear in the “Snapshots” section:

In our S3 rustfs.r4ven.me, you will be able to see the Restic repository files:

Restoring from backup

As mentioned earlier, by default the host filesystem is mounted in read-only mode. In compose.yaml, this place is marked like this: :ro for backup-only; change to :rw to also restore to local paths.

BASH
volumes:
  - /:/hostfs:ro
Click to expand and view more

To allow the agent to write to the host filesystem, change the parameter above to:

BASH
volumes:
  - /:/hostfs:rw
Click to expand and view more

And restart the agent:

BASH
systemctl restart arkeep-agent
Click to expand and view more

Or mount a separate directory with write access.

Go to the “Snapshots” section, select the suitable one, then the three dots and “Restore”:

Here we have two restore modes to choose from:

Let’s look at each of them.

Custom path

In this option, we also have a choice: restore everything to the specified directory - we explicitly set our own path in the host filesystem (without hostfs) where we will restore files from the backup, or restore individual files - here you can select individual files/directories and restore only them.

To start the restore, click “Start restore”. On success, we will see:

Check in the host filesystem:

Excellent!

Original path

When using the “Original path” restore mode, files from the backup will be restored to the original path where they were located at backup time. This procedure may “overwrite” files changed after the backup. Which is what we are warned about:

Start the restore and… success:

Possible problems

While studying Arkeep, I encountered some problems/errors. I will list them here, maybe it will be useful for someone.

Most often, you may encounter exactly this error:

As stated in the message, the destination directory is not writable. I talked about this point earlier, at the beginning of the restore from backup step.

When connecting agents:

BASH
arkeep-agent-app1  | {... "msg":"enrollment failed — check --server-http-addr and --agent-secret","error":"enroll: server returned 405 Method Not Allowed" ...}
Click to expand and view more

The message enrollment failed — server returned 405 Method Not Allowed appears because the agent tries to perform automatic client certificate issuance (enrollment) through the HTTP API, but the server has ARKEEP_GRPC_INSECURE=true enabled, where this mechanism is not used.

The error can be ignored if after it the agent successfully connects to the server over gRPC, which is confirmed by the registered with server and job stream open messages. At the same time, the connection between the agent and the server remains protected by the Angie TLS certificate on port 443, and unencrypted traffic exists only inside the server’s local Docker network.

TXT
tls: failed to verify certificate:
x509: certificate signed by unknown authority
Click to expand and view more

The usual causes are:

TXT
x509: certificate is valid for arkeep-grpc, not arkeep.r4ven.me
Click to expand and view more

Usually it happens if:

In this case, the easiest way is to stop the server, delete the Docker volume directory with data (if you have just deployed it), and start the container again.

The certificates will be reissued automatically.

A few words about voluntary telemetry

By default, Arkeep collects anonymous telemetry. This can be disabled, but first read the developer’s quote:

https://github.com/arkeep-io/arkeep#telemetry

Arkeep sends anonymous usage statistics once per day to help prioritize development. No personal data, backup contents, credentials, or hostnames are ever transmitted.

What is sent: a stable random instance ID, Arkeep version, OS, number of connected agents, and number of active policies.

Aggregate stats are public at: https://telemetry.arkeep.io/stats

To opt out: set ARKEEP_TELEMETRY=false or pass --telemetry=false.

Afterword

Overall, Arkeep seems to me like a very convenient and useful backup tool with great potential. I hope the project becomes popular and the developer has motivation to develop and improve it. The idea of using Restic and Rclone is simple and brilliant 👍.

It seems to me that the project has undeservedly few stars and forks. If you liked the tool and you have a GitHub account, please give the Arkeep repository a star ⭐.

Before writing this article, I ran my own tests on version 0.4.3. At the restore stage, I noticed one bug: when selecting the restore mode, the field was empty 🤷‍♂️. There was also a nuance with an incorrect “error” type message in the agent log.

As a result, I created two issues in the Arkeep repository, and the developer responded to them quite quickly, fixing the bug in the next update (at the time of writing, version 0.4.5) and moving the message to the “info” section, for which many thanks to him 🤝.

If anyone is interested in the details, here are the links:

Thanks for reading. And I will repeat once again: admins are divided into two types: those who make backups, and those who will definitely make them. All the best!

References

Copyright Notice

Author: Ivan Cherniy

Link: https://r4ven.me/en/storage/arkeep-centralizovannyy-backup-server-s-agentami-na-baze-restic-i-rclone/

License: CC BY-NC-SA 4.0

Blog materials may be used with attribution to the author and source, for non-commercial purposes, and under the same license.

Related Posts

Start searching

Enter keywords to search articles

↑↓
↵
ESC
⌘K Shortcut