How to Set Up Promtail, Grafana and Loki on Debian 12/13

Set up Promtail, Grafana and Loki on Debian 12/13: install Grafana, run Loki and Promtail as systemd services, and query logs. Promtail is EOL; see Alloy.

Grafana Loki is an open source log aggregation tool provided by the Grafana Labs. It is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream. This reduces the workload of processing and storing logs.

Promtail is Loki’s log collector, which sends log tags to Grafana Loki for indexing.

Promtail is end of life. Grafana declared Promtail EOL on March 2, 2026: it gets no more updates or security fixes, and newer Loki releases no longer ship a Promtail binary. For a new setup, follow How to Ship Logs to Loki With Grafana Alloy on Debian 12/13 instead. It installs the same Loki and Grafana, uses Grafana Alloy as the agent, and shows how to convert an existing Promtail config. This guide is kept for servers that still run Promtail.

Related posts:

Installing Grafana

Ensure the system is up to date before proceeding:

sudo apt update
sudo apt upgrade -y

Add the Grafana GPG key to a dedicated keyring:

sudo apt install -y wget
sudo mkdir -p /etc/apt/keyrings
sudo wget -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
sudo chmod 644 /etc/apt/keyrings/grafana.asc

Add the Grafana APT repository, signed by that key:

echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" \
  | sudo tee /etc/apt/sources.list.d/grafana.list

Older versions of this guide used curl ... | sudo apt-key add - and add-apt-repository with packages.grafana.com. apt-key is deprecated and removed from current APT, and Grafana’s repository has moved to apt.grafana.com.

Next update apt repo then install grafana:

sudo apt update && sudo apt install -y grafana

Confirm the installed grafana version. The Installed: line shows the version (Grafana 13.x at the time of writing):

apt-cache policy grafana

Finally start and enable Grafana

sudo systemctl start grafana-server

sudo systemctl enable grafana-server

Grafana is now installed and can be accessed through the server’s IP and port 3000. (http://server_IP:3000)

You need to allow port 3000 to pass through the firewall if you have firewall enabled. Use this command:

sudo ufw allow proto tcp from any to any port 3000

Install Grafana Loki

Loki is available as a Linux binary on github release page for the app here. Scroll down to the Assets section under the version that you want to install. Download using this command:

curl -LO https://github.com/grafana/loki/releases/download/v3.7.9/loki-linux-amd64.zip

Extract the downloaded zip file. You need the unzip program installed. Ensure that it is present using this command:

sudo apt install -y unzip

You can then extract the file and move it to the binary directory:

unzip loki-linux-amd64.zip
sudo mv loki-linux-amd64 /usr/local/bin/loki

Create config directory and add the configuration files provided by Grafana Labs:

sudo mkdir /etc/loki
sudo mkdir -p /data/loki

Then edit the config file

sudo vim /etc/loki/config.yaml

Add this content:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /data/loki
  storage:
    filesystem:
      chunks_directory: /data/loki/chunks
      rules_directory: /data/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2024-04-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

This uses the tsdb store with schema v13, which is what Loki 3.x expects. The original version of this guide used boltdb-shipper with schema v11, which Loki 3 has deprecated and which does not support newer features such as structured metadata. If you are upgrading an existing Loki 2.x install, keep the old entry and add the tsdb/v13 one below it with a future from: date, so Loki can still read the data it already has.

Check that Loki accepts the config:

sudo /usr/local/bin/loki -config.file=/etc/loki/config.yaml -verify-config

Next create a loki systemd service. Use this command to create file /etc/systemd/system/loki.service.

sudo tee /etc/systemd/system/loki.service<<EOF
[Unit]
Description=Loki service
After=network.target

[Service]
Type=simple
User=root

ExecStart=/usr/local/bin/loki -config.file /etc/loki/config.yaml

[Install]
WantedBy=multi-user.target
EOF

Reload the system daemon, and then start the Loki service:

sudo systemctl daemon-reload
sudo systemctl start loki

You can check whether the service has started successfully:

$ sudo systemctl daemon-reload
$ sudo systemctl start loki
$ sudo systemctl status loki

● loki.service - Loki service
     Loaded: loaded (/etc/systemd/system/loki.service; disabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-01-20 18:39:08 UTC; 19s ago
   Main PID: 34374 (loki)
      Tasks: 8 (limit: 4626)
     Memory: 17.8M
        CPU: 111ms
     CGroup: /system.slice/loki.service
             └─34374 /usr/local/bin/loki -config.file /etc/loki/config.yaml

Jan 20 18:39:08 dev-debiansrv.inv.re loki[34374]: level=info ts=2022-01-20T18:39:08.487479168Z caller=compactor.go:263 msg="compactor is ACTIVE in the ring"
Jan 20 18:39:08 dev-debiansrv.inv.re loki[34374]: level=info ts=2022-01-20T18:39:08.619449446Z caller=scheduler.go:629 msg="scheduler is ACTIVE in the ring"
Jan 20 18:39:08 dev-debiansrv.inv.re loki[34374]: level=info ts=2022-01-20T18:39:08.619551917Z caller=module_service.go:64 msg=initialising module=query-frontend
Jan 20 18:39:08 dev-debiansrv.inv.re loki[34374]: level=info ts=2022-01-20T18:39:08.619815925Z caller=module_service.go:64 msg=initialising module=querier
Jan 20 18:39:08 dev-debiansrv.inv.re loki[34374]: level=info ts=2022-01-20T18:39:08.619923866Z caller=loki.go:355 msg="Loki started"
Jan 20 18:39:11 dev-debiansrv.inv.re loki[34374]: level=info ts=2022-01-20T18:39:11.620368148Z caller=worker.go:205 msg="adding connection" addr=127.0.0.1:9096

You can now access Loki metrics at http://server-IP:3100/metrics, and http://server-IP:3100/ready returns ready once Loki is up.

Installing Promtail Agent

Promtail is an agent which ships the contents of local logs to a private Grafana Loki instance or Grafana Cloud. It is usually deployed to every machine that has applications needed to be monitored.

Download the Promtail binary from the Loki github releases page. Newer Loki releases (3.7 and later) no longer include Promtail, so the last Promtail binary is under the v3.6.x releases. Download using this command:

curl -LO https://github.com/grafana/loki/releases/download/v3.6.11/promtail-linux-amd64.zip

Extract the downloaded zip file and move the binary to executable directory.

unzip promtail-linux-amd64.zip
sudo mv promtail-linux-amd64 /usr/local/bin/promtail

Confirm the installed version. The first line of output should read promtail, version 3.6.11:

promtail --version

Next create config directory and a data directory:

sudo mkdir /etc/promtail
sudo mkdir -p /data/promtail

Create a YAML configuration file for Promtail:

sudo vim /etc/promtail/config.yaml

Add this content to the file:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /data/promtail/positions.yaml

clients:
  - url: http://127.0.0.1:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log
- job_name: grafanalogs
  static_configs:
  - targets:
      - localhost
    labels:
      job: grafana
      __path__: /var/log/grafana/grafana.log

Next, create a systemd service to manage the promtail service:

sudo tee /etc/systemd/system/promtail.service<<EOF
[Unit]
Description=Promtail service
After=network.target

[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/promtail -config.file /etc/promtail/config.yaml

[Install]
WantedBy=multi-user.target
EOF

Reload and start the Promtail service

sudo systemctl daemon-reload
sudo systemctl start promtail.service

Confirm whether the service is running:

$ systemctl status promtail

● promtail.service - Promtail service
     Loaded: loaded (/etc/systemd/system/promtail.service; disabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-01-20 18:51:10 UTC; 12s ago
   Main PID: 34488 (promtail)
      Tasks: 9 (limit: 4626)
     Memory: 29.9M
        CPU: 532ms
     CGroup: /system.slice/promtail.service
             └─34488 /usr/local/bin/promtail -config.file /etc/promtail/config.yaml

Jan 20 18:51:15 dev-debiansrv.inv.re promtail[34488]: ts=2022-01-20T18:51:15.303156121Z caller=log.go:168 level=info msg="Seeked /var/log/fontconfig.log - &{Offset:0 Whence:0}"
Jan 20 18:51:15 dev-debiansrv.inv.re promtail[34488]: level=info ts=2022-01-20T18:51:15.303172481Z caller=tailer.go:126 component=tailer msg="tail routine: started" path=/var/log/fontconfig.log
Jan 20 18:51:15 dev-debiansrv.inv.re promtail[34488]: ts=2022-01-20T18:51:15.303201692Z caller=log.go:168 level=info msg="Seeked /var/log/kern.log - &{Offset:0 Whence:0}"
Jan 20 18:51:15 dev-debiansrv.inv.re promtail[34488]: level=info ts=2022-01-20T18:51:15.303214345Z caller=tailer.go:126 component=tailer msg="tail routine: started" path=/var/log/kern.log
Jan 20 18:51:15 dev-debiansrv.inv.re promtail[34488]: ts=2022-01-20T18:51:15.303238377Z caller=log.go:168 level=info msg="Seeked /var/log/lastlog - &{Offset:0 Whence:0}"
Jan 20 18:51:15 dev-debiansrv.inv.re promtail[34488]: level=info ts=2022-01-20T18:51:15.305348131Z caller=tailer.go:126 component=tailer msg="tail routine: started" path=/var/log/lastlog
Jan 20 18:51:15 dev-debiansrv.inv.re promtail[34488]: ts=2022-01-20T18:51:15.30581234Z caller=log.go:168 level=info msg="Seeked /var/log/syslog - &{Offset:0 Whence:0}"
Jan 20 18:51:15 dev-debiansrv.inv.re promtail[34488]: level=info ts=2022-01-20T18:51:15.305873583Z caller=tailer.go:126 component=tailer msg="tail routine: started" path=/var/log/syslog
Jan 20 18:51:15 dev-debiansrv.inv.re promtail[34488]: ts=2022-01-20T18:51:15.306786457Z caller=log.go:168 level=info msg="Seeked /var/log/user.log - &{Offset:0 Whence:0}"
Jan 20 18:51:15 dev-debiansrv.inv.re promtail[34488]: level=info ts=2022-01-20T18:51:15.307060365Z caller=tailer.go:126 component=tailer msg="tail routine: started" path=/var/log/user.log

We have managed to install Grafana, Loki and promtail.

Configuring Loki Data Source

Now that everything is set up, we need to add loki in the Grafana UI. Log in to the Grafana web interface and click on Connections -> Data sources. Then click Add data source. Search for Loki and add these values:

Name: Loki
URL: http://127.0.0.1:3100

Scroll to the bottom then Click Save & test. You should see a notification that the data source has been successfully added.

Visualize Loki logs on Grafana

With everything set up, we can use Grafana to visualize the logs. Click on explore Then choose Loki In the data source. In the log browser, search for {job="varlogs"} to search var logs messages:

That is it! We have successfully installed Grafana Loki with Promtail and can visualize logs on the Grafana dashboard.

When you are ready to move off Promtail, the Grafana Alloy guide shows how to convert /etc/promtail/config.yaml with alloy convert and keep reading from the same file positions.

comments powered by Disqus
Citizix Ltd
Built with Hugo
Theme Stack designed by Jimmy