Skip to main content

Observability Metrics Tutorial

This tutorial shows you how to set up a complete monitoring stack for ZeroTier networks using Prometheus and Grafana. You can use this type of monitoring pipeline to measure the health and activity of your networks, whether you're using our managed environment or hosting your own controller.

Overview​

ZeroTier Inc doesn't have access to your traffic. We don't currently supply a monitoring "dashboard" for your networks and nodes, but you can build your own powerful observability system!

This tutorial includes an example docker-compose configuration and setup scripts to run a ZeroTier network along with an observability stack based on Prometheus and Grafana.

Prerequisites​

To run this locally, you'll need the following tools installed:

  • Docker + docker-compose
  • jq (JSON command-line processor)

We've tested this demo on MacOS and Linux. On Windows, you'll need to use WSL2 for a local Linux environment. (Note: pull requests to add PowerShell support for the setup script are welcome!)

info

For this example, we're running a standalone controller, which allows us to avoid capturing and injecting a valid ZeroTier Central API token into the demo environment. However, the same observability tools and configuration methods will work for managed networks as well.

Setup​

1. Get the Tutorial Files​

Clone or download the tutorial repository:

git clone https://github.com/zerotier/metrics-tutorial.git
cd metrics-tutorial

2. Start the Stack​

Run the docker-compose setup and initialization script:

docker-compose up -d
./setup.sh

This will start:

  • ZeroTier controller
  • Prometheus metrics collection
  • Grafana dashboard service
  • Example network configuration

Grafana Configuration​

1. Access Grafana​

Open http://localhost:3000/ in your browser.

2. Initial Login​

  • Username: admin
  • Password: admin
  • Set a new password or click 'Skip'

3. Configure Prometheus Data Source​

  1. Navigate to Connections → Data Sources → Add New Data Source
  2. Select Prometheus (first option)
  3. Set the Prometheus server URL: http://metrics-tutorial-prometheus-1:9090
  4. Click Save and test
  5. Click Explore view

4. Create Your First Query​

  1. Select zt_packet in the Metric dropdown
  2. Choose Last 15 minutes in the range picker (upper-right)
  3. Click Run query

The graph will populate in the bottom half of the screen showing ZeroTier packet metrics.

5. Monitor Prometheus Jobs​

You can also check the status of Prometheus jobs directly at: http://localhost:9090

Key Metrics for Monitoring​

The following metrics are particularly useful for general health and activity monitoring:

Core Network Metrics​

  • zt_packet - Network packet flow (labeled by rx/tx direction, peer node ID, network ID, etc.)
  • zt_peer_path - Active paths to known network peers
  • zt_network_status - Network connection status
  • zt_peer_status - Peer connection status

Performance Metrics​

  • zt_packet_error - Packet transmission errors
  • zt_path_stability - Connection path stability
  • zt_latency - Network latency measurements

Creating Custom Dashboards​

Now that you have the basic pipeline running, you can create custom dashboards reporting ZeroTier metrics. Consider visualizing:

  • Network traffic patterns over time
  • Peer connectivity status
  • Error rates and network health
  • Geographic distribution of connections
  • Bandwidth utilization

Cleanup​

When you're done with the tutorial, clean up the Docker containers:

./cleanup.sh

Alternative Monitoring Tools​

Beyond Prometheus and Grafana, you can use your preferred monitoring tools over your ZeroTier networks:

Additional Resources​

Community​

If you've found a cool use for this set of observability hooks, we'd love to hear about it! Please reach out via: