Metrics and Monitoring

Metrics allow you to gather run time statistics on your aggregator or attester node.

Overview

Prometheus is a robust, open-source tool designed for event monitoring and alerting. It collects and stores metrics as time series data, enabling querying and visualization of system performance. Metrics for Aggregator or Attester nodes can be gathered from:

  1. Standard Prometheus metrics: Node uptime, performance statistics, and more.

  2. libp2p metrics: For peer-to-peer networking data.

  3. gossipsub metrics: For the topic-based publish/subscribe communication protocol.

Enabling metrics may impact performance and is therefore optional.

Enabling Metrics

To activate metrics, use the --metrics flag with the node commands (aggregator or attester).

libp2p Metrics Scraping

Since libp2p metrics do not emit events, data must be actively scraped. To facilitate this, a dedicated server will be launched to expose metrics.

  • By default, this server listens on port 6060.

  • You can specify an alternative port using the --metrics.port <port> option.

Example Usage

Enable metrics on an aggregator node using the default port (6060):

othentic-cli node aggregator --metrics

Enable metrics on an attester node with a custom port:

othentic-cli node attester --metrics --metrics.port 7070

Example for Prometheus and Grafana Integration

Overview

To monitor your AVS nodes, you need to set up Prometheus for collecting metrics, configure your Attester or Aggregator node to capture system-level metrics (as explained earlier), and use Grafana to visualize the data. By using Docker Compose, you can streamline the process by defining and managing these services in a single configuration file.

For a comprehensive example, refer to the Simple Price Oracle Example repository, which contains a working configuration for Prometheus, Grafana, and the node setup.

Docker Compose Configuration

Prometheus

Prometheus acts as the core of the monitoring system, collecting metrics from the node. The setup uses the prom/prometheus:latest image and binds a custom configuration file, prometheus.yaml, to the container.

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yaml  # Bind mount the config file
    ports:
      - "9090:9090"  # Expose Prometheus on port 9090
    command:
      - '--config.file=/etc/prometheus/prometheus.yaml'  # Specify the config file location
    restart: unless-stopped

Grafana

Grafana provides a user-friendly interface to visualize and analyze the data collected by Prometheus. This setup uses the grafana/grafana:latest image and configures persistent data storage using a Docker volume.

services:
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3000:3000"  # Expose Grafana on port 3000
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin  # Set the admin user password
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning
      - ./grafana/dashboards:/var/lib/grafana/dashboards
      - grafana-storage:/var/lib/grafana

volumes:
  grafana-storage: {}

Volume Details:

./grafana/provisioning:/etc/grafana/provisioning

  • This volume is used for provisioning Grafana with predefined configurations, such as data sources and dashboard templates. You can place YAML files in this directory to automate the setup of data sources or other settings upon container startup.

  • Example YAML file for a data source:

    apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    access: proxy
    isDefault: true

./grafana/dashboards:/var/lib/grafana/dashboards

  • This volume stores dashboard JSON files, which define the visual layouts and metrics to display in Grafana. By placing preconfigured dashboards here, you ensure they are automatically available after the container starts.

  • Example dashboard configuration

grafana-storage:/var/lib/grafana

  • This Docker volume ensures persistent storage for Grafana's internal database, including user-created dashboards, settings, and other data. If the Grafana container is recreated, this volume retains all saved configurations.

Starting the Setup

Follow these steps to start the monitoring setup:

  1. Prepare the Configuration:

    • Ensure the prometheus.yaml file is in the same directory as the Docker Compose file.

    • Customize the Grafana provisioning and dashboards directories as needed.

  2. Start the Services:

    • Run the following command to start Prometheus and Grafana in detached mode:

      docker-compose up -d

  3. Access the Services:

  4. Visualize Metrics:

    • Once the setup is complete, use Grafana to create or import dashboards to visualize the metrics collected by Prometheus.

Screenshots of the Othentic-CLI dashboard in Grafana

PreviousAdd-OnsNextPersistent storage

Last updated