Collecting and reporting metrics

Honeybadger’s Ruby gem (v5.11+) can be used to collect metrics and send them to Insights.

Enabling Insights

To enable collecting of metrics, you’ll first need to enable Insights in your honeybadger.yml configuration file:

insights:
  enabled: true

Enable metrics collection

You can enable the metrics collection of each plugin by adding the relevant configuration to your honeybadger.yml file:

rails:
  insights:
    metrics: true
karafka:
  insights:
    metrics: true
sidekiq:
  insights:
    metrics: true
net_http:
  insights:
    metrics: true
solid_queue:
  insights:
    metrics: true
puma:
  insights:
    metrics: true
autotuner:
  insights:
    metrics: true

Enabling this will collect metric data for the libraries below and display it in the Insights section of your project. Each metric is emitted as a metric.hb event with a metric_source identifying the plugin and a metric_name identifying the metric.

rails 12

Kind	Metric name
gauge	duration.sql.active_record
gauge	duration.process_action.action_controller
gauge	db_runtime.process_action.action_controller
gauge	view_runtime.process_action.action_controller
gauge	duration.cache_read.active_support
gauge	duration.cache_fetch_hit.active_support
gauge	duration.cache_write.active_support
gauge	duration.cache_exist?.active_support
gauge	duration.render_partial.action_view
gauge	duration.render_template.action_view
gauge	duration.render_collection.action_view
gauge	duration.perform.active_job

sidekiq 13

Kind	Metric name
gauge	active_workers
gauge	active_processes
gauge	jobs_processed
gauge	jobs_failed
gauge	jobs_scheduled
gauge	jobs_enqueued
gauge	jobs_dead
gauge	jobs_retry
gauge	queue_latency
gauge	queue_depth
gauge	queue_busy
gauge	capacity
gauge	utilization

solid_queue 8

Kind	Metric name
gauge	jobs_in_progress
gauge	jobs_blocked
gauge	jobs_failed
gauge	jobs_scheduled
gauge	jobs_processed
gauge	active_workers
gauge	active_dispatchers
gauge	queue_depth

autotuner 5

Kind	Metric name
gauge	diff.minor_gc_count
gauge	diff.major_gc_count
gauge	diff.time
gauge	request_time
gauge	heap_pages

puma 5

Kind	Metric name
gauge	pool_capacity
gauge	max_threads
gauge	requests_count
gauge	backlog
gauge	running

net_http 1

Kind	Metric name
gauge	duration.request

karafka 29

Kind	Metric name
counter	messages_consumed
counter	messages_consumed_bytes
counter	consume_attempts
counter	consume_errors
counter	receive_errors
counter	connection_connects
counter	connection_disconnects
gauge	network_latency_avg
gauge	network_latency_p95
gauge	network_latency_p99
gauge	consumer_lags
gauge	consumer_lags_delta
gauge	consumer_aggregated_lag
counter	error_occurred
histogram	listener_polling_time_taken
histogram	listener_polling_messages
counter	consumer_messages
counter	consumer_batches
gauge	consumer_offset
histogram	consumer_consumed_time_taken
histogram	consumer_batch_size
histogram	consumer_processing_lag
histogram	consumer_consumption_lag
counter	consumer_revoked
counter	consumer_shutdown
counter	consumer_tick
gauge	worker_total_threads
histogram	worker_processing
histogram	worker_enqueued_jobs

These metrics may be found using the following BadgerQL query:

fields @ts, @preview
| filter event_type::str == "metric.hb"
| filter metric_source::str == "sidekiq"
| filter metric_name::str == "active_workers"
| sort @ts

Customizing Insights for a specific plugin

When Insights is active, all plugins are enabled if the required library is present. For example, if Sidekiq is present in your app, the Sidekiq plugin will be loaded. You can disable automatic Insights instrumentation for a specific plugin by adding a configuration like this:

sidekiq:
  insights:
    enabled: false

This will only affect Insights-related data capture and not the error notification portion of the plugin.

Some plugins allow for an easy way to choose if you want to capture events, metrics, or both for a particular plugin. The following configuration options are available:

rails:
  insights:
    events: true
    metrics: false
karafka:
  insights:
    events: true
    metrics: false
sidekiq:
  insights:
    events: true
    metrics: false
net_http:
  insights:
    events: true
    metrics: false
solid_queue:
  insights:
    events: true
    metrics: false
puma:
  insights:
    events: true
    metrics: false
autotuner:
  insights:
    events: true
    metrics: false

Event options are all true by default. It is recommened to turn off events for plugins that may be producing more data than you actually need. Metric data collection is false by default since most metrics can be calculated from events.

Customizing cluster metrics collection

For certain stats, collection is limited by a polling interval. Honeybadger will periodically collect stats. This can be tailored per plugin through a configuration parameter:

sidekiq:
  insights:
    collection_interval: 5

solid_queue:
  insights:
    collection_interval: 5

puma:
  insights:
    collection_interval: 1

By reducing or increasing the frequency the gem collect stats will all you to fine tune the accuracy of your stats and the resources used to do so.

Some metrics collection methods collect data based on the entire cluster of an application. In these cases, you would only need to collect data from a single instance of the Honeybadger gem. This helps save on unecessary load as well as data usage. Plugins collect data by default, but can be customized through configuration.

sidekiq:
  insights:
    cluster_collection: false

solid_queue:
  insights:
    cluster_collection: false

You can use this configuration paramter to control which instances you want collecting cluster based data. If you are using Sidekiq Enterprise, we automatically detect the leader instance and will enable cluster collection on that instance and disable it on others without any additional configuration.

Data aggregation

When you collect metrics using the Honeybadger gem, the gem will aggregate the data and report the results to Insights every 60 seconds. This allows you to collect data as much and as quickly as you want, while making efficient use of your daily data quota. If you want to tweak the resolution of the timing, you can configure it in the honeybadger.yml config file.

insights:
  registry_flush_interval: 120

The above configuration will adjust the metric registry so that it reports every 2 minutes and help save on data usage.

Manually collecting your own metrics

The Honeybadger gem provides a API for defining and collecting your own metrics to feed into Insights.

Types of metrics

Gauge

A gauge tracks a specific value at a point in time. During aggregation, the metric will record the values: max, min, avg, and latest.

Honeybadger.gauge('data_size', ->{ file.byte_size })

Timer

Timers are similar to gauges in that they track a specific value in time. However, the time methods provides a convenient way to measure duration across your ruby operations.

Honeybadger.time('process_application', ->{ application.process })

Counter

Counters are simple numbers that you can increment or decrement by any value you wish.

Honeybadger.increment_counter('add_to_basket', { item_id: item.id })

Histogram

Histograms allows you to collate data values into predefined bins. The default bins are [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]. You can define your own set by passing a bins attribute to the metric. You may pass a callable lambda, which will be timed and the duration recorded. Or you may also pass a duration keyword argument if you have the value at hand.

Honeybadger.histogram('execute_request', ->{ request.execute })

# or

Honeybadger.histogram('execute_request', duration: duration)

Helper module

You can also include the helper module Honeybadger::InstrumentationHelper into any of your classes. The module provdes shortened forms to create the same metrics as metioned above, as well as other helper methods to customize your metrics.

Here is an example of how we can rewrite the example metrics above, while adding more custom attributes:

class MyMetrics
  include Honeybadger::InstrumentationHelper

  attr_reader :region

  def initialize(region)
    @region = region
  end

  def example
    metric_source 'custom_metrics'
    metric_attributes { region: region }

    gauge 'data_size', ->{ file.byte_size }
    time 'process_application', ->{ application.process }
    increment_counter 'add_to_basket', { item_id: item.id }
    histogram 'execute_request', ->{ request.execute }
  end
end

Aside from a less verbose API, there are two available helper methods that will aid in organizing your metrics. The metric_source method accepts the name of where your metrics are coming from. This can be the name of a library, or the class you are calling from. The metric_attributes method accepts a hash that will be passed to all metrics that follow.

Then you can find these metrics by using the following BadgerQL query:

fields @ts, @preview
| filter event_type::str == "metric.hb"
| filter metric_source::str == "custom_metrics"
| filter region::str == "some-region"
| sort @ts

Ignoring metrics

You can use the before_event callback to inspect or modify metric data, as well as calling halt! to prevent the metric from being sent to Honeybadger:

Honeybadger.configure do |config|
  config.before_event do |event|
    if event.event_type == "metric.hb" && event[:metric_name] == "jobs_processed"
      event.halt!
    end
  end
end

before_event can be called multiple times to add multiple callbacks.

Similarly, you may also ignore metric events by configuring your honeybadger.yml config file by specifying a hash object:

events:
  ignore:
    - event_type: "metric.hb"
      metric_name: "jobs_processed"

Puma

Puma has it’s own plugin system and requires a small change to your puma.rb. The Honeybadger gem comes with Puma plugin and can be enabled by adding the following to your puma.rb:

plugin :honeybadger

Autotuner

To enable Autotuner, follow the directions in the README.md file. You can skip the parts about setting Autotuner.reporter and Autotuner.metrics_reporter as the Honeybadger gem will configure this for you.

More automatic instrumentation

The Honeybadger Ruby gem provides more instrumentation than just metrics. When you enable Insights, you also enable automatic event logging. Check out Automatic instrumentation for more information.