Architecture Deep-Dive

This guide explains the architecture of the honeybadger Ruby gem, and how it interacts with your application.

Who Is This Guide For?

This guide is for anyone interested in learning about how our gem works internally or is attempting to debug/rule out a gem-related issue.

The Major Components of the Gem

The honeybadger gem has the following components:

  • Notice — Represents a single exception/error report
  • Backend — Responsible for reporting a Notice to the honeybadger.io API
  • Queue — A first-in-first-out (FIFO) queue which drops items after reaching a maximum size
  • Worker — A single-threaded worker that is responsible for processing Notice items in the Queue and notifying the Backend
  • Initializer — An integration with a detected framework (such as Rails)
  • Plugin — An isolated integration with a Ruby or 3rd party gem feature
  • Config — The user configuration for the gem
  • Agent — An instance of the gem composed of a Config and a Worker (multiple Agents are supported, but are uncommon)
  • Honeybadger — The global singleton Agent

What Happens When Your App Boots

The gem has two modes of booting:

  1. Normal mode: loads Initializers, Config, and Plugins automatically (this is what we'll be discussing here)
  2. Plain Ruby Mode: Skips automatically loading Initializers, Config, and Plugins

If your app is configured to require 'honeybadger' (the default when you install our gem), then it boots in Normal Mode. Here is the order of events in a Rails app:

  1. When 'honeybadger' is required, we immediately:
    1. Detect your framework (Rails, Sinatra, etc.) and load the respective Initializer
    2. Load our Rake Initializer if Rake is present in your application
    3. Install our global at_exit handler
  2. As Rails initializes, our Rack middleware are inserted in the Rails middleware stack via the honeybadger.install_middleware initializer (see our Railtie)
  3. Rails finishes initializing
  4. Honeybadger reads Config from supported sources
  5. Honeybadger loads Plugins
  6. Rails finishes booting

The Life Cycle of an Exception

The honeybadger gem integrates with popular frameworks and libraries to automatically report exceptions when they occur. Examples of where this can happen:

  • Rails and Sinatra requests
  • Background jobs (ActiveJob, Sidekiq, Resque, etc.)
  • Rake tasks
  • Ruby crashes (via our global at_exit handler)

Here is the order of events when an unhandled exception occurs in one of these scenarios:

  1. The exception is reported to the global singleton Agent using Honeybadger.notify
  2. A Notice is built from the exception and any other data passed to Honeybadger.notify, Honeybadger.context, Honeybadger.add_breadcrumb, etc.
  3. Configured before_notify callbacks are executed, passing the Notice to each callback (which may modify it)
  4. If the Notice is ignored via Config, before_notify callbacks, etc., then it's immediately dropped. Otherwise, it's pushed to the Worker.
  5. The Worker processes each Notice in the order that it was added to its Queue. The Queue holds up to 100 Notices by default (this number is configurable via the max_queue_size option). If the number of Notices in the Queue equals the max_queue_size, new Notices are dropped until the number is reduced.
  6. When the Worker processes a notice, it removes it from the Queue, passes it to the Backend, and waits for a response from the honeybadger.io API:
    1. 429, 503 (throttled): Applies an exponential throttle of 1.05. When a throttle is added, the Worker will briefly pause between processing each Notice in the Queue. Additional throttles are added until the server stops throttling the client. Each new throttle multiplies the previous throttle by 1.05; for example, three 429 responses would result in a ~0.158-second pause (((1.05*1.05*1.05)-1)—we subtract 1 to account for the initial throttle).
    2. 402, 403 (payment required/invalid API key): Suspends the Worker for 1 hour. During this time, all Notices are dropped.
    3. 201 (success): if throttled, one throttle per 201 response is removed until the Worker is back to processing the Queue in real-time.

What Happens When Your App Shuts Down

Honeybadger performs the following via our global at_exit handler:

  1. If there is an exception that is crashing the Ruby process, it's reported to Honeybadger.notify, which calls the backend synchronously (it skips the Worker)
  2. The Worker shuts down. By default, it will wait to process remaining exceptions in the Queue. See send_data_at_exit and max_queue_size