Use Vector to ship your systemd logs to Honeybadger Insights

Journald is the logging system used by systemd, the init system used on most modern Linux distributions. It’s a replacement for syslog and rsyslog, and it captures the logs for just about everything running on a Linux server, including services like web and database servers that are managed by systemd. Any systemd-managed process that sends output to stdout will show that output in journald.

You can use Vector to watch journald and relay the events it captures. Here’s a sample configuration that will encode the journald’s data into the newline-delimited JSON format that our API expects:

# Put this in /etc/vector/vector.yaml
sources:
  journald:
    type: journald
    include_matches:
      _TRANSPORT:
        - stdout

# See the Vector Remap Language reference for more info: https://vrl.dev
transforms:
  parse_logs:
    type: "remap"
    inputs: ["journald"]
    source: |
      . = {"host": .host, "unit": ._SYSTEMD_USER_UNIT || ._SYSTEMD_UNIT, "message": .message, "timestamp": .timestamp}
      structured = parse_json(.message) ?? {}
      . = merge!(., structured)

sinks:
  honeybadger:
    type: "http"
    inputs: ["parse_logs"]
    uri: "https://api.honeybadger.io/v1/events"
    request:
      headers:
        X-API-Key: "PROJECT_API_KEY"
    encoding:
      codec: "json"
    framing:
      method: "newline_delimited"
    batch:
      max_bytes: 1000000

Since journald captures everything that happens on your server, and since you probably don’t care about stuff like kernel messages, the sources section of the configuration limits what it will pass on to Honeybadger. This configuration will only relay events that were emitted to stdout, like web server logs, Rails application logs, and that sort of thing. If you really want to send everything that gets logged to journald, you can delete the include_matches portion of the configuration. See the Vector documentation for more info on filtering the journald input.

The parse_logs transformation also reduces the amount of data sent to Insights by stripping out things like the process ID, the user running the service, etc. If you decide you want to be able to query that data in Insights, you can remove the transform and change the honeybadger sink inputs from “parse_logs” to “journald”.

Please see our documentation on ingesting log files for a few more handy transformations you can use in your Vector configuration.

Quick setup method

If you’re running a system that uses apt to manage packages, like Debian or Ubuntu, you can use the following command to fetch and run a script that installs the Vector package, sets up the configuration file, and starts Vector as a service:

curl -sL https://gist.github.com/stympy/9ccb5a809a6731f53b3335fb4e020c2c/raw/bc5741a4e277ea3a7d6d0f5e70a67e0767aec221/install_vector.sh > install_vector.sh && \
chmod a+x install_vector.sh && \
HONEYBADGER_API_KEY="PROJECT_API_KEY" ./install_vector.sh