Use Vector to ship your log files to Honeybadger Insights

You can use Vector to watch your existing log files and send the events they record. Here’s a sample configuration that will encode the log lines into the newline-delimited JSON format that our API expects:

# Put this in /etc/vector/vector.yaml
sources:
  app:
    type: "file"
    include: ["/home/app/shared/log/*.log"]

sinks:
  honeybadger_events:
    type: "http"
    inputs: ["app"]
    uri: "https://api.honeybadger.io/v1/events"
    request:
      headers:
        X-API-Key: "PROJECT_API_KEY"
    encoding:
      codec: "json"
    framing:
      method: "newline_delimited"

If you are using something like Lograge to emit JSON-formatted logs (and you should — it’s awesome), you can have Vector replace the message field with a JSON payload:

# Put this in /etc/vector/vector.yaml
sources:
  app:
    type: "file"
    include: ["/home/app/shared/log/*.log"]

transforms:
  parse_logs:
    type: "remap"
    inputs: ["app"]
    source: |
      payload, err = parse_json(string!(.message))
      if err == null {
        .payload = payload
        del(.message)
      }

sinks:
  honeybadger_events:
    type: "http"
    inputs: ["parse_logs"]
    uri: "https://api.honeybadger.io/v1/events"
    request:
      headers:
        X-API-Key: "PROJECT_API_KEY"
    encoding:
      codec: "json"
    framing:
      method: "newline_delimited"

Or if you are using logfmt-style logs, like “controller=pages action=index”, then you can add a transform that parses that into JSON:

---
transforms:
  parse_logs:
    type: "remap"
    inputs: ["app"]
    source: |
      payload, err = parse_key_value(string!(.message))
      if err == null {
        .payload = payload
        del(.message)
      }

Again, we highly recommend structured logging. 😉

By the way, Vector supports a variety of input sources, such as Docker logs, Redis metrics, etc., in addition to log files. You can define whatever sources and transforms make sense for what you want to capture, then use the sinks section provided in the examples above to send everything to Insights.

The easiest way to run Vector is via Docker. Here’s a sample Docker Compose configuration you can use, assuming your Vector configuration is in a file named vector.yaml:

version: "3.2"
services:
  vector:
    image: timberio/vector:latest-alpine
    volumes:
      - "vector.yaml:/etc/vector/vector.yaml:ro"

Additional Vector configuration examples

Nginx logs

You can use regular expressions to extract the fields of an Nginx log to create a JSON structure in a transform:

sources:
  nginx_logs:
    type: "file"
    ignore_older: 86400
    include:
      - "/var/log/nginx/access.log"
    read_from: "end"

transforms:
  parse_nginx:
    type: "remap"
    inputs:
      - "nginx_logs"
    source: |
      match, err = parse_regex(.message, r'(?P<remote_addr>[^ ]*) - (?P<user>[^ ]*) \[(?P<timestamp>[^\]]*)\] "(?P<method>[^ ]*) ?(?P<url>[^ ]*) ?(?P<protocol>[^"]*)" (?P<status>[^ ]*) (?P<bytes_sent>[^ ]*) "(?P<referer>[^"]*)" "(?P<user_agent>[^"]*)" (?<duration>[0-9\.]+)', true)

      if err == null {
        .remote_addr = match.remote_addr
        .user = match.user
        .timestamp = parse_timestamp(match.timestamp, "%d/%b/%Y:%H:%M:%S %z") ?? match.timestamp
        .request = match.request
        .method = match.method
        .url = match.url
        .protocol = match.protocol
        .status, err = to_int(match.status)
        .bytes_sent, err = to_int(match.bytes_sent)
        .referer = match.referer
        .user_agent = match.user_agent
        .duration, err = to_float(match.duration)
        del(.message)
      } else {
        log("Failed to parse log line: " + err, level: "error")
      }