Ship your rsyslog logs to Honeybadger Insights
rsyslog is the default syslog daemon on most Linux distributions. You can configure it to forward logs to Honeybadger Insights over syslog-TLS (RFC 5425), tagging each message with your project’s API key in the structured-data section of the RFC 5424 payload.
Requirements
Section titled “Requirements”Install the TLS driver package for rsyslog. On Debian and Ubuntu:
sudo apt-get install rsyslog-gnutlsOn RHEL, Fedora, and derivatives:
sudo dnf install rsyslog-gnutlsYou’ll also need the CA certificate bundle for your system. On Debian/Ubuntu
this is /etc/ssl/certs/ca-certificates.crt. On RHEL/Fedora it’s
/etc/pki/tls/certs/ca-bundle.crt.
Configuration
Section titled “Configuration”# Load the TLS network stream driver. Set the CA file to match your OS:# Debian/Ubuntu: /etc/ssl/certs/ca-certificates.crt# RHEL/Fedora: /etc/pki/tls/certs/ca-bundle.crtglobal(DefaultNetstreamDriver="gtls" DefaultNetstreamDriverCAFile="/etc/ssl/certs/ca-certificates.crt")
# RFC 5424 template with Honeybadger structured datatemplate(name="HoneybadgerFormat" type="string" string="<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% [honeybadger@61642 api_key=\"PROJECT_API_KEY\" event_type=\"rsyslog\"] %msg%\n")
# Forward all logs to Honeybadger over syslog-TLS (RFC 5425)action(type="omfwd" Target="in.honeybadger.io" Port="6514" Protocol="tcp" TCP_Framing="octet-counted" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="*.honeybadger.io" template="HoneybadgerFormat")Restart rsyslog to pick up the change:
sudo systemctl restart rsyslogYou can add additional key/value pairs to the structured-data section of the
template. For example, to tag every event with an environment, replace the
string= value inside the template(name="HoneybadgerFormat" ...) block above
with the following:
string="<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% [honeybadger@61642 api_key=\"PROJECT_API_KEY\" event_type=\"rsyslog\" environment=\"production\"] %msg%\n"Shipping application log files with imfile
Section titled “Shipping application log files with imfile”rsyslog’s imfile module can tail arbitrary log files and feed them through
the same pipeline, which is handy if your application writes to its own log
file instead of stdout. Add the following to the top of
/etc/rsyslog.d/60-honeybadger.conf (before the action(...) block):
# Load the file input modulemodule(load="imfile" PollingInterval="10")
# Tail your application's log filesinput(type="imfile" File="/var/log/myapp/*.log" Tag="myapp" Severity="info" Facility="local7")Each line written to a matching file will be forwarded to Honeybadger using the
HoneybadgerFormat template, with APP-NAME set to the Tag value
(myapp). Adjust File, Tag, Severity, and Facility to match your
application.
If you’d rather only forward the events captured by imfile (and not every
other message rsyslog processes), wrap the action in a conditional:
if ($programname == "myapp") then { action(type="omfwd" Target="in.honeybadger.io" Port="6514" Protocol="tcp" TCP_Framing="octet-counted" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="*.honeybadger.io" template="HoneybadgerFormat")}Querying your data
Section titled “Querying your data”Once your data is flowing, you can query it in Insights using BadgerQL. The following query will return events sent via rsyslog:
fields @ts, hostname::str, appname::str, severity::str, message::str| filter event_type::str == "rsyslog"| sort @tsTroubleshooting
Section titled “Troubleshooting”If events aren’t showing up in Insights, check rsyslog’s own log for TLS or forwarding errors:
sudo journalctl -u rsyslog -fA missing or incorrect CA file is the most common cause of connection failures
— double-check the DefaultNetstreamDriverCAFile path against what’s
installed on your system.