This effort aims to solve three problems: enriching logging information with context to improve filtering, adding possibility to transfer logs over network (Smart Proxy to Foreman server, Capsule to Satellite etc) and enabling integration with 3rd party logging systems (ELK, systemd, rsyslog, syslog-ng) for easier maintenance.

Currently we are using simple file appenders (production.log, proxy.log). The goal is to use system logging daemon which provides better control (configuration, throttling, alerting, filtering) and log rotation. The best fit for the future is to use native systemd-journald daemon for easy structured logging in foreman core and smart-proxy, backend systems can either use system journal or syslog as a fallback mechanism.

The journald version in RHEL 7.4 (219) is stable and capable enough for all we need, including logs transfer via systemd-journal-gateway which provides SSL secured HTTPS endpoint for remote logs download. System journal is enabled by default on all supported OSes (RHEL, Fedora, Debian/Ubuntu), in the default configuration on RHEL passes data into rsyslog effectively dropping all structured logging but users will be able to change that to leverage all features of system journal (e.g. filter logs by additional flags).

For best benefit of both journald and rsyslog users, in the default configuration Foreman and Proxy will log all data with extra fields but also include the most important fields in log lines in human-readable format for easy grepping. System journal users can still filter logs via native journald utilities while traditional log service users can also correlate data manually via grep.

Users will be able to configure Foreman also with syslog and traditional file appending and rotation when necessary. For this, we will add puppet variables and installer options to support that. Further integration will be possible via 3rd party plugins (ELK) that can pull information from systemd directly via systemd-journal-gateway or via rsyslog which has module to send data into Elasticsearch directly (omelasticsearch). Installer will have options to configure JSON output, but further manual configuration will be needed to get full integration.

Container environment is also important deployment type for the future, in OpenShift v3 environment it is recommended to have all logging on standard output in JSON notation. This is a feature of logging gem, it offers JSON formatter out of box and this will be supported, so Foreman logging stack will be prepared for the future.

The key concept is to have sane default, so users still see all log entries in traditional production.log in human-readable form but after few configuration changes, they can start using structured logging via system journal or grab logs into ELK.