Project

General

Profile

Tracker #22569

Unified and structured logging

Added by Lukas Zapletal 10 months ago. Updated 7 months ago.

Status:
New
Priority:
Normal
Category:
Logging
Target version:
-
% Done:

0%

Difficulty:
Triaged:
No
Bugzilla link:
Team Backlog:
Fixed in Releases:
Found in Releases:

Description

This effort aims to solve three problems: enriching logging information with context to improve filtering, adding possibility to transfer logs over network (Smart Proxy to Foreman server, Capsule to Satellite etc) and enabling integration with 3rd party logging systems (ELK, systemd, rsyslog, syslog-ng) for easier maintenance.

Currently we are using simple file appenders (production.log, proxy.log). The goal is to use system logging daemon which provides better control (configuration, throttling, alerting, filtering) and log rotation. The best fit for the future is to use native systemd-journald daemon for easy structured logging in foreman core and smart-proxy, backend systems can either use system journal or syslog as a fallback mechanism.

The journald version in RHEL 7.4 (219) is stable and capable enough for all we need, including logs transfer via systemd-journal-gateway which provides SSL secured HTTPS endpoint for remote logs download. System journal is enabled by default on all supported OSes (RHEL, Fedora, Debian/Ubuntu), in the default configuration on RHEL passes data into rsyslog effectively dropping all structured logging but users will be able to change that to leverage all features of system journal (e.g. filter logs by additional flags).

For best benefit of both journald and rsyslog users, in the default configuration Foreman and Proxy will log all data with extra fields but also include the most important fields in log lines in human-readable format for easy grepping. System journal users can still filter logs via native journald utilities while traditional log service users can also correlate data manually via grep.

Users will be able to configure Foreman also with syslog and traditional file appending and rotation when necessary. For this, we will add puppet variables and installer options to support that. Further integration will be possible via 3rd party plugins (ELK) that can pull information from systemd directly via systemd-journal-gateway or via rsyslog which has module to send data into Elasticsearch directly (omelasticsearch). Installer will have options to configure JSON output, but further manual configuration will be needed to get full integration.

Container environment is also important deployment type for the future, in OpenShift v3 environment it is recommended to have all logging on standard output in JSON notation. This is a feature of logging gem, it offers JSON formatter out of box and this will be supported, so Foreman logging stack will be prepared for the future.

The key concept is to have sane default, so users still see all log entries in traditional production.log in human-readable form but after few configuration changes, they can start using structured logging via system journal or grab logs into ELK.


Related issues

Related to Foreman - Feature #22493: Add new logging options to integrate with logging-journald gemClosed2018-02-02
Related to Installer - Feature #22559: Configure foreman to log to system journalClosed
Related to Installer - Feature #22570: Enable syslog for candlepinNew2018-02-13
Related to Katello - Feature #22296: Send X-Correlation-ID to Pulp REST APINew
Related to Installer - Feature #22584: Create syslog filter for pulpNew2018-02-15
Related to Installer - Feature #22588: Installer support for JSON logging for Foreman and backend systemsNew2018-02-15
Related to Foreman - Bug #23059: Unable to install foreman-journaldClosed2018-03-29
Related to Smart Proxy - Refactor #25100: Replace Ruby logging stack with logging gemClosed

History

#1 Updated by Lukas Zapletal 10 months ago

  • Related to Feature #22493: Add new logging options to integrate with logging-journald gem added

#2 Updated by Lukas Zapletal 10 months ago

  • Related to Feature #22559: Configure foreman to log to system journal added

#3 Updated by Lukas Zapletal 10 months ago

#4 Updated by Lukas Zapletal 10 months ago

  • Related to Feature #22296: Send X-Correlation-ID to Pulp REST API added

#5 Updated by Lukas Zapletal 10 months ago

#6 Updated by Lukas Zapletal 10 months ago

  • Related to Feature #22588: Installer support for JSON logging for Foreman and backend systems added

#7 Updated by Lukas Zapletal 9 months ago

  • Related to Bug #23059: Unable to install foreman-journald added

#8 Updated by Lukas Zapletal 7 months ago

  • Bugzilla link set to 1581719

#9 Updated by Lukas Zapletal 3 months ago

  • Related to Refactor #25100: Replace Ruby logging stack with logging gem added

Also available in: Atom PDF