Project

General

Profile

LogstashIntegration » History » Version 2

Version 1 (Lukas Zapletal, 07/22/2015 04:17 AM) → Version 2/6 (Lukas Zapletal, 07/22/2015 04:20 AM)

h1. Logstash Integration

h2. Summary

Foreman has many compoments involved and when Katello plugin is installed with it's
orchestration capabilities as well as other plugins, possibility to effectively collect
logs in a single place with flexible metadata and interface is a big plus.

Logstash open-source project was picked up as the target to integrate with. It is a Ruby
modular framework (jRuby actually) with modular backends and inputs.

https://github.com/elastic/logstash

h2. Owners

* Lukáš Zapletal (Foreman) - `lzap_at_redhat_dot_com`

h2. Current status

* Design done
* Not yet implemented

h2. Implementation

Foreman uses logging gem which is modular log4j-like logging framework with great
extensibility. Although direct integration via logging-logstash should be possible (https://rubygems.org/gems/logging-logstash/versions/0.0.2),
integration via journald would provide better flexibility of logging from all other
components (foreman-proxy, backend systems) or even scripts (seed, migrate, foreman_hooks) with support of all
metadata. There is a POC journald reader for logstash available: https://github.com/stuart-warren/logstash-input-journald

This solution has limitation as it can only be used on systems with journald. For the initial version, this is not a problem and if this feature proves to be useful, we can add a fallback mode via syslog. This will not support custom metadata (see below) tho.


h2. Metadata

Foreman would benefit from the following extra logging fields:

* Session ID: User session - usually a browser session if applicable.

* Transaction/Correlation ID - Usually UUID of the transaction that the logs belong to. This way we can correlate logs from orchestration runs for all services that are under our control: Foreman, Foreman-Proxy, Plugins, Candlepin, Pulp