LogstashIntegration » History » Version 3

Version 2 (Lukas Zapletal, 07/22/2015 04:20 AM) → Version 3/6 (Lukas Zapletal, 07/22/2015 04:38 AM)

h1. Logstash Integration

h2. Summary

Foreman has many compoments involved and when Katello plugin is installed with it's orchestration capabilities as well as other plugins, possibility to effectively collect logs in a single place with flexible metadata and interface is a big plus.

Logstash open-source project was picked up as the target to integrate with. It is a Ruby modular framework (jRuby actually) with modular backends and inputs.

h2. Owners

* Lukáš Zapletal (Foreman) - `lzap_at_redhat_dot_com`

h2. Current status

* Design done
* Not yet implemented

h2. Implementation

Foreman uses logging gem which is modular log4j-like logging framework with great extensibility. Although direct integration via logging-logstash should be possible ( tries to implement logstash API), (, integration via journald would provide better flexibility of logging from all other components (foreman-proxy, backend systems) or even scripts (seed, migrate, foreman_hooks) with support of all metadata. There is a POC journald reader for logstash available:

Using journald as a mediator avoids working with logstash completely and opens other possibilities with integration to other (even proprietary) logging solutions for those who need it.

This solution has limitation as it can only be used on systems with journald. For the initial version, this is not a problem and if this feature proves to be useful, we can add a fallback mode via syslog. This will not support custom metadata (see below) tho.

h2. Required steps

1. Extend Foreman logging framework with custom metadata (below)
1. Send session for each message
1. Send corelation ids for Host orchestration
1. Implement logging-journald plugin with support of custom metadata (using gem)
1. Document how to configure journald with logstash for remote collection

Optional steps

1. Hand over correlation id to smart proxy
1. Implement logging gem on smart proxy as well
1. Configure journald on smart-proxy machines via Puppet to send logs on Foreman node

h2. Metadata

Foreman would benefit from the following extra logging fields:

* Session ID: User session - usually a browser session if applicable.

* Transaction/Correlation ID - Usually UUID of the transaction that the logs belong to. This way we can correlate logs from orchestration runs for all services that are under our control: Foreman, Foreman-Proxy, Plugins, Candlepin, Pulp