Project

General

Profile

LogstashIntegration » History » Version 4

« Previous - Version 4/6 (diff) - Next » - Current version
Lukas Zapletal, 09/02/2015 08:20 AM


Logstash Integration

Summary

Foreman has many compoments involved and when Katello plugin is installed with it's orchestration capabilities as well as other plugins, possibility to effectively collect logs in a single place with flexible metadata and interface is a big plus.

Logstash open-source project was picked up as the target to integrate with. It is a Ruby modular framework (jRuby actually) with modular backends and inputs.

https://github.com/elastic/logstash

Owners

  • Lukáš Zapletal (Foreman) - `lzap_at_redhat_dot_com`

Current status

  • Design done
  • Not yet implemented

Implementation

Foreman uses logging gem which is modular log4j-like logging framework with great extensibility. Although direct integration via logging-logstash should be possible (https://rubygems.org/gems/logging-logstash/versions/0.0.2 tries to implement logstash API), integration via journald would provide better flexibility of logging from all other components (foreman-proxy, backend systems) or even scripts (seed, migrate, foreman_hooks) with support of all metadata. There is a POC journald reader for logstash available: https://github.com/stuart-warren/logstash-input-journald.

Using journald as a mediator opens other possibilities with integration to other (even proprietary) logging solutions for those who need it. And the best thing is it allows to configure Foreman without any additional logging solution, but having all the logs in one place (Foreman server log journal).

This solution has limitation as it can only be used on systems with journald. For the initial version, this is not a problem and if this feature proves to be useful, we can add a fallback mode via syslog. This will not support custom metadata (see below) tho.

Required steps

1. Extend Foreman logging framework with custom metadata (below)
1. Send session for each message
1. Send corelation ids for Host orchestration
1. Implement logging-journald plugin with support of custom metadata (using https://github.com/sandfox-im/journald-logger gem)
1. Document how to configure journald with logstash for remote collection

Optional steps

1. Hand over correlation id to smart proxy
1. Implement logging gem on smart proxy as well
1. Configure journald on smart-proxy machines via Puppet to send logs on Foreman node

Metadata

Foreman would benefit from the following extra logging fields:

  • Session ID: User session - usually a browser session if applicable.
  • Transaction/Correlation ID - Usually UUID of the transaction that the logs belong to. This way we can correlate logs from orchestration runs for all services that are under our control: Foreman, Foreman-Proxy, Plugins, Candlepin, Pulp