Project

General

Profile

LogstashIntegration » History » Version 4

Lukas Zapletal, 09/02/2015 08:20 AM

1 1 Lukas Zapletal
h1. Logstash Integration
2 1 Lukas Zapletal
3 1 Lukas Zapletal
h2. Summary
4 1 Lukas Zapletal
5 2 Lukas Zapletal
Foreman has many compoments involved and when Katello plugin is installed with it's orchestration capabilities as well as other plugins, possibility to effectively collect logs in a single place with flexible metadata and interface is a big plus.
6 1 Lukas Zapletal
7 2 Lukas Zapletal
Logstash open-source project was picked up as the target to integrate with. It is a Ruby modular framework (jRuby actually) with modular backends and inputs.
8 1 Lukas Zapletal
9 1 Lukas Zapletal
https://github.com/elastic/logstash
10 1 Lukas Zapletal
11 1 Lukas Zapletal
h2. Owners
12 1 Lukas Zapletal
13 1 Lukas Zapletal
* Lukáš Zapletal (Foreman) - `lzap_at_redhat_dot_com`
14 1 Lukas Zapletal
15 1 Lukas Zapletal
h2. Current status
16 1 Lukas Zapletal
17 1 Lukas Zapletal
* Design done
18 1 Lukas Zapletal
* Not yet implemented
19 1 Lukas Zapletal
20 1 Lukas Zapletal
h2. Implementation
21 1 Lukas Zapletal
22 3 Lukas Zapletal
Foreman uses logging gem which is modular log4j-like logging framework with great extensibility. Although direct integration via logging-logstash should be possible (https://rubygems.org/gems/logging-logstash/versions/0.0.2 tries to implement logstash API), integration via journald would provide better flexibility of logging from all other components (foreman-proxy, backend systems) or even scripts (seed, migrate, foreman_hooks) with support of all metadata. There is a POC journald reader for logstash available: https://github.com/stuart-warren/logstash-input-journald.
23 1 Lukas Zapletal
24 4 Lukas Zapletal
Using journald as a mediator opens other possibilities with integration to other (even proprietary) logging solutions for those who need it. And the best thing is it allows to configure Foreman without any additional logging solution, but having all the logs in one place (Foreman server log journal).
25 3 Lukas Zapletal
26 1 Lukas Zapletal
This solution has limitation as it can only be used on systems with journald. For the initial version, this is not a problem and if this feature proves to be useful, we can add a fallback mode via syslog. This will not support custom metadata (see below) tho.
27 3 Lukas Zapletal
28 3 Lukas Zapletal
h2. Required steps
29 3 Lukas Zapletal
30 3 Lukas Zapletal
1. Extend Foreman logging framework with custom metadata (below)
31 3 Lukas Zapletal
1. Send session for each message
32 3 Lukas Zapletal
1. Send corelation ids for Host orchestration
33 3 Lukas Zapletal
1. Implement logging-journald plugin with support of custom metadata (using https://github.com/sandfox-im/journald-logger gem)
34 3 Lukas Zapletal
1. Document how to configure journald with logstash for remote collection
35 3 Lukas Zapletal
36 3 Lukas Zapletal
Optional steps
37 3 Lukas Zapletal
38 3 Lukas Zapletal
1. Hand over correlation id to smart proxy
39 3 Lukas Zapletal
1. Implement logging gem on smart proxy as well
40 3 Lukas Zapletal
1. Configure journald on smart-proxy machines via Puppet to send logs on Foreman node 
41 1 Lukas Zapletal
42 1 Lukas Zapletal
h2. Metadata
43 1 Lukas Zapletal
44 1 Lukas Zapletal
Foreman would benefit from the following extra logging fields:
45 1 Lukas Zapletal
46 1 Lukas Zapletal
* Session ID: User session - usually a browser session if applicable.
47 1 Lukas Zapletal
48 1 Lukas Zapletal
* Transaction/Correlation ID - Usually UUID of the transaction that the logs belong to. This way we can correlate logs from orchestration runs for all services that are under our control: Foreman, Foreman-Proxy, Plugins, Candlepin, Pulp