Project

General

Profile

LogstashIntegration » History » Version 3

Lukas Zapletal, 07/22/2015 04:38 AM
Added steps

1 1 Lukas Zapletal
h1. Logstash Integration
2 1 Lukas Zapletal
3 1 Lukas Zapletal
h2. Summary
4 1 Lukas Zapletal
5 2 Lukas Zapletal
Foreman has many compoments involved and when Katello plugin is installed with it's orchestration capabilities as well as other plugins, possibility to effectively collect logs in a single place with flexible metadata and interface is a big plus.
6 1 Lukas Zapletal
7 2 Lukas Zapletal
Logstash open-source project was picked up as the target to integrate with. It is a Ruby modular framework (jRuby actually) with modular backends and inputs.
8 1 Lukas Zapletal
9 1 Lukas Zapletal
https://github.com/elastic/logstash
10 1 Lukas Zapletal
11 1 Lukas Zapletal
h2. Owners
12 1 Lukas Zapletal
13 1 Lukas Zapletal
* Lukáš Zapletal (Foreman) - `lzap_at_redhat_dot_com`
14 1 Lukas Zapletal
15 1 Lukas Zapletal
h2. Current status
16 1 Lukas Zapletal
17 1 Lukas Zapletal
* Design done
18 1 Lukas Zapletal
* Not yet implemented
19 1 Lukas Zapletal
20 1 Lukas Zapletal
h2. Implementation
21 1 Lukas Zapletal
22 3 Lukas Zapletal
Foreman uses logging gem which is modular log4j-like logging framework with great extensibility. Although direct integration via logging-logstash should be possible (https://rubygems.org/gems/logging-logstash/versions/0.0.2 tries to implement logstash API), integration via journald would provide better flexibility of logging from all other components (foreman-proxy, backend systems) or even scripts (seed, migrate, foreman_hooks) with support of all metadata. There is a POC journald reader for logstash available: https://github.com/stuart-warren/logstash-input-journald.
23 1 Lukas Zapletal
24 3 Lukas Zapletal
Using journald as a mediator avoids working with logstash completely and opens other possibilities with integration to other (even proprietary) logging solutions for those who need it.
25 3 Lukas Zapletal
26 1 Lukas Zapletal
This solution has limitation as it can only be used on systems with journald. For the initial version, this is not a problem and if this feature proves to be useful, we can add a fallback mode via syslog. This will not support custom metadata (see below) tho.
27 3 Lukas Zapletal
28 3 Lukas Zapletal
h2. Required steps
29 3 Lukas Zapletal
30 3 Lukas Zapletal
1. Extend Foreman logging framework with custom metadata (below)
31 3 Lukas Zapletal
1. Send session for each message
32 3 Lukas Zapletal
1. Send corelation ids for Host orchestration
33 3 Lukas Zapletal
1. Implement logging-journald plugin with support of custom metadata (using https://github.com/sandfox-im/journald-logger gem)
34 3 Lukas Zapletal
1. Document how to configure journald with logstash for remote collection
35 3 Lukas Zapletal
36 3 Lukas Zapletal
Optional steps
37 3 Lukas Zapletal
38 3 Lukas Zapletal
1. Hand over correlation id to smart proxy
39 3 Lukas Zapletal
1. Implement logging gem on smart proxy as well
40 3 Lukas Zapletal
1. Configure journald on smart-proxy machines via Puppet to send logs on Foreman node 
41 1 Lukas Zapletal
42 1 Lukas Zapletal
h2. Metadata
43 1 Lukas Zapletal
44 1 Lukas Zapletal
Foreman would benefit from the following extra logging fields:
45 1 Lukas Zapletal
46 1 Lukas Zapletal
* Session ID: User session - usually a browser session if applicable.
47 1 Lukas Zapletal
48 1 Lukas Zapletal
* Transaction/Correlation ID - Usually UUID of the transaction that the logs belong to. This way we can correlate logs from orchestration runs for all services that are under our control: Foreman, Foreman-Proxy, Plugins, Candlepin, Pulp