Actions
Bug #26060
closed
Fact parse error - IPAddr::InvalidAddressError: invalid address
Status:
Closed
Priority:
Normal
Assignee:
-
Category:
Facts
Target version:
-
Description
When this error happens, load spikes on the server and all of our infrastructure shows as down for a period of time. We're running version 1.20.2 on CentOS 7.4.
2019-02-15T03:39:16 [I|app|53a70] Import facts for 'asswweb061.hahosting.local' completed. Added: 63, Updated: 74, Deleted 72 facts 2019-02-15T03:39:16 [W|app|53a70] Action failed IPAddr::InvalidAddressError: invalid address /opt/rh/rh-ruby25/root/usr/share/ruby/ipaddr.rb:649:in `in6_addr' /opt/rh/rh-ruby25/root/usr/share/ruby/ipaddr.rb:586:in `initialize' /usr/share/foreman/app/models/subnet.rb:295:in `new' /usr/share/foreman/app/models/subnet.rb:295:in `subnet_for' /usr/share/foreman/app/models/host/base.rb:466:in `set_interface' /usr/share/foreman/app/models/host/base.rb:208:in `block in set_interfaces' /usr/share/foreman/app/models/host/base.rb:205:in `each' /usr/share/foreman/app/models/host/base.rb:205:in `set_interfaces' /usr/share/foreman/app/models/host/base.rb:183:in `populate_fields_from_facts' /usr/share/foreman/app/models/host/managed.rb:411:in `populate_fields_from_facts' /usr/share/foreman/app/models/concerns/puppet_host_extensions.rb:3:in `populate_fields_from_facts' /usr/share/foreman/app/models/host/base.rb:161:in `block in parse_facts' /usr/share/foreman/app/services/foreman/telemetry_helper.rb:27:in `telemetry_duration_histogram' /usr/share/foreman/app/models/host/base.rb:160:in `parse_facts' /usr/share/foreman/app/models/host/base.rb:149:in `import_facts' /usr/share/foreman/app/models/host/managed.rb:324:in `import_facts' /usr/share/foreman/app/controllers/api/v2/hosts_controller.rb:288:in `facts' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks' /usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks' /usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks' /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks' /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks' /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks' /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call' /usr/share/foreman/lib/middleware/telemetry.rb:9:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call' /usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/tempfile_reaper.rb:15:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/etag.rb:25:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/conditional_get.rb:38:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/head.rb:12:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call' /usr/share/foreman/lib/middleware/logging_context.rb:23:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:232:in `context' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:226:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:68:in `block in call' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `each' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `call' /usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request' /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request' /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop' /usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context' 2019-02-15T03:39:16 [I|app|53a70] Rendering api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout 2019-02-15T03:39:16 [I|app|53a70] Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (0.4ms) 2019-02-15T03:39:16 [I|app|53a70] Completed 500 Internal Server Error in 4731ms (Views: 1.4ms | ActiveRecord: 957.3ms)
Updated by Karl Katzke almost 6 years ago
- Subject changed from Fact parse error to Fact parse error - IPAddr::InvalidAddressError: invalid address
Updated by Lukas Zapletal almost 6 years ago
- Status changed from New to Need more information
Karl, can you share facts in JSON format for the node which failed? Ideally gather them from the host itself.
Updated by Karl Katzke almost 6 years ago
Lukas, I'd love to, but it's large and has a lot of information in it that I would need to redact. Is there a specific set of lines that I can provide that would help isolate the issue?
Since this involves ipv6 addresses, the only JSON line that includes anything regarding ipv6 is below.
FWIW, this is a custom fact from a Windows server.
Snippet including the ipv6 address :
"ha_ips_v4_count": "9",
"ha_ips_v6": "fe80::b9c2:f4b5:55d1:9d82",
"ha_livedate": "2012-10-16",
Updated by Karl Katzke almost 6 years ago
- Status changed from Need more information to New
Updated by Karl Katzke almost 6 years ago
We found another place where ipv6 is used, and this has an unfamiliar character in it.
"bindings6": [
{
"address": "fe80::b9c2:f4b5:55d1:9d82%14",
"netmask": "ffff:ffff:ffff:ffff::",
"network": "fe80::%14"
}
],
"ip": "10.89.121.91",
"ip6": "fe80::b9c2:f4b5:55d1:9d82%14",
"mac": "00:50:56:B7:51:FB",
"mtu": 1500,
Updated by Karl Katzke almost 6 years ago
Further research indicates that's a Windows link local address. Shouldn't those not be being stored? Could the link-local scope be causing Foreman to bomb out on that address?
Updated by Karl Katzke almost 6 years ago
To confirm that it's an issue parsing ipv6 host addresses, we disabled ipv6 on one host (which required a reboot) and it now makes it into Foreman correctly.
Unfortunately, it's not feasible at this time to reboot our entire infrastructure. so we still need a code workaround.
Updated by Lukas Zapletal almost 6 years ago
- Triaged changed from No to Yes
Great you isolated the problem, indeed managing Windows is not our primary focus and there are hidden dragons. Please provide us a valid JSON which we can then put into our test suite, simulate the problem and fix it. Feel free to redact everything except the relevant bits, but it must be a valid JSON.
Feel free to send the unmodified JSON to me directly to lzap at redhat_dot_com and I can try to reproduce, fix and safely delete the data afterwards.
Updated by Lukas Zapletal almost 6 years ago
- Status changed from New to Closed
The JSON data you have provided me is in an invalid encoding. I don't think we will be able to provide a workaround for that in Foreman, this needs to be fixed and workarounded in facter itself.
require "json"
filepath = "asswweb061.json"
text = File.read(filepath, mode: "rb", encoding: "UTF-16LE")
json = JSON.parse(text)
JSON::ParserError (765: unexpected token at '{)
"aio_agent_version": "5.5.7",
I am going to close the ticket, reopen if you have a specific request or solution to the problem.
Updated by Martyn Smith over 5 years ago
- Related to Bug #27246: IPAddr::InvalidAddressError on valid link-local address on Windows added
Actions