Project

General

Profile

Actions

Bug #26060

closed

Fact parse error - IPAddr::InvalidAddressError: invalid address

Added by Karl Katzke about 5 years ago. Updated about 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
Facts
Target version:
-
Difficulty:
Triaged:
Yes
Fixed in Releases:
Found in Releases:

Description

When this error happens, load spikes on the server and all of our infrastructure shows as down for a period of time. We're running version 1.20.2 on CentOS 7.4.

2019-02-15T03:39:16 [I|app|53a70] Import facts for 'asswweb061.hahosting.local' completed. Added: 63, Updated: 74, Deleted 72 facts
2019-02-15T03:39:16 [W|app|53a70] Action failed
IPAddr::InvalidAddressError: invalid address
/opt/rh/rh-ruby25/root/usr/share/ruby/ipaddr.rb:649:in `in6_addr'
/opt/rh/rh-ruby25/root/usr/share/ruby/ipaddr.rb:586:in `initialize'
/usr/share/foreman/app/models/subnet.rb:295:in `new'
/usr/share/foreman/app/models/subnet.rb:295:in `subnet_for'
/usr/share/foreman/app/models/host/base.rb:466:in `set_interface'
/usr/share/foreman/app/models/host/base.rb:208:in `block in set_interfaces'
/usr/share/foreman/app/models/host/base.rb:205:in `each'
/usr/share/foreman/app/models/host/base.rb:205:in `set_interfaces'
/usr/share/foreman/app/models/host/base.rb:183:in `populate_fields_from_facts'
/usr/share/foreman/app/models/host/managed.rb:411:in `populate_fields_from_facts'
/usr/share/foreman/app/models/concerns/puppet_host_extensions.rb:3:in `populate_fields_from_facts'
/usr/share/foreman/app/models/host/base.rb:161:in `block in parse_facts'
/usr/share/foreman/app/services/foreman/telemetry_helper.rb:27:in `telemetry_duration_histogram'
/usr/share/foreman/app/models/host/base.rb:160:in `parse_facts'
/usr/share/foreman/app/models/host/base.rb:149:in `import_facts'
/usr/share/foreman/app/models/host/managed.rb:324:in `import_facts'
/usr/share/foreman/app/controllers/api/v2/hosts_controller.rb:288:in `facts'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/usr/share/foreman/lib/middleware/telemetry.rb:9:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/etag.rb:25:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/conditional_get.rb:38:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/head.rb:12:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/usr/share/foreman/lib/middleware/logging_context.rb:23:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:232:in `context'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:226:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:68:in `block in call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `call'
/usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
/usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2019-02-15T03:39:16 [I|app|53a70]   Rendering api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout
2019-02-15T03:39:16 [I|app|53a70]   Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (0.4ms)
2019-02-15T03:39:16 [I|app|53a70] Completed 500 Internal Server Error in 4731ms (Views: 1.4ms | ActiveRecord: 957.3ms)

Related issues 1 (0 open1 closed)

Related to Foreman - Bug #27246: IPAddr::InvalidAddressError on valid link-local address on WindowsClosedEwoud Kohl van WijngaardenActions
Actions #1

Updated by Karl Katzke about 5 years ago

  • Subject changed from Fact parse error to Fact parse error - IPAddr::InvalidAddressError: invalid address
Actions #2

Updated by Karl Katzke about 5 years ago

  • Found in Releases 1.20.1 added
Actions #3

Updated by Lukas Zapletal about 5 years ago

  • Status changed from New to Need more information

Karl, can you share facts in JSON format for the node which failed? Ideally gather them from the host itself.

Actions #4

Updated by Karl Katzke about 5 years ago

Lukas, I'd love to, but it's large and has a lot of information in it that I would need to redact. Is there a specific set of lines that I can provide that would help isolate the issue?

Since this involves ipv6 addresses, the only JSON line that includes anything regarding ipv6 is below.

FWIW, this is a custom fact from a Windows server.

Snippet including the ipv6 address :

"ha_ips_v4_count": "9",
"ha_ips_v6": "fe80::b9c2:f4b5:55d1:9d82",
"ha_livedate": "2012-10-16",
Actions #5

Updated by Karl Katzke about 5 years ago

  • Status changed from Need more information to New
Actions #6

Updated by Karl Katzke about 5 years ago

We found another place where ipv6 is used, and this has an unfamiliar character in it.

        "bindings6": [
          {
            "address": "fe80::b9c2:f4b5:55d1:9d82%14",
            "netmask": "ffff:ffff:ffff:ffff::",
            "network": "fe80::%14" 
          }
        ],
        "ip": "10.89.121.91",
        "ip6": "fe80::b9c2:f4b5:55d1:9d82%14",
        "mac": "00:50:56:B7:51:FB",
        "mtu": 1500,
Actions #7

Updated by Karl Katzke about 5 years ago

Further research indicates that's a Windows link local address. Shouldn't those not be being stored? Could the link-local scope be causing Foreman to bomb out on that address?

Actions #8

Updated by Karl Katzke about 5 years ago

To confirm that it's an issue parsing ipv6 host addresses, we disabled ipv6 on one host (which required a reboot) and it now makes it into Foreman correctly.

Unfortunately, it's not feasible at this time to reboot our entire infrastructure. so we still need a code workaround.

Actions #9

Updated by Lukas Zapletal about 5 years ago

  • Triaged changed from No to Yes

Great you isolated the problem, indeed managing Windows is not our primary focus and there are hidden dragons. Please provide us a valid JSON which we can then put into our test suite, simulate the problem and fix it. Feel free to redact everything except the relevant bits, but it must be a valid JSON.

Feel free to send the unmodified JSON to me directly to lzap at redhat_dot_com and I can try to reproduce, fix and safely delete the data afterwards.

Actions #10

Updated by Lukas Zapletal about 5 years ago

  • Status changed from New to Closed

The JSON data you have provided me is in an invalid encoding. I don't think we will be able to provide a workaround for that in Foreman, this needs to be fixed and workarounded in facter itself.

require "json" 
filepath = "asswweb061.json" 
text = File.read(filepath, mode: "rb", encoding: "UTF-16LE")
json = JSON.parse(text)

JSON::ParserError (765: unexpected token at '{)
  "aio_agent_version": "5.5.7",

I am going to close the ticket, reopen if you have a specific request or solution to the problem.

Actions #11

Updated by Martyn Smith over 4 years ago

  • Related to Bug #27246: IPAddr::InvalidAddressError on valid link-local address on Windows added
Actions

Also available in: Atom PDF