Bug #10692
closed
foreman-salt reports errors while refreshing salt pillar data for minions
Description
Hello,
I run into errors in the salt-master logs when trying to a refresh_pillar for all minions:
2015-06-03 12:43:35,727 [salt.loaded.int.module.cmdmod][ERROR ]8841 Command '/usr/bin/foreman-node rdsmtl539vcs01.sv.atk' failed with return code: 1
2015-06-03 12:43:35,727 [salt.loaded.int.module.cmdmod][ERROR ]8841 output: Couldn't retrieve ENC data: execution expired
2015-06-03 12:43:35,728 [salt.loaded.int.pillar.puppet][CRITICAL]8841 YAML data from /usr/bin/foreman-node failed to parse
While the production.log reports:
2015-06-03 12:43:38 [E] Invalid IP Address 0 (Net::Validations::Error)
/usr/share/foreman/lib/net/validations.rb:29:in `validate_ip'
/usr/share/foreman/lib/net/dhcp/record.rb:9:in `initialize'
/usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:19:in `new'
/usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:19:in `dhcp_record'
/usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:151:in `dhcp_conflict_detected?'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:464:in `_run__3097717088980338679__validation__3617667107269492963__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:385:in `_run_validation_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activemodel-3.2.21/lib/active_model/validations/callbacks.rb:53:in `run_validations!'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activemodel-3.2.21/lib/active_model/validations.rb:195:in `valid?'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/validations.rb:69:in `valid?'
/usr/share/foreman/app/models/concerns/orchestration.rb:48:in `valid?'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/validations.rb:77:in `perform_validations'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/validations.rb:50:in `save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/attribute_methods/dirty.rb:22:in `save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:259:in `block (2 levels) in save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:313:in `block in with_transaction_returning_status'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:208:in `transaction'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:311:in `with_transaction_returning_status'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:259:in `block in save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:270:in `rollback_active_record_state!'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:258:in `save'
/usr/share/foreman/app/models/concerns/foreman/sti.rb:29:in `save_with_type'
/usr/share/foreman/app/models/host/base.rb:344:in `set_interface'
/usr/share/foreman/app/models/host/base.rb:187:in `block in set_interfaces'
/usr/share/foreman/app/models/host/base.rb:169:in `each'
/usr/share/foreman/app/models/host/base.rb:169:in `set_interfaces'
/usr/share/foreman/app/models/host/base.rb:146:in `populate_fields_from_facts'
/usr/share/foreman/app/models/host/managed.rb:441:in `populate_fields_from_facts'
/usr/share/foreman/app/models/host/base.rb:122:in `import_facts'
/usr/share/foreman/app/models/host/managed.rb:432:in `import_host_and_facts'
/usr/share/foreman/app/controllers/api/v2/hosts_controller.rb:177:in `facts'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/abstract_controller/base.rb:167:in `process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/rendering.rb:10:in `process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/abstract_controller/callbacks.rb:18:in `block in process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:507:in `block (5 levels) in run_1779170634723895284__process_action__5381284405718166__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:215:in `block in conditional_callback_around_7912'
/usr/share/foreman/app/controllers/api/v2/base_controller.rb:151:in `disable_json_root'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_7912'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:484:in `block (4 levels) in _run_1779170634723895284__process_action__5381284405718166__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:215:in `block in conditional_callback_around_7911'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:326:in `around'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:310:in `_callback_around_2823'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_7911'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:472:in `block (3 levels) in _run_1779170634723895284__process_action__5381284405718166__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:215:in `block in conditional_callback_around_7910'
/usr/share/foreman/app/controllers/concerns/application_shared.rb:13:in `set_timezone'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_7910'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:471:in `block (2 levels) in _run_1779170634723895284__process_action__5381284405718166__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:215:in `block in conditional_callback_around_7909'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_7909'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:437:in `block in _run_1779170634723895284__process_action__5381284405718166__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:215:in `block in conditional_callback_around_7908'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:326:in `around'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:310:in `_callback_around_13'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_7908'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:403:in `_run_1779170634723895284__process_action__5381284405718166__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/abstract_controller/callbacks.rb:17:in `process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/rescue.rb:29:in `process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/notifications.rb:123:in `block in instrument'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/notifications.rb:123:in `instrument'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/instrumentation.rb:29:in `process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/params_wrapper.rb:207:in `process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/abstract_controller/base.rb:121:in `process'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/abstract_controller/rendering.rb:45:in `process'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal.rb:203:in `dispatch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal.rb:246:in `block in action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/routing/route_set.rb:73:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/routing/route_set.rb:73:in `dispatch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/routing/route_set.rb:36:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/routing/mapper.rb:43:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'
/usr/share/foreman/vendor/ruby/1.9.1/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/routing/route_set.rb:608:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/apipie-rails-0.2.6/lib/apipie/extractor/recorder.rb:97:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/apipie-rails-0.2.6/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/etag.rb:23:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/conditionalget.rb:35:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/head.rb:14:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/params_parser.rb:21:in `call'
/usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:9:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/flash.rb:242:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:210:in `context'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:205:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/cookies.rb:341:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/query_cache.rb:64:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/connection_pool.rb:479:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `_run__2781054435483615307__call__3617667107269492963__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/rack/logger.rb:32:in `call_app'
/usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/rack/logger.rb:16:in `block in call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/tagged_logging.rb:22:in `tagged'
/usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/rack/logger.rb:16:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/request_id.rb:22:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/methodoverride.rb:21:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/runtime.rb:17:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/lock.rb:15:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/static.rb:83:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:143:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:155:in `invalidate'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:71:in `call!'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/engine.rb:484:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/application.rb:231:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/railtie/configurable.rb:30:in `method_missing'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/builder.rb:134:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/urlmap.rb:64:in `block in call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/urlmap.rb:49:in `each'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/urlmap.rb:49:in `call'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:77:in `process_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:142:in `accept_and_process_next_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:448:in `block (3 levels) in start_threads'
2015-06-03 12:43:38 [I] Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (0.6ms)
2015-06-03 12:43:38 [I] Completed 500 Internal Server Error in 2591.0ms (Views: 1.9ms | ActiveRecord: 31.0ms)
Thanks!
Updated by Dominic Cleal almost 10 years ago
How was the host created? Via the UI or from running Salt on it? It looks like there's an invalid IP address stored on the host somehow, which looks likely to have come from Salt somehow.
Updated by Sébastien Bariteau almost 10 years ago
Dominic Cleal wrote:
How was the host created? Via the UI or from running Salt on it? It looks like there's an invalid IP address stored on the host somehow, which looks likely to have come from Salt somehow.
Via the UI yes.
Updated by Stephen Benjamin almost 10 years ago
Can your try setting 'ignore_puppet_facts_for_provisioning' in Foreman to false, and seeing what happens?
And also provide the output of 'salt-call grains.items' on rdsmtl539vcs01.sv.atk?
Updated by Stephen Benjamin almost 10 years ago
- Project changed from Foreman to Salt
- Status changed from New to Need more information
Updated by Stephen Benjamin almost 10 years ago
Can you provide the output of 'salt-call grains.items' on rdsmtl539vcs01.sv.atk? It would help to understand the interface configuration of the host, and where the bogus IP was coming from.
Updated by Stephen Benjamin over 9 years ago
- Has duplicate Bug #10693: Errors in salt-master logs about foreman-salt when doing pillar refresh on more than 4 minions added
Updated by Anonymous almost 8 years ago
- Status changed from Need more information to Resolved
no reaction, closing.