Actions
Feature #9949
open
deep merge broken when using facts in hash
Status:
New
Priority:
Normal
Assignee:
-
Category:
Parameters
Target version:
-
Description
Since upgrade to 1.7.4, when I use facts as values in hashes, if hashes are merged the yaml generation for the node fails and I can't edit the node in Foreman.
In foreman logs (debug mode), I have the message :
Failed to generate external nodes for XXXXX.XXXXXX.XXXXX.XXXXX.fr with undefined method `each_pair' for #<String:0x00000001fdfdd0> /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/core_ext/hash/deep_merge.rb:15:in `deep_merge!' /usr/share/foreman/app/services/classification/base.rb:199:in `block in update_hash_matcher' /usr/share/foreman/app/services/classification/base.rb:194:in `each' /usr/share/foreman/app/services/classification/base.rb:194:in `update_hash_matcher' /usr/share/foreman/app/services/classification/base.rb:54:in `block in values_hash' /usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/relation/delegation.rb:6:in `each' /usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/relation/delegation.rb:6:in `each' /usr/share/foreman/app/services/classification/base.rb:40:in `values_hash' /usr/share/foreman/app/services/classification/class_param.rb:6:in `enc' /usr/share/foreman/app/models/host/managed.rb:845:in `lookup_keys_class_params' /usr/share/foreman/app/models/host/managed.rb:386:in `info' /usr/share/foreman/app/controllers/hosts_controller.rb:172:in `block (2 levels) in externalNodes' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/mime_responds.rb:196:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/mime_responds.rb:196:in `respond_to' /usr/share/foreman/app/controllers/hosts_controller.rb:170:in `externalNodes' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/implicit_render.rb:4:in `send_action' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/abstract_controller/base.rb:167:in `process_action' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/rendering.rb:10:in `process_action' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/abstract_controller/callbacks.rb:18:in `block in process_action' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:549:in `block (3 levels) in _run__1662959498352087448__process_action__946288385558330859__callbacks' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_7294' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:326:in `around' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:310:in `_callback_around_1843' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_7294' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:526:in `block (2 levels) in _run__1662959498352087448__process_action__946288385558330859__callbacks' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_7293' /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:33:in `clear_thread' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_7293' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:415:in `block in _run__1662959498352087448__process_action__946288385558330859__callbacks' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_7292' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:326:in `around' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:310:in `_callback_around_13' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_7292' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:403:in `_run__1662959498352087448__process_action__946288385558330859__callbacks' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `__run_callback' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:81:in `run_callbacks' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/abstract_controller/callbacks.rb:17:in `process_action' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/rescue.rb:29:in `process_action' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/notifications.rb:123:in `block in instrument' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/notifications/instrumenter.rb:20:in `instrument' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/notifications.rb:123:in `instrument' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/instrumentation.rb:29:in `process_action' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/params_wrapper.rb:207:in `process_action' /usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/railties/controller_runtime.rb:18:in `process_action' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/abstract_controller/base.rb:121:in `process' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/abstract_controller/rendering.rb:45:in `process' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal.rb:203:in `dispatch' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal.rb:246:in `block in action' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/routing/route_set.rb:73:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/routing/route_set.rb:73:in `dispatch' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/routing/route_set.rb:36:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call' /usr/share/foreman/vendor/ruby/1.9.1/gems/journey-1.0.4/lib/journey/router.rb:56:in `each' /usr/share/foreman/vendor/ruby/1.9.1/gems/journey-1.0.4/lib/journey/router.rb:56:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/routing/route_set.rb:608:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/apipie-rails-0.2.6/lib/apipie/static_dispatcher.rb:65:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/apipie-rails-0.2.6/lib/apipie/extractor/recorder.rb:97:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/apipie-rails-0.2.6/lib/apipie/middleware/checksum_in_headers.rb:27:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/etag.rb:23:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/conditionalget.rb:25:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/head.rb:14:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/params_parser.rb:21:in `call' /usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:9:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/flash.rb:242:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:210:in `context' /usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:205:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/cookies.rb:341:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/query_cache.rb:64:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/connection_pool.rb:479:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `_run__2006362726624907426__call__2506133319130815461__callbacks' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `__run_callback' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:385:in `_run_call_callbacks' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:81:in `run_callbacks' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/callbacks.rb:27:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/remote_ip.rb:31:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/rack/logger.rb:32:in `call_app' /usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/rack/logger.rb:16:in `block in call' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/tagged_logging.rb:22:in `tagged' /usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/rack/logger.rb:16:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/request_id.rb:22:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/methodoverride.rb:21:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/runtime.rb:17:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/cache/strategy/local_cache.rb:72:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/static.rb:83:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward' /usr/share/foreman/vendor/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:245:in `fetch' /usr/share/foreman/vendor/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:185:in `lookup' /usr/share/foreman/vendor/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:66:in `call!' /usr/share/foreman/vendor/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/engine.rb:484:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/application.rb:231:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/railtie/configurable.rb:30:in `method_missing' /usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/builder.rb:134:in `call' /usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/urlmap.rb:64:in `block in call' /usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/urlmap.rb:49:in `each' /usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/urlmap.rb:49:in `call' /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/request_handler.rb:96:in `process_request' /usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_request_handler.rb:516:in `accept_and_process_next_request' /usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_request_handler.rb:274:in `main_loop' /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/application_spawner.rb:206:in `start_request_handler' /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/application_spawner.rb:171:in `block in handle_spawn_application' /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:479:in `safe_fork' /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/application_spawner.rb:166:in `handle_spawn_application' /usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server.rb:357:in `server_main_loop' /usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server.rb:206:in `start_synchronously' /usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server.rb:180:in `start' /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/application_spawner.rb:129:in `start' /usr/lib/ruby/vendor_ruby/phusion_passenger/spawn_manager.rb:253:in `block (2 levels) in spawn_rack_application' /usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server_collection.rb:132:in `lookup_or_add' /usr/lib/ruby/vendor_ruby/phusion_passenger/spawn_manager.rb:246:in `block in spawn_rack_application' /usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server_collection.rb:82:in `block in synchronize' <internal:prelude>:10:in `synchronize' /usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server_collection.rb:79:in `synchronize' /usr/lib/ruby/vendor_ruby/phusion_passenger/spawn_manager.rb:244:in `spawn_rack_application' /usr/lib/ruby/vendor_ruby/phusion_passenger/spawn_manager.rb:137:in `spawn_application' /usr/lib/ruby/vendor_ruby/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application' /usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server.rb:357:in `server_main_loop' /usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server.rb:206:in `start_synchronously' /usr/share/phusion-passenger/helper-scripts/passenger-spawn-server:99:in `<main>'
For example, for my apt configuration, the "sources" smart variable is :
Matcher : osfamily=debian Value : puppetlabs: location: http://apt.puppetlabs.com/ release: <%= @host.facts['lsbdistcodename'] %> repos: main dependencies include_src: false Matcher : lsbdistcodename=wheezy Value : wheezy-updates: location: http://ftp.fr.debian.org/debian/ release: wheezy-updates repos: main include_src: false wheezy: location: http://ftp.fr.debian.org/debian/ release: wheezy repos: main contrib non-free wheezy-security: location: http://security.debian.org/ release: wheezy/updates repos: main contrib non-free include_src: false
Files
Updated by Dominic Cleal over 9 years ago
- Related to Feature #3309: Support deep merging of hash structures in smart class parameters added
Updated by Marek Hulán over 9 years ago
Thanks for the report. Could you share more information please? How do you use these facts as values? From which version did you upgrade?
Updated by Brice Sauvajon over 9 years ago
There is an example in the report. The apt puppet module, i want to merge hashes to define the apt sources to realize. So for the smart variable apt::sources I have two matchers :
Matcher : osfamily=debian Value : puppetlabs: location: http://apt.puppetlabs.com/ release: <%= @host.facts['lsbdistcodename'] %> repos: main dependencies include_src: false Matcher : lsbdistcodename=wheezy Value : wheezy-updates: location: http://ftp.fr.debian.org/debian/ release: wheezy-updates repos: main include_src: false wheezy: location: http://ftp.fr.debian.org/debian/ release: wheezy repos: main contrib non-free wheezy-security: location: http://security.debian.org/ release: wheezy/updates repos: main contrib non-free include_src: false
For a debian wheezy the resulting value for that smart variable would be :
puppetlabs: location: http://apt.puppetlabs.com/ release: wheezy repos: main dependencies include_src: false wheezy-updates: location: http://ftp.fr.debian.org/debian/ release: wheezy-updates repos: main include_src: false wheezy: location: http://ftp.fr.debian.org/debian/ release: wheezy repos: main contrib non-free wheezy-security: location: http://security.debian.org/ release: wheezy/updates repos: main contrib non-free include_src: false
If i replace on the first matcher "<%= @host.facts['lsbdistcodename'] %>" with "wheezy" or if I put eveything in one matcher and uncheck "merge overrides" everything works OK.
I upgraded Foreman from 1.7.2 to 1.7.4, using debian package. When I discovered the problem I tried to downgrade to 1.7.2 but it was still there, maybe a problem with ruby gems or database.
Updated by Marek Hulán over 9 years ago
- Tracker changed from Bug to Feature
Currently we don't support ERB to be only a part of value. If you want to use ERB it must generate the complete value. Without merging it probably works because the value that contains ERB is not used, since it's overridden by other value (wheezy repos). If you don't mind, I would consider this as a RFE rather than a bug. Also removing Found in release, since it's not specific to any release.
EDIT: this only applies when you merge values that contains ERBs, if you don't merge them, they evaluate fine
Updated by Ori Rabin over 9 years ago
- Related to Tracker #4470: Usability of parameters and overrides added
Updated by Brice Sauvajon over 9 years ago
OK, thanks for the reply. We have a workaround for our environment so no problem to consider it as an RFE.
Updated by Dominic Cleal about 9 years ago
- Has duplicate Bug #12122: Enabling deep merge causes NoMethodError: undefined method `each_pair' added
Updated by Nikolay Miscenkov almost 9 years ago
Brice Sauvajon wrote:
OK, thanks for the reply. We have a workaround for our environment so no problem to consider it as an RFE.
Please write, how you solved this problem?
Updated by Nikolay Miscenkov almost 9 years ago
Can you tell deadlines to solve the problem? For us it is very important.
Updated by Brice Sauvajon almost 9 years ago
Nikolay Miscenkov wrote:
Please write, how you solved this problem?
When we need facts we don't use erb but we insert a special string : facts['factname']
Then on the puppet server we have a custom enc script, which calls the foreman enc script to get the host's yaml and replace these strings by their fact values by doing a request on the puppetdb (but you can do it using foreman api too)
Updated by Dave Garbus almost 9 years ago
We are also hitting this after upgrading from 1.7.2 to 1.10.1. In our environment we use hashes to configure our applications, with overrides that get deep merged from a parent hostgroup all the way down an individual host. Several of these layers were making use of ERB templating and are rendered broken after the upgrade.
It looks like the change that broke this behavior was introduced in #8052, even with safe_rendering set to off. In my limited testing, setting the contains_erb method to always return false gets things working again. Of course, this is horribly dirty, so I would appreciate any input on the best way to solve this.
Are you guys open to adding another option to disable the late validation/typecasting on values containing ERB? At the very least, if safe_rendering is disabled, we should fall back to the old behavior.
Updated by 
Updated by
Actions