Project

General

Profile

Feature #9949

deep merge broken when using facts in hash

Added by Brice Sauvajon over 7 years ago. Updated almost 2 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Parameters
Target version:
-
Difficulty:
Triaged:
No
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:

Description

Since upgrade to 1.7.4, when I use facts as values in hashes, if hashes are merged the yaml generation for the node fails and I can't edit the node in Foreman.
In foreman logs (debug mode), I have the message :

Failed to generate external nodes for XXXXX.XXXXXX.XXXXX.XXXXX.fr with undefined method `each_pair' for #<String:0x00000001fdfdd0>
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/core_ext/hash/deep_merge.rb:15:in `deep_merge!'
/usr/share/foreman/app/services/classification/base.rb:199:in `block in update_hash_matcher'
/usr/share/foreman/app/services/classification/base.rb:194:in `each'
/usr/share/foreman/app/services/classification/base.rb:194:in `update_hash_matcher'
/usr/share/foreman/app/services/classification/base.rb:54:in `block in values_hash'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/relation/delegation.rb:6:in `each'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/relation/delegation.rb:6:in `each'
/usr/share/foreman/app/services/classification/base.rb:40:in `values_hash'
/usr/share/foreman/app/services/classification/class_param.rb:6:in `enc'
/usr/share/foreman/app/models/host/managed.rb:845:in `lookup_keys_class_params'
/usr/share/foreman/app/models/host/managed.rb:386:in `info'
/usr/share/foreman/app/controllers/hosts_controller.rb:172:in `block (2 levels) in externalNodes'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/mime_responds.rb:196:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/mime_responds.rb:196:in `respond_to'
/usr/share/foreman/app/controllers/hosts_controller.rb:170:in `externalNodes'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/abstract_controller/base.rb:167:in `process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/rendering.rb:10:in `process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/abstract_controller/callbacks.rb:18:in `block in process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:549:in `block (3 levels) in _run__1662959498352087448__process_action__946288385558330859__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_7294'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:326:in `around'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:310:in `_callback_around_1843'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_7294'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:526:in `block (2 levels) in _run__1662959498352087448__process_action__946288385558330859__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_7293'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:33:in `clear_thread'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_7293'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:415:in `block in _run__1662959498352087448__process_action__946288385558330859__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_7292'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:326:in `around'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:310:in `_callback_around_13'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_7292'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:403:in `_run__1662959498352087448__process_action__946288385558330859__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/abstract_controller/callbacks.rb:17:in `process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/rescue.rb:29:in `process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/notifications.rb:123:in `block in instrument'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/notifications.rb:123:in `instrument'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/instrumentation.rb:29:in `process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/params_wrapper.rb:207:in `process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/abstract_controller/base.rb:121:in `process'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/abstract_controller/rendering.rb:45:in `process'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal.rb:203:in `dispatch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal.rb:246:in `block in action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/routing/route_set.rb:73:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/routing/route_set.rb:73:in `dispatch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/routing/route_set.rb:36:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'
/usr/share/foreman/vendor/ruby/1.9.1/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/routing/route_set.rb:608:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/apipie-rails-0.2.6/lib/apipie/static_dispatcher.rb:65:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/apipie-rails-0.2.6/lib/apipie/extractor/recorder.rb:97:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/apipie-rails-0.2.6/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/etag.rb:23:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/conditionalget.rb:25:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/head.rb:14:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/params_parser.rb:21:in `call'
/usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:9:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/flash.rb:242:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:210:in `context'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:205:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/cookies.rb:341:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/query_cache.rb:64:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/connection_pool.rb:479:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `_run__2006362726624907426__call__2506133319130815461__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/rack/logger.rb:32:in `call_app'
/usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/rack/logger.rb:16:in `block in call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/tagged_logging.rb:22:in `tagged'
/usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/rack/logger.rb:16:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/request_id.rb:22:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/methodoverride.rb:21:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/runtime.rb:17:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/static.rb:83:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:245:in `fetch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:185:in `lookup'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:66:in `call!'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/engine.rb:484:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/application.rb:231:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/railtie/configurable.rb:30:in `method_missing'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/builder.rb:134:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/urlmap.rb:64:in `block in call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/urlmap.rb:49:in `each'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/urlmap.rb:49:in `call'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/request_handler.rb:96:in `process_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_request_handler.rb:516:in `accept_and_process_next_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/application_spawner.rb:206:in `start_request_handler'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/application_spawner.rb:171:in `block in handle_spawn_application'
/usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:479:in `safe_fork'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/application_spawner.rb:166:in `handle_spawn_application'
/usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
/usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
/usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server.rb:180:in `start'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/application_spawner.rb:129:in `start'
/usr/lib/ruby/vendor_ruby/phusion_passenger/spawn_manager.rb:253:in `block (2 levels) in spawn_rack_application'
/usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server_collection.rb:132:in `lookup_or_add'
/usr/lib/ruby/vendor_ruby/phusion_passenger/spawn_manager.rb:246:in `block in spawn_rack_application'
/usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server_collection.rb:82:in `block in synchronize'
<internal:prelude>:10:in `synchronize'
/usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server_collection.rb:79:in `synchronize'
/usr/lib/ruby/vendor_ruby/phusion_passenger/spawn_manager.rb:244:in `spawn_rack_application'
/usr/lib/ruby/vendor_ruby/phusion_passenger/spawn_manager.rb:137:in `spawn_application'
/usr/lib/ruby/vendor_ruby/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'
/usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
/usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
/usr/share/phusion-passenger/helper-scripts/passenger-spawn-server:99:in `<main>'

For example, for my apt configuration, the "sources" smart variable is :

Matcher : osfamily=debian
Value :
puppetlabs:
  location: http://apt.puppetlabs.com/
  release: <%= @host.facts['lsbdistcodename'] %>
  repos: main dependencies
  include_src: false
Matcher : lsbdistcodename=wheezy
Value :
wheezy-updates:
  location: http://ftp.fr.debian.org/debian/
  release: wheezy-updates
  repos: main
  include_src: false
wheezy:
  location: http://ftp.fr.debian.org/debian/
  release: wheezy
  repos: main contrib non-free
wheezy-security:
  location: http://security.debian.org/
  release: wheezy/updates
  repos: main contrib non-free
  include_src: false
foreman-debug-EBTT5.tar.xz foreman-debug-EBTT5.tar.xz 3.02 MB foreman-debug output Brice Sauvajon, 03/30/2015 10:54 AM

Related issues

Related to Foreman - Feature #3309: Support deep merging of hash structures in smart class parametersClosed2013-10-17
Related to Foreman - Tracker #4470: Usability of parameters and overridesNew

Has duplicate Foreman - Bug #12122: Enabling deep merge causes NoMethodError: undefined method `each_pair'Duplicate2015-10-09

History

#1 Updated by Dominic Cleal over 7 years ago

  • Related to Feature #3309: Support deep merging of hash structures in smart class parameters added

#2 Updated by Marek Hulán over 7 years ago

Thanks for the report. Could you share more information please? How do you use these facts as values? From which version did you upgrade?

#3 Updated by Brice Sauvajon over 7 years ago

There is an example in the report. The apt puppet module, i want to merge hashes to define the apt sources to realize. So for the smart variable apt::sources I have two matchers :

Matcher : osfamily=debian
Value :
puppetlabs:
  location: http://apt.puppetlabs.com/
  release: <%= @host.facts['lsbdistcodename'] %>
  repos: main dependencies
  include_src: false
Matcher : lsbdistcodename=wheezy
Value :
wheezy-updates:
  location: http://ftp.fr.debian.org/debian/
  release: wheezy-updates
  repos: main
  include_src: false
wheezy:
  location: http://ftp.fr.debian.org/debian/
  release: wheezy
  repos: main contrib non-free
wheezy-security:
  location: http://security.debian.org/
  release: wheezy/updates
  repos: main contrib non-free
  include_src: false

For a debian wheezy the resulting value for that smart variable would be :

puppetlabs:
  location: http://apt.puppetlabs.com/
  release: wheezy
  repos: main dependencies
  include_src: false
wheezy-updates:
  location: http://ftp.fr.debian.org/debian/
  release: wheezy-updates
  repos: main
  include_src: false
wheezy:
  location: http://ftp.fr.debian.org/debian/
  release: wheezy
  repos: main contrib non-free
wheezy-security:
  location: http://security.debian.org/
  release: wheezy/updates
  repos: main contrib non-free
  include_src: false

If i replace on the first matcher "<%= @host.facts['lsbdistcodename'] %>" with "wheezy" or if I put eveything in one matcher and uncheck "merge overrides" everything works OK.

I upgraded Foreman from 1.7.2 to 1.7.4, using debian package. When I discovered the problem I tried to downgrade to 1.7.2 but it was still there, maybe a problem with ruby gems or database.

#4 Updated by Marek Hulán over 7 years ago

  • Tracker changed from Bug to Feature

Currently we don't support ERB to be only a part of value. If you want to use ERB it must generate the complete value. Without merging it probably works because the value that contains ERB is not used, since it's overridden by other value (wheezy repos). If you don't mind, I would consider this as a RFE rather than a bug. Also removing Found in release, since it's not specific to any release.

EDIT: this only applies when you merge values that contains ERBs, if you don't merge them, they evaluate fine

#5 Updated by Ori Rabin over 7 years ago

  • Related to Tracker #4470: Usability of parameters and overrides added

#6 Updated by Brice Sauvajon over 7 years ago

OK, thanks for the reply. We have a workaround for our environment so no problem to consider it as an RFE.

#7 Updated by Dominic Cleal almost 7 years ago

  • Has duplicate Bug #12122: Enabling deep merge causes NoMethodError: undefined method `each_pair' added

#8 Updated by Nikolay Miscenkov over 6 years ago

Brice Sauvajon wrote:

OK, thanks for the reply. We have a workaround for our environment so no problem to consider it as an RFE.

Please write, how you solved this problem?

#9 Updated by Nikolay Miscenkov over 6 years ago

Can you tell deadlines to solve the problem? For us it is very important.

#10 Updated by Brice Sauvajon over 6 years ago

Nikolay Miscenkov wrote:

Please write, how you solved this problem?

When we need facts we don't use erb but we insert a special string : facts['factname']
Then on the puppet server we have a custom enc script, which calls the foreman enc script to get the host's yaml and replace these strings by their fact values by doing a request on the puppetdb (but you can do it using foreman api too)

#11 Updated by Dave Garbus over 6 years ago

We are also hitting this after upgrading from 1.7.2 to 1.10.1. In our environment we use hashes to configure our applications, with overrides that get deep merged from a parent hostgroup all the way down an individual host. Several of these layers were making use of ERB templating and are rendered broken after the upgrade.

It looks like the change that broke this behavior was introduced in #8052, even with safe_rendering set to off. In my limited testing, setting the contains_erb method to always return false gets things working again. Of course, this is horribly dirty, so I would appreciate any input on the best way to solve this.

Are you guys open to adding another option to disable the late validation/typecasting on values containing ERB? At the very least, if safe_rendering is disabled, we should fall back to the old behavior.

#12 Updated by Ondřej Pražák over 3 years ago

  • Bugzilla link set to 1582272

#13 Updated by Anthony Chevalet almost 2 years ago

Any update here?

Also available in: Atom PDF