Project

General

Profile

Actions

Feature #9949

open

deep merge broken when using facts in hash

Added by Brice Sauvajon over 9 years ago. Updated about 4 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Parameters
Target version:
-
Difficulty:
Triaged:
No
Fixed in Releases:
Found in Releases:

Description

Since upgrade to 1.7.4, when I use facts as values in hashes, if hashes are merged the yaml generation for the node fails and I can't edit the node in Foreman.
In foreman logs (debug mode), I have the message :

Failed to generate external nodes for XXXXX.XXXXXX.XXXXX.XXXXX.fr with undefined method `each_pair' for #<String:0x00000001fdfdd0>
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/core_ext/hash/deep_merge.rb:15:in `deep_merge!'
/usr/share/foreman/app/services/classification/base.rb:199:in `block in update_hash_matcher'
/usr/share/foreman/app/services/classification/base.rb:194:in `each'
/usr/share/foreman/app/services/classification/base.rb:194:in `update_hash_matcher'
/usr/share/foreman/app/services/classification/base.rb:54:in `block in values_hash'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/relation/delegation.rb:6:in `each'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/relation/delegation.rb:6:in `each'
/usr/share/foreman/app/services/classification/base.rb:40:in `values_hash'
/usr/share/foreman/app/services/classification/class_param.rb:6:in `enc'
/usr/share/foreman/app/models/host/managed.rb:845:in `lookup_keys_class_params'
/usr/share/foreman/app/models/host/managed.rb:386:in `info'
/usr/share/foreman/app/controllers/hosts_controller.rb:172:in `block (2 levels) in externalNodes'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/mime_responds.rb:196:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/mime_responds.rb:196:in `respond_to'
/usr/share/foreman/app/controllers/hosts_controller.rb:170:in `externalNodes'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/abstract_controller/base.rb:167:in `process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/rendering.rb:10:in `process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/abstract_controller/callbacks.rb:18:in `block in process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:549:in `block (3 levels) in _run__1662959498352087448__process_action__946288385558330859__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_7294'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:326:in `around'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:310:in `_callback_around_1843'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_7294'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:526:in `block (2 levels) in _run__1662959498352087448__process_action__946288385558330859__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_7293'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:33:in `clear_thread'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_7293'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:415:in `block in _run__1662959498352087448__process_action__946288385558330859__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_7292'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:326:in `around'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:310:in `_callback_around_13'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_7292'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:403:in `_run__1662959498352087448__process_action__946288385558330859__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/abstract_controller/callbacks.rb:17:in `process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/rescue.rb:29:in `process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/notifications.rb:123:in `block in instrument'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/notifications.rb:123:in `instrument'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/instrumentation.rb:29:in `process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/params_wrapper.rb:207:in `process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/abstract_controller/base.rb:121:in `process'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/abstract_controller/rendering.rb:45:in `process'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal.rb:203:in `dispatch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_controller/metal.rb:246:in `block in action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/routing/route_set.rb:73:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/routing/route_set.rb:73:in `dispatch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/routing/route_set.rb:36:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'
/usr/share/foreman/vendor/ruby/1.9.1/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/routing/route_set.rb:608:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/apipie-rails-0.2.6/lib/apipie/static_dispatcher.rb:65:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/apipie-rails-0.2.6/lib/apipie/extractor/recorder.rb:97:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/apipie-rails-0.2.6/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/etag.rb:23:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/conditionalget.rb:25:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/head.rb:14:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/params_parser.rb:21:in `call'
/usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:9:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/flash.rb:242:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:210:in `context'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:205:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/cookies.rb:341:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/query_cache.rb:64:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/connection_pool.rb:479:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `_run__2006362726624907426__call__2506133319130815461__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/rack/logger.rb:32:in `call_app'
/usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/rack/logger.rb:16:in `block in call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/tagged_logging.rb:22:in `tagged'
/usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/rack/logger.rb:16:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/request_id.rb:22:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/methodoverride.rb:21:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/runtime.rb:17:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/actionpack-3.2.21/lib/action_dispatch/middleware/static.rb:83:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:245:in `fetch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:185:in `lookup'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:66:in `call!'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/engine.rb:484:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/application.rb:231:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/railties-3.2.21/lib/rails/railtie/configurable.rb:30:in `method_missing'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/builder.rb:134:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/urlmap.rb:64:in `block in call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/urlmap.rb:49:in `each'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rack-1.4.5/lib/rack/urlmap.rb:49:in `call'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/request_handler.rb:96:in `process_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_request_handler.rb:516:in `accept_and_process_next_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/application_spawner.rb:206:in `start_request_handler'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/application_spawner.rb:171:in `block in handle_spawn_application'
/usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:479:in `safe_fork'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/application_spawner.rb:166:in `handle_spawn_application'
/usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
/usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
/usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server.rb:180:in `start'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/application_spawner.rb:129:in `start'
/usr/lib/ruby/vendor_ruby/phusion_passenger/spawn_manager.rb:253:in `block (2 levels) in spawn_rack_application'
/usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server_collection.rb:132:in `lookup_or_add'
/usr/lib/ruby/vendor_ruby/phusion_passenger/spawn_manager.rb:246:in `block in spawn_rack_application'
/usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server_collection.rb:82:in `block in synchronize'
<internal:prelude>:10:in `synchronize'
/usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server_collection.rb:79:in `synchronize'
/usr/lib/ruby/vendor_ruby/phusion_passenger/spawn_manager.rb:244:in `spawn_rack_application'
/usr/lib/ruby/vendor_ruby/phusion_passenger/spawn_manager.rb:137:in `spawn_application'
/usr/lib/ruby/vendor_ruby/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'
/usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
/usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
/usr/share/phusion-passenger/helper-scripts/passenger-spawn-server:99:in `<main>'

For example, for my apt configuration, the "sources" smart variable is :

Matcher : osfamily=debian
Value :
puppetlabs:
  location: http://apt.puppetlabs.com/
  release: <%= @host.facts['lsbdistcodename'] %>
  repos: main dependencies
  include_src: false
Matcher : lsbdistcodename=wheezy
Value :
wheezy-updates:
  location: http://ftp.fr.debian.org/debian/
  release: wheezy-updates
  repos: main
  include_src: false
wheezy:
  location: http://ftp.fr.debian.org/debian/
  release: wheezy
  repos: main contrib non-free
wheezy-security:
  location: http://security.debian.org/
  release: wheezy/updates
  repos: main contrib non-free
  include_src: false

Files

foreman-debug-EBTT5.tar.xz foreman-debug-EBTT5.tar.xz 3.02 MB foreman-debug output Brice Sauvajon, 03/30/2015 10:54 AM

Related issues 3 (1 open2 closed)

Related to Foreman - Feature #3309: Support deep merging of hash structures in smart class parametersClosedOri Rabin10/17/2013Actions
Related to Foreman - Tracker #4470: Usability of parameters and overridesNew

Actions
Has duplicate Foreman - Bug #12122: Enabling deep merge causes NoMethodError: undefined method `each_pair'Duplicate10/09/2015Actions
Actions #1

Updated by Dominic Cleal over 9 years ago

  • Related to Feature #3309: Support deep merging of hash structures in smart class parameters added
Actions #2

Updated by Marek Hulán over 9 years ago

Thanks for the report. Could you share more information please? How do you use these facts as values? From which version did you upgrade?

Actions #3

Updated by Brice Sauvajon over 9 years ago

There is an example in the report. The apt puppet module, i want to merge hashes to define the apt sources to realize. So for the smart variable apt::sources I have two matchers :

Matcher : osfamily=debian
Value :
puppetlabs:
  location: http://apt.puppetlabs.com/
  release: <%= @host.facts['lsbdistcodename'] %>
  repos: main dependencies
  include_src: false
Matcher : lsbdistcodename=wheezy
Value :
wheezy-updates:
  location: http://ftp.fr.debian.org/debian/
  release: wheezy-updates
  repos: main
  include_src: false
wheezy:
  location: http://ftp.fr.debian.org/debian/
  release: wheezy
  repos: main contrib non-free
wheezy-security:
  location: http://security.debian.org/
  release: wheezy/updates
  repos: main contrib non-free
  include_src: false

For a debian wheezy the resulting value for that smart variable would be :

puppetlabs:
  location: http://apt.puppetlabs.com/
  release: wheezy
  repos: main dependencies
  include_src: false
wheezy-updates:
  location: http://ftp.fr.debian.org/debian/
  release: wheezy-updates
  repos: main
  include_src: false
wheezy:
  location: http://ftp.fr.debian.org/debian/
  release: wheezy
  repos: main contrib non-free
wheezy-security:
  location: http://security.debian.org/
  release: wheezy/updates
  repos: main contrib non-free
  include_src: false

If i replace on the first matcher "<%= @host.facts['lsbdistcodename'] %>" with "wheezy" or if I put eveything in one matcher and uncheck "merge overrides" everything works OK.

I upgraded Foreman from 1.7.2 to 1.7.4, using debian package. When I discovered the problem I tried to downgrade to 1.7.2 but it was still there, maybe a problem with ruby gems or database.

Actions #4

Updated by Marek Hulán over 9 years ago

  • Tracker changed from Bug to Feature

Currently we don't support ERB to be only a part of value. If you want to use ERB it must generate the complete value. Without merging it probably works because the value that contains ERB is not used, since it's overridden by other value (wheezy repos). If you don't mind, I would consider this as a RFE rather than a bug. Also removing Found in release, since it's not specific to any release.

EDIT: this only applies when you merge values that contains ERBs, if you don't merge them, they evaluate fine

Actions #5

Updated by Ori Rabin over 9 years ago

  • Related to Tracker #4470: Usability of parameters and overrides added
Actions #6

Updated by Brice Sauvajon over 9 years ago

OK, thanks for the reply. We have a workaround for our environment so no problem to consider it as an RFE.

Actions #7

Updated by Dominic Cleal about 9 years ago

  • Has duplicate Bug #12122: Enabling deep merge causes NoMethodError: undefined method `each_pair' added
Actions #8

Updated by Nikolay Miscenkov over 8 years ago

Brice Sauvajon wrote:

OK, thanks for the reply. We have a workaround for our environment so no problem to consider it as an RFE.

Please write, how you solved this problem?

Actions #9

Updated by Nikolay Miscenkov over 8 years ago

Can you tell deadlines to solve the problem? For us it is very important.

Actions #10

Updated by Brice Sauvajon over 8 years ago

Nikolay Miscenkov wrote:

Please write, how you solved this problem?

When we need facts we don't use erb but we insert a special string : facts['factname']
Then on the puppet server we have a custom enc script, which calls the foreman enc script to get the host's yaml and replace these strings by their fact values by doing a request on the puppetdb (but you can do it using foreman api too)

Actions #11

Updated by Dave Garbus over 8 years ago

We are also hitting this after upgrading from 1.7.2 to 1.10.1. In our environment we use hashes to configure our applications, with overrides that get deep merged from a parent hostgroup all the way down an individual host. Several of these layers were making use of ERB templating and are rendered broken after the upgrade.

It looks like the change that broke this behavior was introduced in #8052, even with safe_rendering set to off. In my limited testing, setting the contains_erb method to always return false gets things working again. Of course, this is horribly dirty, so I would appreciate any input on the best way to solve this.

Are you guys open to adding another option to disable the late validation/typecasting on values containing ERB? At the very least, if safe_rendering is disabled, we should fall back to the old behavior.

Actions #12

Updated by Ondřej Pražák over 5 years ago

  • Bugzilla link set to 1582272
Actions #13

Updated by Anthony Chevalet about 4 years ago

Any update here?

Actions

Also available in: Atom PDF