Actions
Bug #18401
closedNoMethodError - undefined method `each' for nil:NilClass on complex tasks
Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Description
Hi,
I'm having an issue with complex remote execution tasks. If I run something like "echo test" with the "Run Command - SSH Default" template it works fine, if I run something more complex like upgrade to puppet4 it fails on a lot of hosts with "Error loading data from proxy: NoMethodError - undefined method `each' for nil:NilClass". I'm not sure if this is inside the task/dynflow part or in the remote execution part.
The commands I run:
if [ -e /etc/puppetlabs/puppet/puppet.conf ] ; then exit 0; fi if ! yum info puppet-agent; then exit 1 fi yum update -y puppet mv /var/lib/puppet/ssl /etc/puppetlabs/puppet/ sed -i 's#/var/lib/puppet/ssl#/etc/puppetlabs/puppet/ssl#' /etc/puppet/puppet.conf.rpmsave sed -i 's#/var/lib/puppet#/opt/puppetlabs/puppet/cache#' /etc/puppet/puppet.conf.rpmsave sed -i 's#/var/run/puppet#/var/run/puppetlabs#' /etc/puppet/puppet.conf.rpmsave sed -i 's#/var/log/puppet#/var/log/puppetlabs/puppet#' /etc/puppet/puppet.conf.rpmsave mv -f /etc/puppet/puppet.conf.rpmsave /etc/puppetlabs/puppet/puppet.conf /opt/puppetlabs/bin/puppet agent -t --no-noop rm -rf /etc/puppet rm -rf /var/lib/puppet
Example of a failed task that still returns "success":
Id: 145ee1fa-c808-42c7-b0ea-b3e192ef8a6f Label: Actions::RemoteExecution::RunHostJob Name: Remote action: Owner: admin Execution type: Delayed Start at: 2017-02-06 09:47:49 +0100 Start before: - Started at: 2017-02-06 09:47:49 +0100 Ended at: 2017-02-06 09:49:35 +0100 State: stopped Result: success Params: Run #!/bin/bash if [ -e /etc/puppetlabs/puppet/puppet.conf ] ; then exit 0; fi if ! yum info puppet-agent; then exit 1 fi yum update -y puppet mv /var/lib/puppet/ssl /etc/puppetlabs/puppet/ sed -i 's#/var/lib/puppet/ssl#/etc/puppetlabs/puppet/ssl on target.tld Error loading data from proxy: NoMethodError - undefined method `each' for nil:NilClass
I see errors like this in my smart_proxy_dynflow_core.log that seem like they are related:
[2017-02-06 10:57:48.296 #45700] ERROR -- dynflow: SQLite3::BusyException: database is locked (Sequel::DatabaseError) /opt/rh/sclo-ror42/root/usr/share/gems/gems/sqlite3-1.3.10/lib/sqlite3/statement.rb:108:in `step' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sqlite3-1.3.10/lib/sqlite3/statement.rb:108:in `block in each' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sqlite3-1.3.10/lib/sqlite3/statement.rb:107:in `loop' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sqlite3-1.3.10/lib/sqlite3/statement.rb:107:in `each' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sqlite3-1.3.10/lib/sqlite3/database.rb:158:in `to_a' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sqlite3-1.3.10/lib/sqlite3/database.rb:158:in `block in execute' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sqlite3-1.3.10/lib/sqlite3/database.rb:95:in `prepare' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sqlite3-1.3.10/lib/sqlite3/database.rb:134:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/sqlite.rb:187:in `block (2 levels) in _execute' /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/logging.rb:33:in `log_yield' /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/sqlite.rb:187:in `block in _execute' /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/connecting.rb:250:in `block in synchronize' /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/connection_pool/threaded.rb:98:in `hold' /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/connecting.rb:250:in `synchronize' /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/sqlite.rb:178:in `_execute' /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/sqlite.rb:149:in `execute_insert' /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/dataset/actions.rb:927:in `execute_insert' /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/dataset/actions.rb:336:in `insert' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/persistence_adapters/sequel.rb:240:in `block in save' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/persistence_adapters/sequel.rb:326:in `with_retry' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/persistence_adapters/sequel.rb:240:in `save' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/persistence_adapters/sequel.rb:80:in `save_execution_plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/persistence.rb:51:in `save_execution_plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan.rb:338:in `save' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan.rb:185:in `prepare' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/world.rb:167:in `block in plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/world.rb:166:in `tap' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/world.rb:166:in `plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/world.rb:142:in `trigger' /opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow_core-0.1.5/lib/smart_proxy_dynflow_core/helpers.rb:25:in `trigger_task' /opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow_core-0.1.5/lib/smart_proxy_dynflow_core/api.rb:16:in `block in <class:Api>' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1610:in `call' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1610:in `block in compile!' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:974:in `[]' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:974:in `block (3 levels) in route!' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:993:in `route_eval' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:974:in `block (2 levels) in route!' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1014:in `block in process_route' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1012:in `catch' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1012:in `process_route' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:972:in `block in route!' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:971:in `each' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:971:in `route!' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1084:in `block in dispatch!' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `block in invoke' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `catch' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `invoke' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1081:in `dispatch!' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:906:in `block in call!' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `block in invoke' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `catch' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `invoke' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:906:in `call!' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:894:in `call' /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call' /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call' /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call' /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call' /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call' /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call' /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/nulllogger.rb:9:in `call' /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/head.rb:13:in `call' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/show_exceptions.rb:21:in `call' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:181:in `call' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:2021:in `call' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1486:in `block in call' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1795:in `synchronize' /opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1486:in `call' /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/urlmap.rb:66:in `block in call' /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/urlmap.rb:50:in `each' /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/urlmap.rb:50:in `call' /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/builder.rb:153:in `call' /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/handler/webrick.rb:88:in `service' /opt/rh/rh-ruby22/root/usr/share/ruby/webrick/httpserver.rb:138:in `service' /opt/rh/rh-ruby22/root/usr/share/ruby/webrick/httpserver.rb:94:in `run' /opt/rh/rh-ruby22/root/usr/share/ruby/webrick/server.rb:294:in `block in start_thread' [2017-02-06 10:57:48.296 #45700] ERROR -- dynflow: Persistence retry no. 1
How can I debug this better? Setting loglevel to debug in smart-proxy config didn't show much more.
Greetings
Klaas
Actions