Project

General

Profile

Actions
Copy link

Bug #18401

closed

NoMethodError - undefined method `each' for nil:NilClass on complex tasks

Added by Klaas D over 7 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Difficulty:
Triaged:
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:
Red Hat JIRA:

Description

Hi,
I'm having an issue with complex remote execution tasks. If I run something like "echo test" with the "Run Command - SSH Default" template it works fine, if I run something more complex like upgrade to puppet4 it fails on a lot of hosts with "Error loading data from proxy: NoMethodError - undefined method `each' for nil:NilClass". I'm not sure if this is inside the task/dynflow part or in the remote execution part.

The commands I run:

if [ -e /etc/puppetlabs/puppet/puppet.conf ] ; then exit 0; fi
if ! yum info puppet-agent; then
  exit 1
fi
yum update -y puppet
mv /var/lib/puppet/ssl /etc/puppetlabs/puppet/
sed -i 's#/var/lib/puppet/ssl#/etc/puppetlabs/puppet/ssl#' /etc/puppet/puppet.conf.rpmsave
sed -i 's#/var/lib/puppet#/opt/puppetlabs/puppet/cache#' /etc/puppet/puppet.conf.rpmsave
sed -i 's#/var/run/puppet#/var/run/puppetlabs#' /etc/puppet/puppet.conf.rpmsave
sed -i 's#/var/log/puppet#/var/log/puppetlabs/puppet#' /etc/puppet/puppet.conf.rpmsave
mv -f /etc/puppet/puppet.conf.rpmsave /etc/puppetlabs/puppet/puppet.conf
/opt/puppetlabs/bin/puppet agent -t --no-noop
rm -rf /etc/puppet
rm -rf /var/lib/puppet

Example of a failed task that still returns "success":

Id: 145ee1fa-c808-42c7-b0ea-b3e192ef8a6f
Label: Actions::RemoteExecution::RunHostJob
Name: Remote action:
Owner: admin
Execution type: Delayed
Start at: 2017-02-06 09:47:49 +0100
Start before: -
Started at: 2017-02-06 09:47:49 +0100
Ended at: 2017-02-06 09:49:35 +0100
State: stopped
Result: success
Params: Run #!/bin/bash if [ -e /etc/puppetlabs/puppet/puppet.conf ] ; then exit 0; fi if ! yum info puppet-agent; then exit 1 fi yum update -y puppet mv /var/lib/puppet/ssl /etc/puppetlabs/puppet/ sed -i 's#/var/lib/puppet/ssl#/etc/puppetlabs/puppet/ssl on target.tld

Error loading data from proxy: NoMethodError - undefined method `each' for nil:NilClass

I see errors like this in my smart_proxy_dynflow_core.log that seem like they are related:

[2017-02-06 10:57:48.296 #45700] ERROR -- dynflow: SQLite3::BusyException: database is locked (Sequel::DatabaseError)
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sqlite3-1.3.10/lib/sqlite3/statement.rb:108:in `step'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sqlite3-1.3.10/lib/sqlite3/statement.rb:108:in `block in each'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sqlite3-1.3.10/lib/sqlite3/statement.rb:107:in `loop'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sqlite3-1.3.10/lib/sqlite3/statement.rb:107:in `each'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sqlite3-1.3.10/lib/sqlite3/database.rb:158:in `to_a'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sqlite3-1.3.10/lib/sqlite3/database.rb:158:in `block in execute'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sqlite3-1.3.10/lib/sqlite3/database.rb:95:in `prepare'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sqlite3-1.3.10/lib/sqlite3/database.rb:134:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/sqlite.rb:187:in `block (2 levels) in _execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/logging.rb:33:in `log_yield'
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/sqlite.rb:187:in `block in _execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/connecting.rb:250:in `block in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/connection_pool/threaded.rb:98:in `hold'
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/connecting.rb:250:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/sqlite.rb:178:in `_execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/sqlite.rb:149:in `execute_insert'
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/dataset/actions.rb:927:in `execute_insert'
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/dataset/actions.rb:336:in `insert'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/persistence_adapters/sequel.rb:240:in `block in save'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/persistence_adapters/sequel.rb:326:in `with_retry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/persistence_adapters/sequel.rb:240:in `save'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/persistence_adapters/sequel.rb:80:in `save_execution_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/persistence.rb:51:in `save_execution_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan.rb:338:in `save'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan.rb:185:in `prepare'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/world.rb:167:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/world.rb:166:in `tap'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/world.rb:166:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/world.rb:142:in `trigger'
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow_core-0.1.5/lib/smart_proxy_dynflow_core/helpers.rb:25:in `trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow_core-0.1.5/lib/smart_proxy_dynflow_core/api.rb:16:in `block in <class:Api>'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1610:in `call'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1610:in `block in compile!'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:974:in `[]'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:974:in `block (3 levels) in route!'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:993:in `route_eval'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:974:in `block (2 levels) in route!'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1014:in `block in process_route'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1012:in `catch'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1012:in `process_route'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:972:in `block in route!'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:971:in `each'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:971:in `route!'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1084:in `block in dispatch!'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `block in invoke'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `catch'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `invoke'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1081:in `dispatch!'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:906:in `block in call!'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `block in invoke'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `catch'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `invoke'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:906:in `call!'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:894:in `call'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/nulllogger.rb:9:in `call'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/head.rb:13:in `call'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/show_exceptions.rb:21:in `call'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:181:in `call'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:2021:in `call'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1486:in `block in call'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1795:in `synchronize'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/sinatra-1.4.6/lib/sinatra/base.rb:1486:in `call'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/urlmap.rb:66:in `block in call'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/urlmap.rb:50:in `each'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/urlmap.rb:50:in `call'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/builder.rb:153:in `call'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/handler/webrick.rb:88:in `service'
/opt/rh/rh-ruby22/root/usr/share/ruby/webrick/httpserver.rb:138:in `service'
/opt/rh/rh-ruby22/root/usr/share/ruby/webrick/httpserver.rb:94:in `run'
/opt/rh/rh-ruby22/root/usr/share/ruby/webrick/server.rb:294:in `block in start_thread'
[2017-02-06 10:57:48.296 #45700] ERROR -- dynflow: Persistence retry no. 1

How can I debug this better? Setting loglevel to debug in smart-proxy config didn't show much more.

Greetings
Klaas

Actions
Copy link

Also available in: Atom PDF