Bug #36191: Problem in update from Foreman 3.0 to 3.1 - Foreman

Actions

Copy link

Bug #36191

closed

Problem in update from Foreman 3.0 to 3.1

Added by Sergio Traldi almost 2 years ago. Updated almost 2 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Category:

Target version:

Difficulty:

Triaged:

Bugzilla link:

Pull request:

Fixed in Releases:

Found in Releases:

Red Hat JIRA:

Description

I had a Foreman 2.5.4 and I updated it to Foreman 3.0.2-1 in CentOS8Stream host.
I try to update to Foreman 3.1.3-1 using:

dnf update https://yum.theforeman.org/releases/3.1/el8/x86_64/foreman-release.rpm
dnf update
I notice two packages create a new conf file:
   Upgrading        : foreman-proxy-3.0.2-1.el8.noarch                                                                                                 15/35 
warning: /etc/foreman-proxy/settings.d/tftp.yml created as /etc/foreman-proxy/settings.d/tftp.yml.rpmnew

  Upgrading        : foreman-installer-1:3.0.2-2.el8.noarch                                                                                           18/35 
warning: /etc/foreman-installer/scenarios.d/foreman-answers.yaml created as /etc/foreman-installer/scenarios.d/foreman-answers.yaml.rpmnew

The first just has enabled: https (in my original conf)
The second (foreman-answers.yaml) has a lot of differeces

foreman-maintain service stop
foreman-installer

The foreman installer finisch with problem:

# foreman-installer
2023-03-09 14:42:49 [NOTICE] [root] Loading installer configuration. This will take some time.
2023-03-09 14:42:51 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2023-03-09 14:42:51 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2023-03-09 14:42:55 [NOTICE] [configure] Starting system configuration.
2023-03-09 14:43:01 [NOTICE] [configure] 250 configuration steps out of 1440 steps complete.
2023-03-09 14:43:01 [NOTICE] [configure] 500 configuration steps out of 1442 steps complete.
2023-03-09 14:43:01 [NOTICE] [configure] 750 configuration steps out of 1449 steps complete.
2023-03-09 14:43:56 [NOTICE] [configure] 1000 configuration steps out of 1451 steps complete.
2023-03-09 14:44:43 [NOTICE] [configure] 1250 configuration steps out of 1566 steps complete.
2023-03-09 14:44:43 [NOTICE] [configure] 1500 configuration steps out of 1882 steps complete.
2023-03-09 14:45:02 [ERROR ] [configure] Systemd start for dynflow-sidekiq@orchestrator failed!
2023-03-09 14:45:02 [ERROR ] [configure] journalctl log for dynflow-sidekiq@orchestrator:
2023-03-09 14:45:02 [ERROR ] [configure] -- Logs begin at Wed 2023-01-11 09:57:36 CET, end at Thu 2023-03-09 14:45:01 CET. --
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: [259B blob data]
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/ruby/json/common.rb:156:in `parse'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapters/json_common.rb:14:in `load'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapter.rb:21:in `load'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/multi_json-1.14.1/lib/multi_json.rb:122:in `load'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/persistence_adapters/sequel.rb:362:in `load_data'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/persistence_adapters/sequel.rb:243:in `block in find_coordinator_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/persistence_adapters/sequel.rb:243:in `map'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/persistence_adapters/sequel.rb:243:in `find_coordinator_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/coordinator_adapters/sequel.rb:30:in `find_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/coordinator.rb:366:in `find_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/abstract/core.rb:80:in `heartbeat'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/core.rb:36:in `heartbeat'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: [ concurrent-ruby ]
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/abstract/core.rb:113:in `on_message'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: [ concurrent-ruby ]
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:17 cld-config.cloud.pd.infn.it systemd[1]: Stopping Foreman jobs daemon - orchestrator on sidekiq...
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:19 cld-config.cloud.pd.infn.it systemd[1]: dynflow-sidekiq@orchestrator.service: Succeeded.
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:19 cld-config.cloud.pd.infn.it systemd[1]: Stopped Foreman jobs daemon - orchestrator on sidekiq.
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:44:54 cld-config.cloud.pd.infn.it systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:44:55 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: 2023-03-09T13:44:55.045Z 1808932 TID-12shc INFO: GitLab reliable fetch activated!
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:44:55 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: 2023-03-09T13:44:55.046Z 1808932 TID-12sgc INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-1808932", :url=>"redis://localhost:6379/0"}
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: 268 extra bytes after the deserialized object
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `full_unpack'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `load'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/persistence_adapters/sequel.rb:390:in `load_data'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/persistence_adapters/sequel.rb:260:in `block in find_coordinator_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/persistence_adapters/sequel.rb:260:in `map'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/persistence_adapters/sequel.rb:260:in `find_coordinator_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/coordinator_adapters/sequel.rb:30:in `find_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/coordinator.rb:366:in `find_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/coordinator.rb:372:in `find_worlds'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/world/invalidation.rb:131:in `worlds_validity_check'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/world/invalidation.rb:118:in `perform_validity_checks'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/rails.rb:46:in `block in initialize!'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/rails.rb:38:in `tap'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/rails.rb:38:in `initialize!'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/foreman/extras/dynflow-sidekiq.rb:34:in `<top (required)>'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:292:in `boot_system'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/bin/sidekiq:23:in `load'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/bin/sidekiq:23:in `<main>'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it systemd[1]: dynflow-sidekiq@orchestrator.service: Main process exited, code=exited, status=1/FAILURE
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it systemd[1]: dynflow-sidekiq@orchestrator.service: Failed with result 'exit-code'.
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it systemd[1]: Failed to start Foreman jobs daemon - orchestrator on sidekiq.
2023-03-09 14:45:02 [ERROR ] [configure] /Stage[main]/Foreman::Service/Foreman::Dynflow::Worker[orchestrator]/Service[dynflow-sidekiq@orchestrator]/ensure: change from 'stopped' to 'running' failed: Systemd start for dynflow-sidekiq@orchestrator failed!
2023-03-09 14:45:02 [ERROR ] [configure] journalctl log for dynflow-sidekiq@orchestrator:
2023-03-09 14:45:02 [ERROR ] [configure] -- Logs begin at Wed 2023-01-11 09:57:36 CET, end at Thu 2023-03-09 14:45:01 CET. --
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: [259B blob data]
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/ruby/json/common.rb:156:in `parse'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapters/json_common.rb:14:in `load'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapter.rb:21:in `load'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/multi_json-1.14.1/lib/multi_json.rb:122:in `load'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/persistence_adapters/sequel.rb:362:in `load_data'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/persistence_adapters/sequel.rb:243:in `block in find_coordinator_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/persistence_adapters/sequel.rb:243:in `map'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/persistence_adapters/sequel.rb:243:in `find_coordinator_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/coordinator_adapters/sequel.rb:30:in `find_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/coordinator.rb:366:in `find_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/abstract/core.rb:80:in `heartbeat'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/core.rb:36:in `heartbeat'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: [ concurrent-ruby ]
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/abstract/core.rb:113:in `on_message'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: [ concurrent-ruby ]
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:17 cld-config.cloud.pd.infn.it systemd[1]: Stopping Foreman jobs daemon - orchestrator on sidekiq...
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:19 cld-config.cloud.pd.infn.it systemd[1]: dynflow-sidekiq@orchestrator.service: Succeeded.
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:19 cld-config.cloud.pd.infn.it systemd[1]: Stopped Foreman jobs daemon - orchestrator on sidekiq.
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:44:54 cld-config.cloud.pd.infn.it systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:44:55 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: 2023-03-09T13:44:55.045Z 1808932 TID-12shc INFO: GitLab reliable fetch activated!
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:44:55 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: 2023-03-09T13:44:55.046Z 1808932 TID-12sgc INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-1808932", :url=>"redis://localhost:6379/0"}
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: 268 extra bytes after the deserialized object
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `full_unpack'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `load'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/persistence_adapters/sequel.rb:390:in `load_data'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/persistence_adapters/sequel.rb:260:in `block in find_coordinator_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/persistence_adapters/sequel.rb:260:in `map'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/persistence_adapters/sequel.rb:260:in `find_coordinator_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/coordinator_adapters/sequel.rb:30:in `find_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/coordinator.rb:366:in `find_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/coordinator.rb:372:in `find_worlds'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/world/invalidation.rb:131:in `worlds_validity_check'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/world/invalidation.rb:118:in `perform_validity_checks'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/rails.rb:46:in `block in initialize!'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/rails.rb:38:in `tap'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/rails.rb:38:in `initialize!'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/foreman/extras/dynflow-sidekiq.rb:34:in `<top (required)>'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:292:in `boot_system'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/bin/sidekiq:23:in `load'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/bin/sidekiq:23:in `<main>'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it systemd[1]: dynflow-sidekiq@orchestrator.service: Main process exited, code=exited, status=1/FAILURE
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it systemd[1]: dynflow-sidekiq@orchestrator.service: Failed with result 'exit-code'.
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it systemd[1]: Failed to start Foreman jobs daemon - orchestrator on sidekiq.
2023-03-09 14:45:09 [NOTICE] [configure] 1750 configuration steps out of 1882 steps complete.
2023-03-09 14:45:11 [NOTICE] [configure] System configuration has finished.

  There were errors detected during install.
  Please address the errors and re-run the installer to ensure the system is properly configured.
  Failing to do so is likely to result in broken functionality.

  The full log is at /var/log/foreman-installer/foreman.log

I try to debug but I see redis wotk fine e the dynflow-sidekiq@worker is up and running. Just orchestrator has problem.

I try:

redis-cli ping
PONG

redis-cli INFO | grep ^db
db0:keys=1202,expires=1198,avg_ttl=132325122195

I tru to restart dynflow-sidekiq@orchestrator but the service crash every time.
In my conf file I found:

grep -r redis /etc/foreman/*
/etc/foreman/settings.yaml: :redis_url: redis://localhost:6379/6

I try to change the 6 with 0 but the problem is still there.

Now I restore a backup and I downgrade to 3.0.2-1.
I tried to change something in foreman-answers.yaml but with no benefit.

I have a look in the web and I found:

A)
https://community.theforeman.org/t/upgrading-from-foreman-3-0-1-1-to-foreman-3-1-0-1-is-failing/26804

but I ha no problem in foreman-rake db:migrate nor in foreman-rake db:seed

B)
https://access.redhat.com/discussions/6979460

but I have selinux disabled and I try to change cache to false, but without benefit.

If someone have an hint it could be appreciated

Actions

Copy link

Updated by Ewoud Kohl van Wijngaarden almost 2 years ago

Description updated (diff)

I took the liberty of formatting your message with code tags.

I really wonder how this could happen. It suggests there is some invalid message in the DB, which in this case is PostgreSQL.

Actions

Copy link

Updated by Ewoud Kohl van Wijngaarden almost 2 years ago

So I asked around and it turns out this happens in the msgpack migration. Quoting:

it happened when one process would already be restarted, using the "new" format, while there was still a process around using the old one, or something similar to that

This probably happened because you updates all packages (which can restart services) and then stopped all services, while the other way around is the safe order.

It was recommended to restore from backups and try again with the safe order. The alternative is to manually clean up the DB though I don't have exact instructions for that now.

Actions

Copy link

Updated by Sergio Traldi almost 2 years ago

Hi,
actually I updated the service in this way:
dnf update -y https://yum.theforeman.org/releases/3.1/el8/x86_64/foreman-release.rpm
dnf update -y
foreman-maintain service stop
foreman-installer

I tried just few minutes ago to do:
foreman-maintain service stop

I check the status:
foreman-maintain service status -b

Then update:
dnf update -y https://yum.theforeman.org/releases/3.1/el8/x86_64/foreman-release.rpm
dnf update -y
foreman-installer

The problem disappeared.
Thanks a lot for the good hint.
Ewoud Kohl van Wijngaarden wrote:

So I asked around and it turns out this happens in the msgpack migration. Quoting:

it happened when one process would already be restarted, using the "new" format, while there was still a process around using the old one, or something similar to that

This probably happened because you updates all packages (which can restart services) and then stopped all services, while the other way around is the safe order.

It was recommended to restore from backups and try again with the safe order. The alternative is to manually clean up the DB though I don't have exact instructions for that now.

Actions

Copy link