Project

General

Profile

Actions

Bug #36191

closed

Problem in update from Foreman 3.0 to 3.1

Added by Sergio Traldi about 1 year ago. Updated about 1 year ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Difficulty:
Triaged:
No
Fixed in Releases:
Found in Releases:

Description

I had a Foreman 2.5.4 and I updated it to Foreman 3.0.2-1 in CentOS8Stream host.
I try to update to Foreman 3.1.3-1 using:

dnf update https://yum.theforeman.org/releases/3.1/el8/x86_64/foreman-release.rpm
dnf update
I notice two packages create a new conf file:
   Upgrading        : foreman-proxy-3.0.2-1.el8.noarch                                                                                                 15/35 
warning: /etc/foreman-proxy/settings.d/tftp.yml created as /etc/foreman-proxy/settings.d/tftp.yml.rpmnew

  Upgrading        : foreman-installer-1:3.0.2-2.el8.noarch                                                                                           18/35 
warning: /etc/foreman-installer/scenarios.d/foreman-answers.yaml created as /etc/foreman-installer/scenarios.d/foreman-answers.yaml.rpmnew

The first just has enabled: https (in my original conf)
The second (foreman-answers.yaml) has a lot of differeces

foreman-maintain service stop
foreman-installer

The foreman installer finisch with problem:

# foreman-installer
2023-03-09 14:42:49 [NOTICE] [root] Loading installer configuration. This will take some time.
2023-03-09 14:42:51 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2023-03-09 14:42:51 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2023-03-09 14:42:55 [NOTICE] [configure] Starting system configuration.
2023-03-09 14:43:01 [NOTICE] [configure] 250 configuration steps out of 1440 steps complete.
2023-03-09 14:43:01 [NOTICE] [configure] 500 configuration steps out of 1442 steps complete.
2023-03-09 14:43:01 [NOTICE] [configure] 750 configuration steps out of 1449 steps complete.
2023-03-09 14:43:56 [NOTICE] [configure] 1000 configuration steps out of 1451 steps complete.
2023-03-09 14:44:43 [NOTICE] [configure] 1250 configuration steps out of 1566 steps complete.
2023-03-09 14:44:43 [NOTICE] [configure] 1500 configuration steps out of 1882 steps complete.
2023-03-09 14:45:02 [ERROR ] [configure] Systemd start for dynflow-sidekiq@orchestrator failed!
2023-03-09 14:45:02 [ERROR ] [configure] journalctl log for dynflow-sidekiq@orchestrator:
2023-03-09 14:45:02 [ERROR ] [configure] -- Logs begin at Wed 2023-01-11 09:57:36 CET, end at Thu 2023-03-09 14:45:01 CET. --
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: [259B blob data]
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/ruby/json/common.rb:156:in `parse'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapters/json_common.rb:14:in `load'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapter.rb:21:in `load'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/multi_json-1.14.1/lib/multi_json.rb:122:in `load'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/persistence_adapters/sequel.rb:362:in `load_data'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/persistence_adapters/sequel.rb:243:in `block in find_coordinator_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/persistence_adapters/sequel.rb:243:in `map'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/persistence_adapters/sequel.rb:243:in `find_coordinator_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/coordinator_adapters/sequel.rb:30:in `find_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/coordinator.rb:366:in `find_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/abstract/core.rb:80:in `heartbeat'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/core.rb:36:in `heartbeat'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: [ concurrent-ruby ]
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/abstract/core.rb:113:in `on_message'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: [ concurrent-ruby ]
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:17 cld-config.cloud.pd.infn.it systemd[1]: Stopping Foreman jobs daemon - orchestrator on sidekiq...
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:19 cld-config.cloud.pd.infn.it systemd[1]: dynflow-sidekiq@orchestrator.service: Succeeded.
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:19 cld-config.cloud.pd.infn.it systemd[1]: Stopped Foreman jobs daemon - orchestrator on sidekiq.
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:44:54 cld-config.cloud.pd.infn.it systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:44:55 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: 2023-03-09T13:44:55.045Z 1808932 TID-12shc INFO: GitLab reliable fetch activated!
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:44:55 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: 2023-03-09T13:44:55.046Z 1808932 TID-12sgc INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-1808932", :url=>"redis://localhost:6379/0"}
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: 268 extra bytes after the deserialized object
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `full_unpack'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `load'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/persistence_adapters/sequel.rb:390:in `load_data'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/persistence_adapters/sequel.rb:260:in `block in find_coordinator_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/persistence_adapters/sequel.rb:260:in `map'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/persistence_adapters/sequel.rb:260:in `find_coordinator_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/coordinator_adapters/sequel.rb:30:in `find_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/coordinator.rb:366:in `find_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/coordinator.rb:372:in `find_worlds'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/world/invalidation.rb:131:in `worlds_validity_check'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/world/invalidation.rb:118:in `perform_validity_checks'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/rails.rb:46:in `block in initialize!'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/rails.rb:38:in `tap'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/rails.rb:38:in `initialize!'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/foreman/extras/dynflow-sidekiq.rb:34:in `<top (required)>'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:292:in `boot_system'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/bin/sidekiq:23:in `load'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/bin/sidekiq:23:in `<main>'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it systemd[1]: dynflow-sidekiq@orchestrator.service: Main process exited, code=exited, status=1/FAILURE
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it systemd[1]: dynflow-sidekiq@orchestrator.service: Failed with result 'exit-code'.
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it systemd[1]: Failed to start Foreman jobs daemon - orchestrator on sidekiq.
2023-03-09 14:45:02 [ERROR ] [configure] /Stage[main]/Foreman::Service/Foreman::Dynflow::Worker[orchestrator]/Service[dynflow-sidekiq@orchestrator]/ensure: change from 'stopped' to 'running' failed: Systemd start for dynflow-sidekiq@orchestrator failed!
2023-03-09 14:45:02 [ERROR ] [configure] journalctl log for dynflow-sidekiq@orchestrator:
2023-03-09 14:45:02 [ERROR ] [configure] -- Logs begin at Wed 2023-01-11 09:57:36 CET, end at Thu 2023-03-09 14:45:01 CET. --
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: [259B blob data]
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/ruby/json/common.rb:156:in `parse'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapters/json_common.rb:14:in `load'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapter.rb:21:in `load'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/multi_json-1.14.1/lib/multi_json.rb:122:in `load'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/persistence_adapters/sequel.rb:362:in `load_data'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/persistence_adapters/sequel.rb:243:in `block in find_coordinator_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/persistence_adapters/sequel.rb:243:in `map'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/persistence_adapters/sequel.rb:243:in `find_coordinator_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/coordinator_adapters/sequel.rb:30:in `find_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/coordinator.rb:366:in `find_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/abstract/core.rb:80:in `heartbeat'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/core.rb:36:in `heartbeat'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: [ concurrent-ruby ]
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: /usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/abstract/core.rb:113:in `on_message'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:05 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1801332]: [ concurrent-ruby ]
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:17 cld-config.cloud.pd.infn.it systemd[1]: Stopping Foreman jobs daemon - orchestrator on sidekiq...
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:19 cld-config.cloud.pd.infn.it systemd[1]: dynflow-sidekiq@orchestrator.service: Succeeded.
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:41:19 cld-config.cloud.pd.infn.it systemd[1]: Stopped Foreman jobs daemon - orchestrator on sidekiq.
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:44:54 cld-config.cloud.pd.infn.it systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:44:55 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: 2023-03-09T13:44:55.045Z 1808932 TID-12shc INFO: GitLab reliable fetch activated!
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:44:55 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: 2023-03-09T13:44:55.046Z 1808932 TID-12sgc INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-1808932", :url=>"redis://localhost:6379/0"}
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: 268 extra bytes after the deserialized object
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `full_unpack'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/msgpack-1.3.3/lib/msgpack.rb:37:in `load'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/persistence_adapters/sequel.rb:390:in `load_data'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/persistence_adapters/sequel.rb:260:in `block in find_coordinator_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/persistence_adapters/sequel.rb:260:in `map'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/persistence_adapters/sequel.rb:260:in `find_coordinator_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/coordinator_adapters/sequel.rb:30:in `find_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/coordinator.rb:366:in `find_records'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/coordinator.rb:372:in `find_worlds'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/world/invalidation.rb:131:in `worlds_validity_check'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/world/invalidation.rb:118:in `perform_validity_checks'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/rails.rb:46:in `block in initialize!'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/rails.rb:38:in `tap'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/rails.rb:38:in `initialize!'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/foreman/extras/dynflow-sidekiq.rb:34:in `<top (required)>'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:292:in `boot_system'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/share/gems/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/bin/sidekiq:23:in `load'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it dynflow-sidekiq@orchestrator[1808932]: /usr/bin/sidekiq:23:in `<main>'
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it systemd[1]: dynflow-sidekiq@orchestrator.service: Main process exited, code=exited, status=1/FAILURE
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it systemd[1]: dynflow-sidekiq@orchestrator.service: Failed with result 'exit-code'.
2023-03-09 14:45:02 [ERROR ] [configure] Mar 09 14:45:01 cld-config.cloud.pd.infn.it systemd[1]: Failed to start Foreman jobs daemon - orchestrator on sidekiq.
2023-03-09 14:45:09 [NOTICE] [configure] 1750 configuration steps out of 1882 steps complete.
2023-03-09 14:45:11 [NOTICE] [configure] System configuration has finished.

  There were errors detected during install.
  Please address the errors and re-run the installer to ensure the system is properly configured.
  Failing to do so is likely to result in broken functionality.

  The full log is at /var/log/foreman-installer/foreman.log

I try to debug but I see redis wotk fine e the dynflow-sidekiq@worker is up and running. Just orchestrator has problem.

I try:
  1. redis-cli ping
    PONG
  1. redis-cli INFO | grep ^db
    db0:keys=1202,expires=1198,avg_ttl=132325122195
I tru to restart dynflow-sidekiq@orchestrator but the service crash every time.
In my conf file I found:
  1. grep -r redis /etc/foreman/*
    /etc/foreman/settings.yaml: :redis_url: redis://localhost:6379/6

I try to change the 6 with 0 but the problem is still there.

Now I restore a backup and I downgrade to 3.0.2-1.
I tried to change something in foreman-answers.yaml but with no benefit.

I have a look in the web and I found:

A)
https://community.theforeman.org/t/upgrading-from-foreman-3-0-1-1-to-foreman-3-1-0-1-is-failing/26804

but I ha no problem in foreman-rake db:migrate nor in foreman-rake db:seed

B)
https://access.redhat.com/discussions/6979460

but I have selinux disabled and I try to change cache to false, but without benefit.

If someone have an hint it could be appreciated

Actions #1

Updated by Ewoud Kohl van Wijngaarden about 1 year ago

  • Description updated (diff)

I took the liberty of formatting your message with code tags.

I really wonder how this could happen. It suggests there is some invalid message in the DB, which in this case is PostgreSQL.

Actions #2

Updated by Ewoud Kohl van Wijngaarden about 1 year ago

So I asked around and it turns out this happens in the msgpack migration. Quoting:

it happened when one process would already be restarted, using the "new" format, while there was still a process around using the old one, or something similar to that

This probably happened because you updates all packages (which can restart services) and then stopped all services, while the other way around is the safe order.

It was recommended to restore from backups and try again with the safe order. The alternative is to manually clean up the DB though I don't have exact instructions for that now.

Actions #3

Updated by Sergio Traldi about 1 year ago

Hi,
actually I updated the service in this way:
dnf update -y https://yum.theforeman.org/releases/3.1/el8/x86_64/foreman-release.rpm
dnf update -y
foreman-maintain service stop
foreman-installer

I tried just few minutes ago to do:
foreman-maintain service stop

I check the status:
foreman-maintain service status -b

Then update:
dnf update -y https://yum.theforeman.org/releases/3.1/el8/x86_64/foreman-release.rpm
dnf update -y
foreman-installer

The problem disappeared.
Thanks a lot for the good hint.
Ewoud Kohl van Wijngaarden wrote:

So I asked around and it turns out this happens in the msgpack migration. Quoting:

it happened when one process would already be restarted, using the "new" format, while there was still a process around using the old one, or something similar to that

This probably happened because you updates all packages (which can restart services) and then stopped all services, while the other way around is the safe order.

It was recommended to restore from backups and try again with the safe order. The alternative is to manually clean up the DB though I don't have exact instructions for that now.

Actions #4

Updated by Ewoud Kohl van Wijngaarden about 1 year ago

  • Status changed from New to Closed

Glad to hear.

Actions

Also available in: Atom PDF