Bug #34394
closedDynflow doesn't properly come back if the DB is unavailable for a brief period of time
Description
Ohai,
when I try to restore a Katello 4.3 (or nightly) on EL8 (with rubygem-foreman_maintain-1.0.2-1.el8.noarch), the restore finishes fine, but afterwards not all services are happy:
# hammer ping database: Status: ok Server Response: Duration: 0ms katello_agent: Status: ok message: 0 Processed, 0 Failed Server Response: Duration: 0ms candlepin: Status: ok Server Response: Duration: 58ms candlepin_auth: Status: ok Server Response: Duration: 50ms candlepin_events: Status: ok message: 0 Processed, 0 Failed Server Response: Duration: 0ms katello_events: Status: ok message: 0 Processed, 0 Failed Server Response: Duration: 1ms pulp3: Status: ok Server Response: Duration: 117ms pulp3_content: Status: ok Server Response: Duration: 128ms foreman_tasks: Status: FAIL Server Response: Message: some executors are not responding, check /foreman_tasks/dynflow/status
After restarting dynflow
via systemctl restart dynflow-sidekiq@\*
everything seems to work fine again.
I am not sure this is a maintain bug (or installer, or dynflow, or packaging), but filing it here for investigation.
Updated by Evgeni Golov almost 3 years ago
Looking at the logs, it seems dynflow was disconnected from the database and didn't manage to reconnect:
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: FATAL: terminating connection due to administrator command Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: #<Thread:0x0000563d5bc0f370 /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true): Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec': PG::UnableToSend: no connection to the server (Sequel::DatabaseDisconnectError) Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `ensure in block (2 levels) in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `block (2 levels) in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: FATAL: terminating connection due to administrator command Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: #<Thread:0x000055ff9be65370 /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true): Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec': PG::UnableToSend: no connection to the server (Sequel::DatabaseDisconnectError) Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `ensure in block (2 levels) in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `block (2 levels) in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: FATAL: terminating connection due to administrator command Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: #<Thread:0x00005577dde812a0 /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true): Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec': PG::UnableToSend: no connection to the server (Sequel::DatabaseDisconnectError) Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `ensure in block (2 levels) in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `block (2 levels) in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec': no connection to the server (PG::UnableToSend) Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `ensure in block (2 levels) in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `block (2 levels) in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:476:in `wait_for_notify': PQconsumeInput() server closed the connection unexpectedly (PG::ConnectionBad) Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: This probably means the server terminated abnormally Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: before or while processing the request. Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:476:in `block (3 levels) in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec': no connection to the server (PG::UnableToSend) Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `ensure in block (2 levels) in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `block (2 levels) in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:476:in `wait_for_notify': PQconsumeInput() server closed the connection unexpectedly (PG::ConnectionBad) Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: This probably means the server terminated abnormally Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: before or while processing the request. Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:476:in `block (3 levels) in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec': no connection to the server (PG::UnableToSend) Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `ensure in block (2 levels) in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `block (2 levels) in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:476:in `wait_for_notify': PQconsumeInput() server closed the connection unexpectedly (PG::ConnectionBad) Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: This probably means the server terminated abnormally Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: before or while processing the request. Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:476:in `block (3 levels) in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:473:in `catch' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:473:in `block (2 levels) in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:473:in `catch' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:473:in `block (2 levels) in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:473:in `catch' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:473:in `block (2 levels) in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start' Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context' Feb 04 13:21:14 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: E, [2022-02-04T13:21:14.195825 #75239] ERROR -- /connector-database-core: Receiving envelopes failed on PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly Feb 04 13:21:14 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: This probably means the server terminated abnormally Feb 04 13:21:14 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: before or while processing the request. Feb 04 13:21:15 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: E, [2022-02-04T13:21:15.758015 #75238] ERROR -- /connector-database-core: Receiving envelopes failed on PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly Feb 04 13:21:15 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: This probably means the server terminated abnormally Feb 04 13:21:15 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: before or while processing the request. Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: E, [2022-02-04T13:21:19.208497 #74683] ERROR -- /connector-database-core: Receiving envelopes failed on PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: This probably means the server terminated abnormally Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: before or while processing the request. Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: E, [2022-02-04T13:21:19.249622 #74683] ERROR -- /parallel-executor-core: PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: This probably means the server terminated abnormally Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: before or while processing the request. Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: (Sequel::DatabaseDisconnectError) Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:500:in `_execute' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:318:in `block (2 levels) in execute' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:318:in `block in execute' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:318:in `execute' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/dataset/actions.rb:1093:in `execute' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:614:in `fetch_rows' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/dataset/actions.rb:152:in `each' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/dataset/actions.rb:51:in `block in all' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/dataset/actions.rb:1010:in `_all' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/dataset/actions.rb:51:in `all' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/persistence_adapters/sequel.rb:260:in `find_coordinator_records' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/coordinator_adapters/sequel.rb:30:in `find_records' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/coordinator.rb:366:in `find_records' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/abstract/core.rb:89:in `heartbeat' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/core.rb:36:in `heartbeat' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: [ concurrent-ruby ] Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/abstract/core.rb:122:in `on_message' Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: [ concurrent-ruby ] Feb 04 13:21:44 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: E, [2022-02-04T13:21:44.197521 #75239] ERROR -- /connector-database-core: Receiving envelopes failed on PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly Feb 04 13:21:44 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: This probably means the server terminated abnormally Feb 04 13:21:44 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: before or while processing the request. Feb 04 13:21:45 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: E, [2022-02-04T13:21:45.760312 #75238] ERROR -- /connector-database-core: Receiving envelopes failed on PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly Feb 04 13:21:45 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: This probably means the server terminated abnormally Feb 04 13:21:45 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: before or while processing the request.
Updated by Evgeni Golov almost 3 years ago
Okay, I can reproduce this without Foreman Maintain and without the Installer on a fresh Katello nightly (and also on 4.2). Just turn off the DB for 60 seconds (a restart is not always sufficient):
[root@centos8-stream-katello-nightly ~]# hammer ping database: Status: ok Server Response: Duration: 0ms candlepin: Status: ok Server Response: Duration: 28ms candlepin_auth: Status: ok Server Response: Duration: 35ms candlepin_events: Status: ok message: 0 Processed, 0 Failed Server Response: Duration: 0ms katello_events: Status: ok message: 0 Processed, 0 Failed Server Response: Duration: 0ms pulp3: Status: ok Server Response: Duration: 76ms pulp3_content: Status: ok Server Response: Duration: 147ms foreman_tasks: Status: ok Server Response: Duration: 3ms [root@centos8-stream-katello-nightly ~]# systemctl stop postgresql; sleep 60; systemctl start postgresql; sleep 10; hammer ping database: Status: ok Server Response: Duration: 0ms candlepin: Status: ok Server Response: Duration: 28ms candlepin_auth: Status: ok Server Response: Duration: 29ms candlepin_events: Status: ok message: 0 Processed, 0 Failed Server Response: Duration: 0ms katello_events: Status: ok message: 0 Processed, 0 Failed Server Response: Duration: 0ms pulp3: Status: ok Server Response: Duration: 37ms pulp3_content: Status: ok Server Response: Duration: 36ms foreman_tasks: Status: FAIL Server Response: Message: some executors are not responding, check /foreman_tasks/dynflow/status
The Katello part is only really needed as Katello exports the status of dynflow/tasks in the Ping API, which Foreman itself does not.
Updated by Evgeni Golov almost 3 years ago
- Project changed from Foreman Maintain to Foreman
- Subject changed from Restore of Katello 4.3 leaves the system in not fully functional state to Dynflow doesn't properly come back if the DB is unavailable for a brief period of time
- Found in Releases 2.5.2, 3.0.1, 3.2.0 added
Updated by Evgeni Golov almost 3 years ago
If you want to repro this without Katello:
- Foreman nightly with foreman_tasks installed
- halt the DB for 60 seconds
- on https://foreman.example.com/foreman_tasks/dynflow/worlds click "check", get "invalid"
Updated by Evgeni Golov almost 3 years ago
for the curious, updating sequel to 5.53.0 didn't magically fix this :/
Updated by Evgeni Golov almost 3 years ago
Updated by Evgeni Golov almost 3 years ago
- systemctl stop postgresql; sleep 30; systemctl start postgresql; sleep 10; hammer ping; sleep 10; hammer ping - first ping thinks it's OK, second FAILED
- systemctl stop postgresql; sleep 20; systemctl start postgresql; sleep 10; hammer ping; sleep 10; hammer ping - those two OK, but later flips to FAILED
- systemctl stop postgresql; sleep 10; systemctl start postgresql; sleep 10; hammer ping; sleep 10; hammer ping - all OK, and later ones are also OK
something is trying hard to survive, but fails at some point
Updated by The Foreman Bot almost 3 years ago
- Status changed from New to Ready For Testing
- Assignee set to Evgeni Golov
- Pull request https://github.com/theforeman/puppet-foreman/pull/1028 added
Updated by Evgeni Golov almost 3 years ago
- Fixed in Releases 3.1.2, 3.2.0 added
Updated by Amit Upadhye almost 3 years ago
- Target version changed from 3.1.2 to 3.1.3
Updated the Target version for this issue to 3.1.3,
<aupadhye> Zhenech, I dont think anything is pending to close https://projects.theforeman.org/issues/34394 ?
<Zhenech> it depends how you look at it
<Zhenech> if you say the redmin is for the workaround, then yes
<Zhenech> if you say it's a bug in dynflow (it is) then no
<aupadhye> then.. i keep it open
Updated by Amit Upadhye almost 3 years ago
- Copied to Bug #34485: Dynflow doesn't properly come back if the DB is unavailable for a brief period of time added
Updated by Amit Upadhye almost 3 years ago
- Project changed from Foreman to Installer
- Category set to Foreman modules
Updated by Amit Upadhye almost 3 years ago
- Status changed from Ready For Testing to Closed
I have created a new issue https://projects.theforeman.org/issues/34485 to fix actual issue in dynflow. Accordingly closing this redmine ticket.
Updated by Ewoud Kohl van Wijngaarden almost 3 years ago
- Target version changed from 3.1.3 to 3.1.2
- Triaged changed from No to Yes