Project

General

Profile

Bug #34394

Dynflow doesn't properly come back if the DB is unavailable for a brief period of time

Added by Evgeni Golov 10 months ago. Updated 9 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Foreman modules
Target version:
Difficulty:
Triaged:
Yes
Bugzilla link:
Fixed in Releases:
Red Hat JIRA:

Description

Ohai,

when I try to restore a Katello 4.3 (or nightly) on EL8 (with rubygem-foreman_maintain-1.0.2-1.el8.noarch), the restore finishes fine, but afterwards not all services are happy:

# hammer ping
database:         
    Status:          ok
    Server Response: Duration: 0ms
katello_agent:    
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
candlepin:        
    Status:          ok
    Server Response: Duration: 58ms
candlepin_auth:   
    Status:          ok
    Server Response: Duration: 50ms
candlepin_events: 
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
katello_events:   
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 1ms
pulp3:            
    Status:          ok
    Server Response: Duration: 117ms
pulp3_content:    
    Status:          ok
    Server Response: Duration: 128ms
foreman_tasks:    
    Status:          FAIL
    Server Response: Message: some executors are not responding, check /foreman_tasks/dynflow/status

After restarting dynflow via systemctl restart dynflow-sidekiq@\* everything seems to work fine again.

I am not sure this is a maintain bug (or installer, or dynflow, or packaging), but filing it here for investigation.


Related issues

Copied to foreman-tasks - Bug #34485: Dynflow doesn't properly come back if the DB is unavailable for a brief period of timeNew

Associated revisions

Revision f44d723e (diff)
Added by Evgeni Golov 10 months ago

Refs #34394 - trigger dynflow restart when DB restarts

When PostgreSQL is restarted (or worse, unavailable for a while),
Dynflow doesn't always cope well with this fact and sometimes ends in a
bad state, not able to recover itself.

This is clearly not fixing the underlying Dynflow issue, but at least
lets restart Dynflow in the cases we know the DB dropped connections.

History

#1 Updated by Evgeni Golov 10 months ago

Looking at the logs, it seems dynflow was disconnected from the database and didn't manage to reconnect:

Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: FATAL:  terminating connection due to administrator command
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: #<Thread:0x0000563d5bc0f370 /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true):
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec': PG::UnableToSend: no connection to the server (Sequel::DatabaseDisconnectError)
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `ensure in block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: FATAL:  terminating connection due to administrator command
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: #<Thread:0x000055ff9be65370 /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true):
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec': PG::UnableToSend: no connection to the server (Sequel::DatabaseDisconnectError)
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `ensure in block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: FATAL:  terminating connection due to administrator command
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: #<Thread:0x00005577dde812a0 /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true):
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec': PG::UnableToSend: no connection to the server (Sequel::DatabaseDisconnectError)
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `ensure in block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec': no connection to the server (PG::UnableToSend)
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `ensure in block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:476:in `wait_for_notify': PQconsumeInput() server closed the connection unexpectedly (PG::ConnectionBad)
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         This probably means the server terminated abnormally
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         before or while processing the request.
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:476:in `block (3 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec': no connection to the server (PG::UnableToSend)
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `ensure in block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:476:in `wait_for_notify': PQconsumeInput() server closed the connection unexpectedly (PG::ConnectionBad)
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         This probably means the server terminated abnormally
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         before or while processing the request.
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:476:in `block (3 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec': no connection to the server (PG::UnableToSend)
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `ensure in block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:476:in `wait_for_notify': PQconsumeInput() server closed the connection unexpectedly (PG::ConnectionBad)
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         This probably means the server terminated abnormally
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         before or while processing the request.
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:476:in `block (3 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:473:in `catch'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:473:in `block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:473:in `catch'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:473:in `block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:473:in `catch'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:473:in `block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Feb 04 13:21:14 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: E, [2022-02-04T13:21:14.195825 #75239] ERROR -- /connector-database-core: Receiving envelopes failed on PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
Feb 04 13:21:14 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         This probably means the server terminated abnormally
Feb 04 13:21:14 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         before or while processing the request.
Feb 04 13:21:15 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: E, [2022-02-04T13:21:15.758015 #75238] ERROR -- /connector-database-core: Receiving envelopes failed on PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
Feb 04 13:21:15 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         This probably means the server terminated abnormally
Feb 04 13:21:15 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         before or while processing the request.
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: E, [2022-02-04T13:21:19.208497 #74683] ERROR -- /connector-database-core: Receiving envelopes failed on PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         This probably means the server terminated abnormally
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         before or while processing the request.
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: E, [2022-02-04T13:21:19.249622 #74683] ERROR -- /parallel-executor-core: PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         This probably means the server terminated abnormally
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         before or while processing the request.
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:  (Sequel::DatabaseDisconnectError)
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:500:in `_execute'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:318:in `block (2 levels) in execute'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:318:in `block in execute'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:318:in `execute'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/dataset/actions.rb:1093:in `execute'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:614:in `fetch_rows'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/dataset/actions.rb:152:in `each'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/dataset/actions.rb:51:in `block in all'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/dataset/actions.rb:1010:in `_all'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/dataset/actions.rb:51:in `all'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/persistence_adapters/sequel.rb:260:in `find_coordinator_records'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/coordinator_adapters/sequel.rb:30:in `find_records'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/coordinator.rb:366:in `find_records'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/abstract/core.rb:89:in `heartbeat'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/core.rb:36:in `heartbeat'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: [ concurrent-ruby ]
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/abstract/core.rb:122:in `on_message'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: [ concurrent-ruby ]
Feb 04 13:21:44 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: E, [2022-02-04T13:21:44.197521 #75239] ERROR -- /connector-database-core: Receiving envelopes failed on PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
Feb 04 13:21:44 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         This probably means the server terminated abnormally
Feb 04 13:21:44 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         before or while processing the request.
Feb 04 13:21:45 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: E, [2022-02-04T13:21:45.760312 #75238] ERROR -- /connector-database-core: Receiving envelopes failed on PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
Feb 04 13:21:45 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         This probably means the server terminated abnormally
Feb 04 13:21:45 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         before or while processing the request.

#2 Updated by Evgeni Golov 10 months ago

Okay, I can reproduce this without Foreman Maintain and without the Installer on a fresh Katello nightly (and also on 4.2). Just turn off the DB for 60 seconds (a restart is not always sufficient):

[root@centos8-stream-katello-nightly ~]# hammer ping
database:         
    Status:          ok
    Server Response: Duration: 0ms
candlepin:        
    Status:          ok
    Server Response: Duration: 28ms
candlepin_auth:   
    Status:          ok
    Server Response: Duration: 35ms
candlepin_events: 
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
katello_events:   
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
pulp3:            
    Status:          ok
    Server Response: Duration: 76ms
pulp3_content:    
    Status:          ok
    Server Response: Duration: 147ms
foreman_tasks:    
    Status:          ok
    Server Response: Duration: 3ms

[root@centos8-stream-katello-nightly ~]# systemctl stop postgresql; sleep 60; systemctl start postgresql; sleep 10; hammer ping
database:         
    Status:          ok
    Server Response: Duration: 0ms
candlepin:        
    Status:          ok
    Server Response: Duration: 28ms
candlepin_auth:   
    Status:          ok
    Server Response: Duration: 29ms
candlepin_events: 
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
katello_events:   
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
pulp3:            
    Status:          ok
    Server Response: Duration: 37ms
pulp3_content:    
    Status:          ok
    Server Response: Duration: 36ms
foreman_tasks:    
    Status:          FAIL
    Server Response: Message: some executors are not responding, check /foreman_tasks/dynflow/status

The Katello part is only really needed as Katello exports the status of dynflow/tasks in the Ping API, which Foreman itself does not.

#3 Updated by Evgeni Golov 10 months ago

  • Subject changed from Restore of Katello 4.3 leaves the system in not fully functional state to Dynflow doesn't properly come back if the DB is unavailable for a brief period of time
  • Project changed from Foreman Maintain to Foreman
  • Found in Releases 2.5.2, 3.0.1, 3.2.0 added

#4 Updated by Evgeni Golov 10 months ago

If you want to repro this without Katello:

#5 Updated by Evgeni Golov 10 months ago

for the curious, updating sequel to 5.53.0 didn't magically fix this :/

#7 Updated by Evgeni Golov 10 months ago

more fun:
  • systemctl stop postgresql; sleep 30; systemctl start postgresql; sleep 10; hammer ping; sleep 10; hammer ping - first ping thinks it's OK, second FAILED
  • systemctl stop postgresql; sleep 20; systemctl start postgresql; sleep 10; hammer ping; sleep 10; hammer ping - those two OK, but later flips to FAILED
  • systemctl stop postgresql; sleep 10; systemctl start postgresql; sleep 10; hammer ping; sleep 10; hammer ping - all OK, and later ones are also OK

something is trying hard to survive, but fails at some point

#8 Updated by The Foreman Bot 10 months ago

  • Assignee set to Evgeni Golov
  • Status changed from New to Ready For Testing
  • Pull request https://github.com/theforeman/puppet-foreman/pull/1028 added

#9 Updated by Evgeni Golov 9 months ago

  • Target version set to 3.1.2

#10 Updated by Evgeni Golov 9 months ago

  • Fixed in Releases 3.1.2, 3.2.0 added

#11 Updated by Amit Upadhye 9 months ago

  • Target version changed from 3.1.2 to 3.1.3

Updated the Target version for this issue to 3.1.3,

<aupadhye> Zhenech, I dont think anything is pending to close https://projects.theforeman.org/issues/34394 ?
<Zhenech> it depends how you look at it
<Zhenech> if you say the redmin is for the workaround, then yes
<Zhenech> if you say it's a bug in dynflow (it is) then no
<aupadhye> then.. i keep it open

#12 Updated by Amit Upadhye 9 months ago

  • Copied to Bug #34485: Dynflow doesn't properly come back if the DB is unavailable for a brief period of time added

#13 Updated by Amit Upadhye 9 months ago

  • Category set to Foreman modules
  • Project changed from Foreman to Installer

#14 Updated by Amit Upadhye 9 months ago

  • Status changed from Ready For Testing to Closed

I have created a new issue https://projects.theforeman.org/issues/34485 to fix actual issue in dynflow. Accordingly closing this redmine ticket.

#15 Updated by Ewoud Kohl van Wijngaarden 9 months ago

  • Triaged changed from No to Yes
  • Target version changed from 3.1.3 to 3.1.2

Also available in: Atom PDF