Project

General

Profile

Actions
Copy link

Bug #34394

closed

Dynflow doesn't properly come back if the DB is unavailable for a brief period of time

Added by Evgeni Golov almost 3 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Evgeni Golov
Category:
Foreman modules
Target version:
Foreman - 3.1.2
Difficulty:
Triaged:
Yes
Bugzilla link:
Pull request:
https://github.com/theforeman/puppet-foreman/pull/1028
Fixed in Releases:
Foreman - 3.1.2, Foreman - 3.2.0
Found in Releases:
Foreman - 2.5.2, Foreman - 3.0.1, Foreman - 3.2.0
Red Hat JIRA:

Description

Ohai,

when I try to restore a Katello 4.3 (or nightly) on EL8 (with rubygem-foreman_maintain-1.0.2-1.el8.noarch), the restore finishes fine, but afterwards not all services are happy:

# hammer ping
database:         
    Status:          ok
    Server Response: Duration: 0ms
katello_agent:    
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
candlepin:        
    Status:          ok
    Server Response: Duration: 58ms
candlepin_auth:   
    Status:          ok
    Server Response: Duration: 50ms
candlepin_events: 
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
katello_events:   
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 1ms
pulp3:            
    Status:          ok
    Server Response: Duration: 117ms
pulp3_content:    
    Status:          ok
    Server Response: Duration: 128ms
foreman_tasks:    
    Status:          FAIL
    Server Response: Message: some executors are not responding, check /foreman_tasks/dynflow/status

After restarting dynflow via systemctl restart dynflow-sidekiq@\* everything seems to work fine again.

I am not sure this is a maintain bug (or installer, or dynflow, or packaging), but filing it here for investigation.

Related issues 1 (1 open — 0 closed)

Copied to foreman-tasks - Bug #34485: Dynflow doesn't properly come back if the DB is unavailable for a brief period of timeNewActions
Actions
Copy link
 #1

Updated by Evgeni Golov almost 3 years ago

Looking at the logs, it seems dynflow was disconnected from the database and didn't manage to reconnect:

Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: FATAL:  terminating connection due to administrator command
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: #<Thread:0x0000563d5bc0f370 /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true):
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec': PG::UnableToSend: no connection to the server (Sequel::DatabaseDisconnectError)
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `ensure in block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: FATAL:  terminating connection due to administrator command
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: #<Thread:0x000055ff9be65370 /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true):
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec': PG::UnableToSend: no connection to the server (Sequel::DatabaseDisconnectError)
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `ensure in block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: FATAL:  terminating connection due to administrator command
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: #<Thread:0x00005577dde812a0 /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 run> terminated with exception (report_on_exception is true):
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec': PG::UnableToSend: no connection to the server (Sequel::DatabaseDisconnectError)
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `ensure in block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec': no connection to the server (PG::UnableToSend)
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `ensure in block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:476:in `wait_for_notify': PQconsumeInput() server closed the connection unexpectedly (PG::ConnectionBad)
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         This probably means the server terminated abnormally
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         before or while processing the request.
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:476:in `block (3 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec': no connection to the server (PG::UnableToSend)
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `ensure in block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:476:in `wait_for_notify': PQconsumeInput() server closed the connection unexpectedly (PG::ConnectionBad)
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         This probably means the server terminated abnormally
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         before or while processing the request.
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:476:in `block (3 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec': no connection to the server (PG::UnableToSend)
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `ensure in block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:486:in `block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:476:in `wait_for_notify': PQconsumeInput() server closed the connection unexpectedly (PG::ConnectionBad)
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         This probably means the server terminated abnormally
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         before or while processing the request.
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:476:in `block (3 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:473:in `catch'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:473:in `block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:473:in `catch'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:473:in `block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:473:in `catch'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:473:in `block (2 levels) in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:456:in `block in listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:455:in `listen'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/connectors/database.rb:25:in `block in start'
Feb 04 13:21:08 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         from /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Feb 04 13:21:14 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: E, [2022-02-04T13:21:14.195825 #75239] ERROR -- /connector-database-core: Receiving envelopes failed on PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
Feb 04 13:21:14 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         This probably means the server terminated abnormally
Feb 04 13:21:14 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         before or while processing the request.
Feb 04 13:21:15 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: E, [2022-02-04T13:21:15.758015 #75238] ERROR -- /connector-database-core: Receiving envelopes failed on PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
Feb 04 13:21:15 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         This probably means the server terminated abnormally
Feb 04 13:21:15 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         before or while processing the request.
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: E, [2022-02-04T13:21:19.208497 #74683] ERROR -- /connector-database-core: Receiving envelopes failed on PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         This probably means the server terminated abnormally
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         before or while processing the request.
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: E, [2022-02-04T13:21:19.249622 #74683] ERROR -- /parallel-executor-core: PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         This probably means the server terminated abnormally
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:         before or while processing the request.
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]:  (Sequel::DatabaseDisconnectError)
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `async_exec'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `block in execute_query'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/logging.rb:38:in `log_connection_yield'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:156:in `execute_query'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `block in execute'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:120:in `check_disconnect_errors'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:144:in `execute'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:500:in `_execute'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:318:in `block (2 levels) in execute'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:522:in `check_database_errors'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:318:in `block in execute'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/database/connecting.rb:269:in `synchronize'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:318:in `execute'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/dataset/actions.rb:1093:in `execute'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/adapters/postgres.rb:614:in `fetch_rows'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/dataset/actions.rb:152:in `each'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/dataset/actions.rb:51:in `block in all'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/dataset/actions.rb:1010:in `_all'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/sequel-5.42.0/lib/sequel/dataset/actions.rb:51:in `all'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/persistence_adapters/sequel.rb:260:in `find_coordinator_records'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/coordinator_adapters/sequel.rb:30:in `find_records'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/coordinator.rb:366:in `find_records'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/abstract/core.rb:89:in `heartbeat'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/core.rb:36:in `heartbeat'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: [ concurrent-ruby ]
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/abstract/core.rb:122:in `on_message'
Feb 04 13:21:19 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@orchestrator[74683]: [ concurrent-ruby ]
Feb 04 13:21:44 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]: E, [2022-02-04T13:21:44.197521 #75239] ERROR -- /connector-database-core: Receiving envelopes failed on PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
Feb 04 13:21:44 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         This probably means the server terminated abnormally
Feb 04 13:21:44 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-1[75239]:         before or while processing the request.
Feb 04 13:21:45 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]: E, [2022-02-04T13:21:45.760312 #75238] ERROR -- /connector-database-core: Receiving envelopes failed on PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
Feb 04 13:21:45 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         This probably means the server terminated abnormally
Feb 04 13:21:45 pipe-katello-server-nightly-centos7.n13.example.com dynflow-sidekiq@worker-hosts-queue-1[75238]:         before or while processing the request.
Actions
Copy link
 #2

Updated by Evgeni Golov almost 3 years ago

Okay, I can reproduce this without Foreman Maintain and without the Installer on a fresh Katello nightly (and also on 4.2). Just turn off the DB for 60 seconds (a restart is not always sufficient):

[root@centos8-stream-katello-nightly ~]# hammer ping
database:         
    Status:          ok
    Server Response: Duration: 0ms
candlepin:        
    Status:          ok
    Server Response: Duration: 28ms
candlepin_auth:   
    Status:          ok
    Server Response: Duration: 35ms
candlepin_events: 
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
katello_events:   
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
pulp3:            
    Status:          ok
    Server Response: Duration: 76ms
pulp3_content:    
    Status:          ok
    Server Response: Duration: 147ms
foreman_tasks:    
    Status:          ok
    Server Response: Duration: 3ms

[root@centos8-stream-katello-nightly ~]# systemctl stop postgresql; sleep 60; systemctl start postgresql; sleep 10; hammer ping
database:         
    Status:          ok
    Server Response: Duration: 0ms
candlepin:        
    Status:          ok
    Server Response: Duration: 28ms
candlepin_auth:   
    Status:          ok
    Server Response: Duration: 29ms
candlepin_events: 
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
katello_events:   
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
pulp3:            
    Status:          ok
    Server Response: Duration: 37ms
pulp3_content:    
    Status:          ok
    Server Response: Duration: 36ms
foreman_tasks:    
    Status:          FAIL
    Server Response: Message: some executors are not responding, check /foreman_tasks/dynflow/status

The Katello part is only really needed as Katello exports the status of dynflow/tasks in the Ping API, which Foreman itself does not.

Actions
Copy link
 #3

Updated by Evgeni Golov almost 3 years ago

  • Project changed from Foreman Maintain to Foreman
  • Subject changed from Restore of Katello 4.3 leaves the system in not fully functional state to Dynflow doesn't properly come back if the DB is unavailable for a brief period of time
  • Found in Releases 2.5.2, 3.0.1, 3.2.0 added
Actions
Copy link
 #4

Updated by Evgeni Golov almost 3 years ago

If you want to repro this without Katello:

Actions
Copy link
 #5

Updated by Evgeni Golov almost 3 years ago

for the curious, updating sequel to 5.53.0 didn't magically fix this :/

Actions
Copy link
 #7

Updated by Evgeni Golov almost 3 years ago

more fun:
  • systemctl stop postgresql; sleep 30; systemctl start postgresql; sleep 10; hammer ping; sleep 10; hammer ping - first ping thinks it's OK, second FAILED
  • systemctl stop postgresql; sleep 20; systemctl start postgresql; sleep 10; hammer ping; sleep 10; hammer ping - those two OK, but later flips to FAILED
  • systemctl stop postgresql; sleep 10; systemctl start postgresql; sleep 10; hammer ping; sleep 10; hammer ping - all OK, and later ones are also OK

something is trying hard to survive, but fails at some point

Actions
Copy link
 #8

Updated by The Foreman Bot almost 3 years ago

  • Status changed from New to Ready For Testing
  • Assignee set to Evgeni Golov
  • Pull request https://github.com/theforeman/puppet-foreman/pull/1028 added
Actions
Copy link
 #9

Updated by Evgeni Golov almost 3 years ago

  • Target version set to 3.1.2
Actions
Copy link
 #10

Updated by Evgeni Golov almost 3 years ago

  • Fixed in Releases 3.1.2, 3.2.0 added
Actions
Copy link
 #11

Updated by Amit Upadhye almost 3 years ago

  • Target version changed from 3.1.2 to 3.1.3

Updated the Target version for this issue to 3.1.3,

<aupadhye> Zhenech, I dont think anything is pending to close https://projects.theforeman.org/issues/34394 ?
<Zhenech> it depends how you look at it
<Zhenech> if you say the redmin is for the workaround, then yes
<Zhenech> if you say it's a bug in dynflow (it is) then no
<aupadhye> then.. i keep it open

Actions
Copy link
 #12

Updated by Amit Upadhye almost 3 years ago

  • Copied to Bug #34485: Dynflow doesn't properly come back if the DB is unavailable for a brief period of time added
Actions
Copy link
 #13

Updated by Amit Upadhye almost 3 years ago

  • Project changed from Foreman to Installer
  • Category set to Foreman modules
Actions
Copy link
 #14

Updated by Amit Upadhye almost 3 years ago

  • Status changed from Ready For Testing to Closed

I have created a new issue https://projects.theforeman.org/issues/34485 to fix actual issue in dynflow. Accordingly closing this redmine ticket.

Actions
Copy link
 #15

Updated by Ewoud Kohl van Wijngaarden almost 3 years ago

  • Target version changed from 3.1.3 to 3.1.2
  • Triaged changed from No to Yes
Actions
Copy link

Also available in: Atom PDF