Bug #31113

Unresumable task due to timeout

Added by Markus Bucher over 1 year ago. Updated 7 months ago.

Target version:
Bugzilla link:
Pull request:
Fixed in Releases:
Red Hat JIRA:


After upgrading to foreman-2.1.3, katello-, we experience hanging tasks on some of our systems.

Those are mainly due to Candlepin blocking requests, because /var had more than 90% disk-space used. This can be solved by adding enough disk-space.
However, after adding enough disk space, the task was still hanging and could not be resumed.

In Foreman-Task the task was marked as stopped with result pending. Given that it was the creation of an Activation Key, the ActivationKey was there in Katello, but not in Candlepin and therefore was not removable.

The dynflow-console shows the step (here Katello::Resources::Candlepin::ActivationKey) as failed, with the following error and the later steps as (pending):

Label: Actions::Katello::ActivationKey::Create
Status: stopped
Result: error
Started at: 2020-10-17 18:45:49 UTC
Ended at: 2020-10-18 09:03:50 UTC

3: Actions::Candlepin::ActivationKey::Create (error) [ 55070.43s / 7200.74s ]
Queue: default
Started at: 2020-10-17 18:46:00 UTC
Ended at: 2020-10-18 10:03:50 UTC
Real time: 55070.43s
Execution time (excluding suspended state): 7200.74s



Katello::Resources::Candlepin::ActivationKey: Timed out reading data from server (POST /candlepin/owners/Atix/activation_keys)

- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:733:in
  `rescue in transmit'" 
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:642:in
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/resource.rb:67:in
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in
  `block (3 levels) in execute_run'" 
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-
  `block in run'" 

I guess the Foreman-Task question is: Is stopped:pending a valid Task-state
If so I guess this issue is for Katello to make sure the Task is resumable!?

Screenshot_20201020_160114.png View Screenshot_20201020_160114.png 47.6 KB Markus Bucher, 10/20/2020 03:26 PM
Screenshot_20201020_160105.png View Screenshot_20201020_160105.png 167 KB Markus Bucher, 10/20/2020 03:26 PM
Screenshot 20201020 160114 Screenshot 20201020 160105


#1 Updated by Adam Ruzicka over 1 year ago

Did it get to stopped-pending by itself?

#2 Updated by Richard Stempfl over 1 year ago

Adam Ruzicka wrote:

Did it get to stopped-pending by itself?

The task was running. since the hard disk had less than 90% free memory, it got stuck and did not change anymore.
Then more disk space was added and the services were restarted after which the task got its status described above.

#3 Updated by Adam Ruzicka 7 months ago

  • Project changed from foreman-tasks to Katello

Switching the component to Katello, tasks as an engine cannot really do anything about it if an action decides to stop on error.

#4 Updated by Jonathon Turel 7 months ago

The following line needs to be added to /etc/candlepin/broker.xml somewhere within the main configuration block:


After that, do systemctl restart tomcat and see if things work then. Hopefully you've upgraded from 3.16 by now - I think we made this option permanent in 4.0 and later

#5 Updated by Ryan Verdile 7 months ago

  • Triaged changed from No to Yes
  • Target version set to Katello Recycle Bin
  • Status changed from New to Rejected

Also available in: Atom PDF