Project

General

Profile

Actions

Bug #23061

closed

remote execution tasks stuck in status planned

Added by Christian Meißner over 5 years ago. Updated over 5 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Difficulty:
Triaged:
Fixed in Releases:
Found in Releases:

Description

After upgrading foreman to 1.16.0 we had some problems with duplicated database objects in dynflow schema. So we deleted all schema objects. Now REX doesn't work anymore. All planned jobs stuck in status "planned"

Any hints how we can get REX to work?


Files

dynflow_error.log dynflow_error.log 11.6 KB output from passenger log Christian Meißner, 03/29/2018 11:37 AM
dynflow_status.png View dynflow_status.png 82 KB Christian Meißner, 04/18/2018 07:53 AM
dynflow_status_after_restart.png View dynflow_status_after_restart.png 53.8 KB Christian Meißner, 04/18/2018 10:44 AM

Related issues 2 (0 open2 closed)

Related to foreman-tasks - Bug #12993: PostgreSQL DuplicateColumn ErrorClosed01/05/2016Actions
Is duplicate of Packaging - Bug #20050: ruby-sequel-pg DEB pollutes Foreman ruby gem environmentClosedIvan Necas06/19/2017Actions
Actions #1

Updated by Christian Meißner over 5 years ago

Attachted you find a passenger.log.

Actions #2

Updated by Anonymous over 5 years ago

  • Project changed from Foreman to Foreman Remote Execution
  • Priority changed from High to Normal
Actions #3

Updated by Adam Ruzicka over 5 years ago

What exactly did you do to remove the objects?

Actions #4

Updated by Christian Meißner over 5 years ago

Adam Ruzicka wrote:

What exactly did you do to remove the objects?

As mentioned in https://projects.theforeman.org/issues/12993#note-6

Actions #5

Updated by Adam Ruzicka over 5 years ago

  • Related to Bug #12993: PostgreSQL DuplicateColumn Error added
Actions #6

Updated by Christian Meißner over 5 years ago

https://projects.theforeman.org/issues/22745#change-102511 was the root cause in our environment.

Actions #7

Updated by Christian Meißner over 5 years ago

are there some news for that issue? Is there a way to workaround the problem. We have some hundreds of machines which we have to manage via REX.

Actions #9

Updated by Adam Ruzicka over 5 years ago

Hello, sorry for taking so long. So far I wasn't able to reproduce the issue you're having.

To get on the same page, you're on debian and were on foreman 1.15.x. You upgraded to 1.16 and got the duplicate keys error. So you solved that by dropping dynflow tables and then REX didn't work anymore. Is that correct?

A couple of questions to help us troubleshoot this. Have you restarted any services afterwards? If you go to FOREMAN_SERVER_URL/foreman_tasks/dynflow/status, is there any world marked as executor? If you check in the db, did the dynflow tables get created again?

Actions #10

Updated by Christian Meißner over 5 years ago

Adam Ruzicka wrote:

To get on the same page, you're on debian and were on foreman 1.15.x. You upgraded to 1.16 and got the duplicate keys error. So you solved that by dropping dynflow tables and then REX didn't work anymore. Is that correct?

Yes obviously. Yesterday We update Foreman to 1.16.1 and get also the "duplicate keys" error. Here we solved the issue by downgrading the pg gem.

A couple of questions to help us troubleshoot this. Have you restarted any services afterwards? If you go to FOREMAN_SERVER_URL/foreman_tasks/dynflow/status, is there any world marked as executor? If you check in the db, did the dynflow tables get created again?

Attached you find the output of dynflow status.

And yes the dynflow tables were created again.

public | dynflow_actions                         | table | foreman
public | dynflow_coordinator_records | table | foreman
public | dynflow_delayed_plans | table | foreman
public | dynflow_envelopes | table | foreman
public | dynflow_execution_plans | table | foreman
public | dynflow_schema_info | table | foreman
public | dynflow_steps | table | foreman
Actions #11

Updated by Adam Ruzicka over 5 years ago

All the worlds are client ones and are most likely part of the rails processes. However most of things is handled by a standalone service called foreman-tasks (or dynflowd on newer versions). Could you restart it? Another world record should show up FOREMAN_SERVER_URL/foreman_tasks/dynflow/status and it should be marked as executor. If it does, things should start moving again

Actions #12

Updated by Christian Meißner over 5 years ago

Adam Ruzicka wrote:

All the worlds are client ones and are most likely part of the rails processes. However most of things is handled by a standalone service called foreman-tasks (or dynflowd on newer versions). Could you restart it? Another world record should show up FOREMAN_SERVER_URL/foreman_tasks/dynflow/status and it should be marked as executor. If it does, things should start moving again

There is no service called foreman-tasks or dynflowd on my system. I can only restart apache2 to restart the rail application. But after that restart nothing changed. See attachment.

Actions #13

Updated by Adam Ruzicka over 5 years ago

I'm sorry, I was a bit off with the name, the service is apparently called ruby-foreman-tasks on deb-based systems. The service is provided by ruby-foreman-tasks package, which should get pulled in by remote execution.

Actions #14

Updated by Christian Meißner over 5 years ago

Adam Ruzicka wrote:

I'm sorry, I was a bit off with the name, the service is apparently called ruby-foreman-tasks on deb-based systems. The service is provided by ruby-foreman-tasks package, which should get pulled in by remote execution.

This does the trick. But my question is now. Why does a system reboot not fix the problem?

Actions #15

Updated by Ivan Necas over 5 years ago

  • Status changed from New to Duplicate

I guess the this issue is actually duplicate of http://projects.theforeman.org/issues/20050, see the details for workaround and and feel free to re-open if you think that's actually something else.

Actions #16

Updated by Ivan Necas over 5 years ago

  • Is duplicate of Bug #20050: ruby-sequel-pg DEB pollutes Foreman ruby gem environment added
Actions

Also available in: Atom PDF