Bug #23061
closed
remote execution tasks stuck in status planned
Description
After upgrading foreman to 1.16.0 we had some problems with duplicated database objects in dynflow schema. So we deleted all schema objects. Now REX doesn't work anymore. All planned jobs stuck in status "planned"
Any hints how we can get REX to work?
Files
| dynflow_error.log | dynflow_error.log | 11.6 KB | output from passenger log | ||
| dynflow_status.png | View dynflow_status.png | 82 KB | |||
| dynflow_status_after_restart.png | View dynflow_status_after_restart.png | 53.8 KB |
Updated by Christian Meißner almost 7 years ago
- File dynflow_error.log dynflow_error.log added
Attachted you find a passenger.log.
Updated by Anonymous almost 7 years ago
- Project changed from Foreman to Foreman Remote Execution
- Priority changed from High to Normal
Updated by Adam Ruzicka almost 7 years ago
What exactly did you do to remove the objects?
Updated by Christian Meißner almost 7 years ago
Adam Ruzicka wrote:
What exactly did you do to remove the objects?
As mentioned in https://projects.theforeman.org/issues/12993#note-6
Updated by Adam Ruzicka almost 7 years ago
- Related to Bug #12993: PostgreSQL DuplicateColumn Error added
Updated by Christian Meißner almost 7 years ago
https://projects.theforeman.org/issues/22745#change-102511 was the root cause in our environment.
Updated by Christian Meißner over 6 years ago
are there some news for that issue? Is there a way to workaround the problem. We have some hundreds of machines which we have to manage via REX.
Updated by Christian Meißner over 6 years ago
related to:
Updated by Adam Ruzicka over 6 years ago
Hello, sorry for taking so long. So far I wasn't able to reproduce the issue you're having.
To get on the same page, you're on debian and were on foreman 1.15.x. You upgraded to 1.16 and got the duplicate keys error. So you solved that by dropping dynflow tables and then REX didn't work anymore. Is that correct?
A couple of questions to help us troubleshoot this. Have you restarted any services afterwards? If you go to FOREMAN_SERVER_URL/foreman_tasks/dynflow/status, is there any world marked as executor? If you check in the db, did the dynflow tables get created again?
Updated by Christian Meißner over 6 years ago
- File dynflow_status.png dynflow_status.png added
Adam Ruzicka wrote:
To get on the same page, you're on debian and were on foreman 1.15.x. You upgraded to 1.16 and got the duplicate keys error. So you solved that by dropping dynflow tables and then REX didn't work anymore. Is that correct?
Yes obviously. Yesterday We update Foreman to 1.16.1 and get also the "duplicate keys" error. Here we solved the issue by downgrading the pg gem.
A couple of questions to help us troubleshoot this. Have you restarted any services afterwards? If you go to FOREMAN_SERVER_URL/foreman_tasks/dynflow/status, is there any world marked as executor? If you check in the db, did the dynflow tables get created again?
Attached you find the output of dynflow status.
And yes the dynflow tables were created again.
public | dynflow_actions | table | foreman
public | dynflow_coordinator_records | table | foreman
public | dynflow_delayed_plans | table | foreman
public | dynflow_envelopes | table | foreman
public | dynflow_execution_plans | table | foreman
public | dynflow_schema_info | table | foreman
public | dynflow_steps | table | foreman
Updated by Adam Ruzicka over 6 years ago
All the worlds are client ones and are most likely part of the rails processes. However most of things is handled by a standalone service called foreman-tasks (or dynflowd on newer versions). Could you restart it? Another world record should show up FOREMAN_SERVER_URL/foreman_tasks/dynflow/status and it should be marked as executor. If it does, things should start moving again
Updated by Christian Meißner over 6 years ago
Adam Ruzicka wrote:
All the worlds are client ones and are most likely part of the rails processes. However most of things is handled by a standalone service called foreman-tasks (or dynflowd on newer versions). Could you restart it? Another world record should show up FOREMAN_SERVER_URL/foreman_tasks/dynflow/status and it should be marked as executor. If it does, things should start moving again
There is no service called foreman-tasks or dynflowd on my system. I can only restart apache2 to restart the rail application. But after that restart nothing changed. See attachment.
Updated by Adam Ruzicka over 6 years ago
I'm sorry, I was a bit off with the name, the service is apparently called ruby-foreman-tasks on deb-based systems. The service is provided by ruby-foreman-tasks package, which should get pulled in by remote execution.
Updated by Christian Meißner over 6 years ago
Adam Ruzicka wrote:
I'm sorry, I was a bit off with the name, the service is apparently called ruby-foreman-tasks on deb-based systems. The service is provided by ruby-foreman-tasks package, which should get pulled in by remote execution.
This does the trick. But my question is now. Why does a system reboot not fix the problem?
Updated by Ivan Necas over 6 years ago
- Status changed from New to Duplicate
I guess the this issue is actually duplicate of http://projects.theforeman.org/issues/20050, see the details for workaround and and feel free to re-open if you think that's actually something else.
Updated by Ivan Necas over 6 years ago
- Is duplicate of Bug #20050: ruby-sequel-pg DEB pollutes Foreman ruby gem environment added