Project

General

Profile

Bug #28712

mongo storage upgrade to wiredtiger fails and drops database if mongod is stopped

Added by Chris Roberts 10 months ago. Updated 9 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
foreman-installer script
Target version:
-
Difficulty:
easy
Triaged:
No
Bugzilla link:
Fixed in Releases:
Found in Releases:

Description

Cloned from BZ:

Description of problem:

If "satellite-installer --upgrade-mongo-storage-engine" is launched while mongod is stopped, the installer fails to detect the export failed and ends up dropping the source database.

Version-Release number of selected component (if applicable):

foreman-installer-1.22.0.16-1.el7sat.noarch
foreman-installer-katello-1.22.0.16-1.el7sat.noarch
satellite-installer-6.6.0.21-1.el7sat.noarch

How reproducible:

Always

Steps to Reproduce:
1. systemctl stop mongod
2. satellite-installer --upgrade-mongo-storage-engine

  1. systemctl stop mongod
  2. time satellite-installer --upgrade-mongo-storage-engine
    Starting disk space check for upgrade
    Package versions are locked. Continuing with unlock.
    Running Stop Services ================================================================================
    Check if command is run as root user: [OK]
    --------------------------------------------------------------------------------
    Stop applicable services: Stopping the following service(s):

qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, goferd, httpd
\ All services stopped [OK]
--------------------------------------------------------------------------------

foreman-maintain service stop --exclude "rh-mongodb34-mongod","postgresql","tomcat","dynflowd","foreman-proxy","puppetserver" finished successfully!
2019-11-06T10:14:34.070+0100 Failed: error connecting to db server: no reachable servers
mongodump --host localhost --out /var/lib/pulp/tmp/mongodb_engine_upgrade failed! Check the output for error!
Running Stop Services ================================================================================
Check if command is run as root user: [OK]
--------------------------------------------------------------------------------
Stop applicable services: Stopping the following service(s):

rh-mongodb34-mongod | All services stopped [OK]
--------------------------------------------------------------------------------

foreman-maintain service stop --only rh-mongodb34-mongod finished successfully!
rm -rf /var/lib/mongodb/* finished successfully!
sed -i.bak -e 's/mmapv1/wiredTiger/g' /etc/opt/rh/rh-mongodb34/mongod.conf finished successfully!
mv /etc/opt/rh/rh-mongodb34/mongod.conf.bak /var/lib/pulp/tmp/mongodb_engine_upgrade finished successfully!
Running Start Services ================================================================================
Check if command is run as root user: [OK]
--------------------------------------------------------------------------------
Start applicable services: Starting the following service(s):

rh-mongodb34-mongod
/ All services started [OK]
--------------------------------------------------------------------------------

foreman-maintain service start --only rh-mongodb34-mongod finished successfully!
2019-11-06T10:14:41.885+0100 the --db and --collection args should only be used when restoring from a BSON file. Other uses are deprecated and will not exist in the future; use --nsInclude instead
foreman-maintain service start --only rh-mongodb34-mongod finished successfully!
2019-11-06T10:14:41.885+0100 the --db and --collection args should only be used when restoring from a BSON file. Other uses are deprecated and will not exist in the future; use --nsInclude instead
2019-11-06T10:14:41.885+0100 Failed: mongorestore target '/var/lib/pulp/tmp/mongodb_engine_upgrade/pulp_database' invalid: stat /var/lib/pulp/tmp/mongodb_engine_upgrade/pulp_database: no such file or director
y
mongorestore --host localhost --db=pulp_database --drop --dir=/var/lib/pulp/tmp/mongodb_engine_upgrade/pulp_database failed! Check the output for error!
Running Stop Services ================================================================================
Check if command is run as root user: [OK]
--------------------------------------------------------------------------------
Stop applicable services: Stopping the following service(s):

rh-mongodb34-mongod
/ All services stopped [OK]
--------------------------------------------------------------------------------

foreman-maintain service stop --only rh-mongodb34-mongod finished successfully!
mv -f /var/lib/pulp/tmp/mongodb_engine_upgrade/mongod.conf.bak /etc/opt/rh/rh-mongodb34/mongod.conf finished successfully!
rm -rf /var/lib/mongodb/* finished successfully!
2019-11-06T10:14:47.656+0100 Failed: error connecting to db server: no reachable servers
mongorestore --host localhost --db=pulp_database --drop --dir=/var/lib/pulp/tmp/mongodb_engine_upgrade/pulp_database failed! Check the output for error!
Running Start Services ================================================================================
Check if command is run as root user: [OK]
--------------------------------------------------------------------------------
Start applicable services: Starting the following service(s):

rh-mongodb34-mongod | All services started [OK]
--------------------------------------------------------------------------------

foreman-maintain service start --only rh-mongodb34-mongod finished successfully!

real 0m32.729s
user 0m16.383s
sys 0m7.381s

Expected results:

The migration should check that the mongodump process is successful prior to engaging into a potentially destructive migration.

Associated revisions

Revision 785b3a94 (diff)
Added by Chris Roberts 9 months ago

Fixes #28712 - Verify MongoDB is on before upgrade

History

#1 Updated by The Foreman Bot 9 months ago

  • Status changed from Assigned to Ready For Testing
  • Pull request https://github.com/theforeman/foreman-installer/pull/450 added

#2 Updated by The Foreman Bot 9 months ago

  • Fixed in Releases 2.0.0 added

#3 Updated by Chris Roberts 9 months ago

  • Status changed from Ready For Testing to Closed

Also available in: Atom PDF