Project

General

Profile

Bug #27826

Disk space check during mongo storage upgrade to wiredtiger failing and dropping database

Added by Chris Roberts 7 months ago. Updated 7 months ago.

Status:
Closed
Priority:
High
Assignee:
Category:
foreman-installer script
Target version:
Difficulty:
easy
Triaged:
Yes
Bugzilla link:
Fixed in Releases:
Found in Releases:

Description

Running:

satellite-installer --upgrade-mongo-storage-engine

When this happens:

2019-05-29T15:59:51.497-0400 Failed: error writing data for collection `pulp_database.units_rpm` to disk: error writing to file: write /var/tmp/mongodb_engine_upgrade/pulp_database/units_rpm.bson: no space left on device
mongodump --host localhost --out /var/tmp/mongodb_engine_upgrade failed! Check the output for error!

Then immediately followed by:

mv: error writing ‘/var/tmp/mongodb_backup/diagnostic.data/metrics.2019-05-28T14-59-39Z-00000’: No space left on device
mv: failed to extend ‘/var/tmp/mongodb_backup/diagnostic.data/metrics.2019-05-28T14-59-39Z-00000’: No space left on device
mv /var/lib/mongodb/* /var/tmp/mongodb_backup failed! Check the output for error!
sed -i.bak -e 's/mmapv1/wiredTiger/g' /etc/opt/rh/rh-mongodb34/mongod.conf finished successfully!
mv: cannot create regular file ‘/var/tmp/mongodb_backup/mongod.conf.bak’: No space left on device
mv /etc/opt/rh/rh-mongodb34/mongod.conf.bak /var/tmp/mongodb_backup failed! Check the output for error!

Then:

mongorestore --host localhost --db=pulp_database --drop --quiet --dir=/var/tmp/mongodb_engine_upgrade/pulp_database failed! Check the output for error!

Result is no database.

Expectations:

1. There should be a check and warning that space is required in /var/tmp/mongodb_backup to hold the database
2. A mongorestore with --drop should ONLY be attempted if the prior steps complete successfully

Where are you experiencing the behavior? What environment?

Satellite 6.5 upgrading to wiredtiger

Associated revisions

Revision 37767cf6 (diff)
Added by Chris Roberts 7 months ago

Fixes #27826 - Update Mongo upgrade recovery logic

History

#1 Updated by The Foreman Bot 7 months ago

  • Status changed from New to Ready For Testing
  • Pull request https://github.com/theforeman/foreman-installer/pull/388 added

#2 Updated by The Foreman Bot 7 months ago

  • Fixed in Releases 1.24.0 added

#3 Updated by Chris Roberts 7 months ago

  • Triaged changed from No to Yes
  • Target version set to 1.24.0
  • Status changed from Ready For Testing to Closed

Also available in: Atom PDF