Bug #35142

'foreman-maintain content migration-stats' command stucks and consume all memory

Added by Hao Yu about 1 month ago. Updated about 1 month ago.

Ready For Testing
Bugzilla link:
Fixed in Releases:
Found in Releases:
Red Hat JIRA:


Clone from bugzilla

Description of problem:
"foreman-maintain content migration-stats" command will run for extremely long time (several hours to day) and consume all the system memory when unmigratable contents are large, such 10K+.

  1. foreman-maintain content migration-stats
    Running Retrieve Pulp 2 to Pulp 3 migration statistics ================================================================================
    Retrieve Pulp 2 to Pulp 3 migration statistics: <=========== Stuck in here for long time
  1. Memory consumption is increasing quick.
    foreman 14301 16.0 1.9 816904 384304 ? Ssl 17:23 1:32 /opt/rh/rh-ruby25/root/usr/bin/ruby /opt/rh/rh-ruby25/root/usr/bin/rake katello:pulp3_migration_stats

foreman 14301 15.1 4.5 1337164 901500 ? Rsl 17:23 1:36 /opt/rh/rh-ruby25/root/usr/bin/ruby /opt/rh/rh-ruby25/root/usr/bin/rake katello:pulp3_migration_stats

foreman 14301 11.7 14.3 3311908 2864024 ? Ssl 17:23 1:50 /opt/rh/rh-ruby25/root/usr/bin/ruby /opt/rh/rh-ruby25/root/usr/bin/rake katello:pulp3_migration_stats

foreman 14301 12.4 33.5 7130700 6684896 ? Rsl 17:23 2:18 /opt/rh/rh-ruby25/root/usr/bin/ruby /opt/rh/rh-ruby25/root/usr/bin/rake katello:pulp3_migration_stats

Steps to Reproduce:
1.Prepare a Satellite 6.9.9 with about 20k or more of rpms and many content views and many content view versions.

2. To simulate unmigratable rpms we can run the following command to flag all rpms as missing from migration.

foreman-rake console
Katello::Rpm.update_all(migrated_pulp3_href: nil, missing_from_migration: true)

3. Run "foreman-maintain content migration-stats" command

Actual results:
Stuck and memory consumption is increasing overtime until OOM is triggered.

Expected results:
Run successfully and consume reasonable amount of system memory.

In Addition to the memory issue, the output files also printed many duplicate rows which is the reason that the script can take several hours to days to run.

In my case, it wrote 500 duplicated exactly the same rows:
  1. grep "opa-fm-,1,Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server,Default Organization View,1.0" Rpm | wc l


#1 Updated by The Foreman Bot about 1 month ago

  • Assignee set to Hao Yu
  • Status changed from New to Ready For Testing
  • Pull request added

#2 Updated by Lucy Fu about 1 month ago

  • Triaged changed from No to Yes
  • Target version set to 3.18-no-release-planned

Also available in: Atom PDF