Plugins » Katello

Hi,
system rhel7
Installed clean install of 3.1 katello some time ago:
[root@lv-katello ~]# rpm -qa | grep katello
katello-debug-3.1.0-5.el7.noarch
lv-katello.ne.1dc.com-foreman-proxy-1.0-2.noarch
lv-katello.ne.1dc.com-puppet-client-1.0-1.noarch
lv-katello.ne.1dc.com-apache-1.0-1.noarch
lv-katello.ne.1dc.com-qpid-broker-1.0-1.noarch
tfm-rubygem-katello-3.1.0.1-3.el7.noarch
katello-service-3.1.0-5.el7.noarch
katello-server-ca-1.0-1.noarch
lv-katello.ne.1dc.com-qpid-router-client-1.0-1.noarch
lv-katello.ne.1dc.com-foreman-client-1.0-1.noarch
katello-repos-3.1.0-5.el7.noarch
katello-selinux-3.0.1-1.el7.noarch
katello-certs-tools-2.4.0-1.el7.noarch
katello-installer-base-3.1.0-2.el7.noarch
katello-3.1.0-5.el7.noarch
tfm-rubygem-hammer_cli_katello-0.0.26-1.201606161944gita9fa180.el7.noarch
lv-katello.ne.1dc.com-qpid-router-server-1.0-1.noarch
lv-katello.ne.1dc.com-foreman-proxy-client-1.0-1.noarch
python-urllib3-1.10.2-2.katello.el7.noarch
katello-common-3.1.0-5.el7.noarch
foreman-installer-katello-3.1.0-2.el7.noarch
lv-katello.ne.1dc.com-qpid-client-cert-1.0-1.noarch
katello-default-ca-1.0-1.noarch
katello-client-bootstrap-1.1.0-1.el7.noarch
pulp-katello-1.0.2-1.el7.noarch

But there were problems with pulp - it was returning error 500 and logs showed
Nov 29 14:52:32 lv-katello.ne.1dc.com pulp²¹⁶⁸: pulp.server.webservices.application:ERROR: (2168-64448) The database has not been migrated to the current version. Run pulp-manage-db and restart the application.
Nov 29 14:52:32 lv-katello.ne.1dc.com pulp²¹⁶⁸: pulp.server.webservices.application:ERROR: (2168-64448) Traceback (most recent call last):
Nov 29 14:52:32 lv-katello.ne.1dc.com pulp²¹⁶⁸: pulp.server.webservices.application:ERROR: (2168-64448) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 111, in wsgi_application
Nov 29 14:52:32 lv-katello.ne.1dc.com pulp²¹⁶⁸: pulp.server.webservices.application:ERROR: (2168-64448) _initialize_web_services()
Nov 29 14:52:32 lv-katello.ne.1dc.com pulp²¹⁶⁸: pulp.server.webservices.application:ERROR: (2168-64448) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 74, in _initialize_web_services
Nov 29 14:52:32 lv-katello.ne.1dc.com pulp²¹⁶⁸: pulp.server.webservices.application:ERROR: (2168-64448) migration_models.check_package_versions()
Nov 29 14:52:32 lv-katello.ne.1dc.com pulp²¹⁶⁸: pulp.server.webservices.application:ERROR: (2168-64448) File "/usr/lib/python2.7/site-packages/pulp/server/db/migrate/models.py", line 310, in check_package_versions
Nov 29 14:52:32 lv-katello.ne.1dc.com pulp²¹⁶⁸: pulp.server.webservices.application:ERROR: (2168-64448) raise Exception(error_message)
Nov 29 14:52:32 lv-katello.ne.1dc.com pulp²¹⁶⁸: pulp.server.webservices.application:ERROR: (2168-64448) InitializationException: The database has not been migrated to the current version. Run pulp-manage-db and restart the application.
Nov 29 14:52:32 lv-katello.ne.1dc.com pulp²¹⁶⁸: pulp.server.webservices.application:CRITICAL: *****************************************************

Tried to run upgrade db but no luck
[root@lv-katello log]# sudo -u apache pulp-manage-db
Attempting to connect to localhost:27017
Attempting to connect to localhost:27017
Write concern for Mongo connection: {}
Loading content types.
Loading type descriptors []
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Updating the database with types []
Found the following type definitions that were not present in the update collection [puppet_module, srpm, erratum, docker_blob, drpm, distribution, package_group, package_category, iso, package_environment, docker_tag, package_langpacks, rpm, yum_repo_metadata_file, docker_image, docker_manifest]
Updating the database with types [puppet_module, docker_tag, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, drpm, distribution, rpm, srpm, docker_image]
Found the following type definitions that were not present in the update collection [package_langpacks]
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
The database for migration package pulp.server.db.migrations is at version 24, which is larger than the latest version available, 23.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 193, in main
return _auto_manage_db(options)
File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 256, in _auto_manage_db
migrate_database(options)
File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 73, in migrate_database
raise DataError(msg)
DataError: The database for migration package pulp.server.db.migrations is at version 24, which is larger than the latest version available, 23.

So I left it alone for a while, now I noticed that 3.2 is out so tried to upgrade, downloaded packages, upgrade check was successful, but running actual upgrade script returned error again with pulp but bit different:
[root@lv-katello ~]# foreman-installer --scenario katello --upgrade
Upgrading...
Upgrade Step: stop_services...
Redirecting to /bin/systemctl stop foreman-tasks.service

Redirecting to /bin/systemctl stop httpd.service

Redirecting to /bin/systemctl stop foreman-proxy.service

Redirecting to /bin/systemctl stop tomcat.service

Redirecting to /bin/systemctl stop squid.service

Redirecting to /bin/systemctl stop qpidd.service

Success!

Upgrade Step: start_databases...
Redirecting to /bin/systemctl start mongod.service

Redirecting to /bin/systemctl start postgresql.service

Success!

Upgrade Step: update_http_conf...

Upgrade Step: migrate_pulp...

938

Attempting to connect to localhost:27017
Attempting to connect to localhost:27017
Write concern for Mongo connection: {}
Loading content types.
Loading type descriptors []
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: package_langpacks = pulp_rpm.plugins.db.models:PackageLangpacks
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Updating the database with types []
Found the following type definitions that were not present in the update collection [puppet_module, srpm, erratum, docker_blob, drpm, distribution, package_group, package_category, iso, package_environment, docker_tag, package_langpacks, rpm, yum_repo_metadata_file, docker_image, docker_manifest]
Updating the database with types [puppet_module, drpm, package_langpacks, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, docker_tag, distribution, rpm, srpm, docker_image]
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
Migration package pulp.server.db.migrations is up to date at version 24
Migration package pulp_docker.plugins.migrations is up to date at version 2
Migration package pulp_puppet.plugins.migrations is up to date at version 5
The database for migration package pulp_rpm.plugins.migrations is at version 35, which is larger than the latest version available, 33.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 193, in main
return _auto_manage_db(options)
File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 256, in _auto_manage_db
migrate_database(options)
File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 73, in migrate_database
raise DataError(msg)
DataError: The database for migration package pulp_rpm.plugins.migrations is at version 35, which is larger than the latest version available, 33.

Upgrade step migrate_pulp failed. Check logs for more information.

Any suggestions would be much appreciated. Tnx!