Plugins » Katello

Does your apache logs show requests for those files? You might wanna 'tail -f -n 0 /var/log'/httpd/*' and try to boot.

Also what did you upgrade from?

Justin Sherrill wrote:

Does your apache logs show requests for those files? You might wanna 'tail -f -n 0 /var/log'/httpd/*' and try to boot.

Also what did you upgrade from?

Hi Justin,

well its working now. I can only guess it was a reboot that "fixed" it. Sorry! I upgraded from RC4

Hi this error has reappeared after upgrading to the latest RC - a new VM created via the Hosts > New Host pages using bootdisk-based imaging. Tailing the httpd logs gives the below:

We had the same issue on our installation after migration to 3.0 RC5 to 2.3. Our issue was that the kickstart repository had become corrupted. To fix we simply disabled/re-enabled the kickstart repository.

to 3.0RC5 from 2.3 (for some reason I can't edit my previous post)

Jonathan Dean wrote:

We had the same issue on our installation after migration to 3.0 RC5 to 2.3. Our issue was that the kickstart repository had become corrupted. To fix we simply disabled/re-enabled the kickstart repository.

Hi Jonathan,

not sure what you mean - I can't see anywhere under products, content views, installation media where I can disable/enable a repository. Help?

Thanks,
Dylan

Dylan Baars wrote:

Jonathan Dean wrote:

We had the same issue on our installation after migration to 3.0 RC5 to 2.3. Our issue was that the kickstart repository had become corrupted. To fix we simply disabled/re-enabled the kickstart repository.

Hi Jonathan,

not sure what you mean - I can't see anywhere under products, content views, installation media where I can disable/enable a repository. Help?

Thanks,
Dylan

Look under Content > Red Hat Repositories. Then go to kickstarts and find your kickstart repository(ies) - un-check and re-check it.

Right, so I'm using CentOS on my dev box, not RHEL. On checking things further, one of the content views seemed to be corrupt - could not publish new versions. I created a new content view for CentOS7 Server, moved my test boxes to using it and the same for hostgroup configuration. I also had to make sure in each hostgroup that there was no media selected on the operating system tab. After doing those two things, I could remove all content view versions for that content view - on trying to delete the content view completely it gives me the error

"Cannot delete record because of dependent content_facets"

/var/log/foreman/production.log output when I try and do so:

2016-06-16 15:54:01 [app] [I] Started GET "/errata/views/errata-counts.html" for 192.168.222.132 at 2016-06-16 15:54:01 +1200
2016-06-16 15:54:01 [app] [I] Started GET "/katello/api/v2/content_view_versions?content_view_id=4&page=1&search=" for 192.168.222.132 at 2016-06-16 15:54:01 +1200
2016-06-16 15:54:01 [app] [I] Processing by Katello::Api::V2::ContentViewVersionsController#index as JSON
2016-06-16 15:54:01 [app] [I] Parameters: {"content_view_id"=>"4", "page"=>"1", "search"=>"", "api_version"=>"v2"}
2016-06-16 15:54:01 [app] [I] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.rc8/app/views/katello/api/v2/content_view_versions/index.json.rabl within katello/api/v2/layouts/collection (3.2ms)
2016-06-16 15:54:01 [app] [I] Completed 200 OK in 29ms (Views: 5.5ms | ActiveRecord: 3.4ms)
2016-06-16 15:54:01 [app] [I] Processing by Katello::Api::V2::ContentViewsController#show as JSON
2016-06-16 15:54:01 [app] [I] Parameters: {"organization_id"=>"6", "api_version"=>"v2", "id"=>"4"}
2016-06-16 15:54:02 [app] [I] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.rc8/app/views/katello/api/v2/content_views/show.json.rabl within katello/api/v2/layouts/resource (111.7ms)
2016-06-16 15:54:02 [app] [I] Completed 200 OK in 133ms (Views: 107.3ms | ActiveRecord: 15.5ms)
2016-06-16 15:54:08 [app] [I] Started GET "/content-views/deletion/views/content-view-deletion.html" for 192.168.222.132 at 2016-06-16 15:54:08 +1200
2016-06-16 15:54:08 [app] [I] Started GET "/katello/api/v2/content_view_versions?content_view_id=4&page=1&search=" for 192.168.222.132 at 2016-06-16 15:54:08 +1200
2016-06-16 15:54:08 [app] [I] Processing by Katello::Api::V2::ContentViewVersionsController#index as JSON
2016-06-16 15:54:08 [app] [I] Parameters: {"content_view_id"=>"4", "page"=>"1", "search"=>"", "api_version"=>"v2"}
2016-06-16 15:54:08 [app] [I] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.rc8/app/views/katello/api/v2/content_view_versions/index.json.rabl within katello/api/v2/layouts/collection (13.8ms)
2016-06-16 15:54:08 [app] [I] Completed 200 OK in 63ms (Views: 25.2ms | ActiveRecord: 5.1ms)
2016-06-16 15:54:09 [app] [I] Started DELETE "/katello/api/v2/content_views/4?organization_id=6" for 192.168.222.132 at 2016-06-16 15:54:09 +1200
2016-06-16 15:54:09 [app] [I] Processing by Katello::Api::V2::ContentViewsController#destroy as JSON
2016-06-16 15:54:09 [app] [I] Parameters: {"organization_id"=>"6", "api_version"=>"v2", "id"=>"4"}
2016-06-16 15:54:10 [foreman-tasks/action] [E] Cannot delete record because of dependent content_facets (ActiveRecord::DeleteRestrictionError) | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/has_many_association.rb:13:in `handle_dependency' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/associations/builder/association.rb:135:in `block in add_before_destroy_callbacks' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:440:in `instance_exec' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:440:in `block in make_lambda' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:160:in `call' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:160:in `block in halting' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `call' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `block in halting' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `call' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `block in halting' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `call' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `block in halting' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `call' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `block in halting' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `call' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `block in halting' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `call' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `block in halting' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `call' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `block in halting' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `call' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `block in halting' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:86:in `call' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:86:in `run_callbacks' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/callbacks.rb:292:in `destroy' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:263:in `block in destroy' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:329:in `block in with_transaction_returning_status' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:199:in `transaction' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in `transaction' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:326:in `with_transaction_returning_status' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:263:in `destroy' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.rc8/app/models/katello/model.rb:7:in `destroy!' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.rc8/app/lib/actions/katello/content_view/destroy.rb:27:in `finalize' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:528:in `block (2 levels) in execute_finalize' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.rc8/app/lib/actions/middleware/remote_action.rb:20:in `block in finalize' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.rc8/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.rc8/app/models/katello/concerns/user_extensions.rb:20:in `cp_config' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.rc8/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.rc8/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.rc8/app/lib/actions/middleware/remote_action.rb:20:in `finalize' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/progress.rb:30:in `with_progress_calculation' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/progress.rb:22:in `finalize' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.rc8/app/lib/actions/middleware/keep_locale.rb:15:in `block in finalize' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.rc8/app/lib/actions/middleware/keep_locale.rb:22:in `with_locale' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.rc8/app/lib/actions/middleware/keep_locale.rb:15:in `finalize' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:38:in `finalize' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/world.rb:30:in `execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:527:in `block in execute_finalize' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in `block in with_error_handling' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in `catch' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in `with_error_handling' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:526:in `execute_finalize' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:260:in `execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/sequential_manager.rb:68:in `run_step' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/sequential_manager.rb:53:in `dispatch' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/sequential_manager.rb:60:in `block in run_in_sequence' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/sequential_manager.rb:60:in `each' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/sequential_manager.rb:60:in `all?' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/sequential_manager.rb:60:in `run_in_sequence' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/sequential_manager.rb:49:in `dispatch' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/sequential_manager.rb:27:in `block in finalize' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:46:in `finalize_phase' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:46:in `finalize_phase' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:46:in `finalize_phase' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/common/transaction.rb:16:in `block in rollback_on_error' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `block in transaction' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in `within_new_transaction' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `transaction' | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in `transaction' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/transaction_adapters/active_record.rb:5:in `transaction' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/common/transaction.rb:15:in `rollback_on_error' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/common/transaction.rb:9:in `finalize_phase' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/world.rb:30:in `execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/sequential_manager.rb:26:in `finalize' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/worker.rb:18:in `block in on_message' | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns' | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap' | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns' | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value' | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?' | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each' | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?' | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/worker.rb:12:in `on_message' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/actor.rb:26:in `on_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in `process_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:in `block in on_envelope' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in `block in schedule_execution' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:in `work' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `run_task' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in `block (3 levels) in create_worker' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `loop' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `block (2 levels) in create_worker' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `catch' | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `block in create_worker' | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
2016-06-16 15:54:10 [app] [I] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.rc8/app/views/katello/api/v2/common/async.json.rabl within katello/api/v2/layouts/resource (63.7ms)
2016-06-16 15:54:10 [app] [I] Completed 202 Accepted in 414ms (Views: 64.1ms | ActiveRecord: 66.8ms)

However, I have tried again to create a new host - and it is now booting successfully and imaging as I write this. After install I found one of the other repositories had corrupted (saltstack one) - trying to install salt-minion gave the following

Downloading packages:
PyYAML-3.11-1.el7.x86_64.rpm FAILED
python-tornado-4.2.1-1.el7.x86 FAILED 0% [ ] 0.0 B/s | 0 B --:--:-- ETA
python-zmq-14.7.0-1.el7.x86_64 FAILED 0% [ ] 0.0 B/s | 0 B --:--:-- ETA
(1/3): python-zmq-14.7.0-1.el7.x86_64.rpm 0% [ ] 0.0 B/s | 0 B --:--:-- ETA

Error downloading packages:
python-zmq-14.7.0-1.el7.x86_64: [Errno 256] No more mirrors to try.
PyYAML-3.11-1.el7.x86_64: [Errno 256] No more mirrors to try.
python-tornado-4.2.1-1.el7.x86_64: [Errno 256] No more mirrors to try.

I'm currently removing that repository from my content view, will delete it and re-try. Seems like the upgrade from RC5 to RC8 did something weird with my repositories/content views!

Further update, the imaged VMs were not being attached using subscription-manager. Error is:

[root@testweb01d ~]# subscription-manager refresh
This system is not yet registered. Try 'subscription-manager register --help' for more information.
[root@testweb01d ~]# subscription-manager register --org="NIWA" --activationkey="Apache Production Servers"
Validation failed: Base Content view 'CentOS7 Server' is not in environment 'Prod-Server'

Even though the activation key was configured to point at the newly created content view. I created a new activation key to test, some result

[root@testweb01d ~]# subscription-manager register --org="NIWA" --activationkey="NIWA Apache Production Servers"
Validation failed: Base Content view 'CentOS7 Server' is not in environment 'Prod-Server'
[root@testweb01d ~]#

So I went back to my content views, cancelled the task that was stuck attempting to delete the "CentOS7 Server" content view (which as per the above didn't have any versions associated with it anymore). Then used Pulp-admin to delete the repositories that were still hanging around -

pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-CentOS7-centosplus_x86_64
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-CentOS7-os_x86_64
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-CentOS7-extras_x86_64
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-CentOS7-updates_x86_64
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-CentOS7-cr_x86_64
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-EPEL7-EPEL7_x86_64
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-Postgres-Postgres_9_4_x86_64
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-Foreman-Foreman_1_10
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-Foreman-Foreman_plugins
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-Jenkins-Jenkins_LTS
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-Katello_Agent-Katello_Agent_3_0_x86_64
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-Softwarecollections-rhscl-ruby193-epel-7-x86_64
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-NIWA-sdt_apps
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-CentOS7-extras_x86_64
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-EPEL7-EPEL7_x86_64
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-Foreman-Foreman_plugins
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-CentOS7-os_x86_64
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-CentOS7-centosplus_x86_64
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-CentOS7-updates_x86_64
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-CentOS7-cr_x86_64
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-Jenkins-Jenkins_LTS
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-Foreman-Foreman_1_10
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-Postgres-Postgres_9_4_x86_64
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-NIWA-sdt_apps
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-Softwarecollections-rhscl-ruby193-epel-7-x86_64
pulp-admin rpm repo delete --repo-id=NIWA-Library-CentOS7_Server-Katello_Agent-Katello_Agent_3_0_x86_64

(note: Not all were still there, but a large number were)

and published a new version - which succeeded. I then published this through my environment to the "Prod-Server" environment, and attempted to subscribe a system again

[root@testweb01d ~]# subscription-manager register --org="NIWA" --activationkey="Apache Production Servers"
The system has been registered with ID: e61d4bf3-fd6e-4a83-a9dd-d6f9d08fe4f3

No products installed.
[root@testweb01d ~]#

Perfect. Also, with the saltstack repo deleted and re-created, salt-minion now installs fine again. I am about to try a re-image and will report what happens

Dylan Baars wrote:

Perfect. Also, with the saltstack repo deleted and re-created, salt-minion now installs fine again. I am about to try a re-image and will report what happens

and "new host" creation worked perfectly. Go figure, so it seems to be some sort of corruption to the "base content view" - couple of questions:

1. What is a base content view?
2. How does a "base content view" get created? I can't remember, but it was probably the first content view created
3. Can a base content view be changed?

It seems the base content view is linked to activation keys, so even if an activation key is associated with a different content view, because the base content view didn't have a version at the required environment path it wouldn't allow subscriptions, is this expected behaviour?

Is there anything I can provide to try and help figure out why the content view/pulp backend repository(ies) got corrupted somehow during the RC5-latest RC upgrade?

Thanks :-)
Dylan

Dylan,

I created https://pulp.plan.io/issues/2031 for comment #4.

Can you give a rundown of current issues? It looks like there were a couple of different problems happening, we may need to split them out into multiple redmine issues.

Chris Duryee wrote:

Dylan,

I created https://pulp.plan.io/issues/2031 for comment #4.

Can you give a rundown of current issues? It looks like there were a couple of different problems happening, we may need to split them out into multiple redmine issues.

Hi, sorry for the slow reply. Post blowing away the repos using pulp-admin and publishing a new version, everything has been good again. I think the primary issue was the one you've created a pulp issue for - everything else was fall out from me trying to correct things. I don't think the repo/content view was necessarily corrupted, while I can't test again, I'm 99% sure already managed VMs could still get data/install packages from the content view.