Project

General

Profile

Actions

Bug #22925

closed

REGRESSION: foreman fails to delete a host when getting a 404 from a proxy

Added by Ohad Levy about 6 years ago. Updated about 6 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
Orchestration
Target version:
-
Difficulty:
Triaged:
Fixed in Releases:
Found in Releases:

Description

in my test scenario, I've asked to delete a host which did not have a DHCP reservation, but had a configured DHCP proxy.
the proxy returned a 404 NOT FOUND, in the past, foreman would be smart enough to know that this is the desired state and as such, wont break the orchestration process (e.g. I'm asking it to delete it, its already not there, so continue the process). but this is now not possible with nightly.

2018-03-18T13:11:51 [I|app|] Started DELETE "/hosts/heath-horstead.sat.lab.tlv.example.com" for 10.35.2.157 at 2018-03-18 13:11:51 +0000
2018-03-18T13:11:51 [I|app|d4163] Processing by HostsController#destroy as HTML
2018-03-18T13:11:51 [I|app|d4163]   Parameters: {"authenticity_token"=>"jKa2iHFYeHrw/y0+X8F2kKJQ10PzSvyS9HJI/LQkb1Donb+oJhjs4X4eKiskRHaYh1OArTzBr2KsbgCU/OTaug==", "id"=>"heath-horstead.sat.lab.tlv.example.com"}
2018-03-18T13:11:51 [I|app|d4163] Current user: admin (administrator)
2018-03-18T13:11:51 [I|app|d4163] Delete the DNS A record for heath-horstead.sat.lab.tlv.example.com/10.35.27.9
2018-03-18T13:11:52 [I|app|d4163] Delete the DNS PTR record for 10.35.27.9/heath-horstead.sat.lab.tlv.example.com
2018-03-18T13:11:52 [I|app|d4163] Delete DHCP reservation heath-horstead.sat.lab.tlv.example.com for heath-horstead.sat.lab.tlv.example.com-52:54:00:86:dc:8e/10.35.27.9
2018-03-18T13:11:52 [W|app|d4163] Remove DHCP Settings for heath-horstead.sat.lab.tlv.example.com task failed with the following error: ERF12-0635 [ProxyAPI::ProxyException]: Unable to delete DHCP entry for 52:54:00:86:dc:8e ([RestClient::NotFound]: 404 Not Found) for proxy http://h01.sat.lab.tlv.example.com:8443/dhcp
 | ProxyAPI::ProxyException: ERF12-0635 [ProxyAPI::ProxyException]: Unable to delete DHCP entry for 52:54:00:86:dc:8e ([RestClient::NotFound]: 404 Not Found) for proxy http://h01.sat.lab.tlv.example.com:8443/dhcp
 | /home/foreman/git/foreman/lib/proxy_api/dhcp.rb:99:in `rescue in delete'
 | /home/foreman/git/foreman/lib/proxy_api/dhcp.rb:97:in `delete'
 | /home/foreman/git/foreman/lib/net/dhcp/record.rb:34:in `destroy'
 | /home/foreman/git/foreman/app/models/concerns/orchestration/dhcp.rb:67:in `block in del_dhcp'
 | /home/foreman/git/foreman/app/models/concerns/orchestration/dhcp.rb:67:in `each'
 | /home/foreman/git/foreman/app/models/concerns/orchestration/dhcp.rb:67:in `all?'
 | /home/foreman/git/foreman/app/models/concerns/orchestration/dhcp.rb:67:in `del_dhcp'
 | /home/foreman/git/foreman/app/models/concerns/orchestration.rb:222:in `execute'
 | /home/foreman/git/foreman/app/models/concerns/orchestration.rb:145:in `block in process'
 | /home/foreman/git/foreman/app/models/concerns/orchestration.rb:137:in `each'
 | /home/foreman/git/foreman/app/models/concerns/orchestration.rb:137:in `process'
 | /home/foreman/git/foreman/app/models/concerns/orchestration.rb:60:in `on_destroy'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:413:in `block in make_lambda'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:235:in `block in halting_and_conditional'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:511:in `block in invoke_after'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:511:in `each'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:511:in `invoke_after'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:132:in `run_callbacks'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:827:in `_run_destroy_callbacks'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/callbacks.rb:321:in `destroy'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:303:in `block in destroy'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:384:in `block in with_transaction_returning_status'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/database_statements.rb:233:in `transaction'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:210:in `transaction'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:381:in `with_transaction_returning_status'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:303:in `destroy'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/persistence.rb:211:in `destroy!'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/associations/has_many_association.rb:105:in `each'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/associations/has_many_association.rb:105:in `delete_records'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/associations/collection_association.rb:394:in `remove_records'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/associations/collection_association.rb:387:in `block in delete_or_destroy'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/associations/collection_association.rb:129:in `block in transaction'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/database_statements.rb:233:in `transaction'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:210:in `transaction'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/associations/collection_association.rb:128:in `transaction'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/associations/collection_association.rb:387:in `delete_or_destroy'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/associations/collection_association.rb:196:in `destroy'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/associations/collection_association.rb:169:in `destroy_all'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/associations/has_many_association.rb:26:in `handle_dependency'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/associations/builder/association.rb:140:in `block in add_destroy_callbacks'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:413:in `instance_exec'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:413:in `block in make_lambda'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:197:in `block (2 levels) in halting'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:601:in `block (2 levels) in default_terminator'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:600:in `catch'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:600:in `block in default_terminator'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:198:in `block in halting'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:507:in `block in invoke_before'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:507:in `each'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:507:in `invoke_before'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:130:in `run_callbacks'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:827:in `_run_destroy_callbacks'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/callbacks.rb:321:in `destroy'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:303:in `block in destroy'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:384:in `block in with_transaction_returning_status'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/database_statements.rb:235:in `block in transaction'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/transaction.rb:194:in `block in within_new_transaction'
 | /opt/rh/rh-ruby24/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/transaction.rb:191:in `within_new_transaction'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/database_statements.rb:235:in `transaction'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:210:in `transaction'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:381:in `with_transaction_returning_status'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:303:in `destroy'
 | /home/foreman/git/foreman/app/controllers/hosts_controller.rb:136:in `destroy'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/abstract_controller/base.rb:186:in `process_action'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_controller/metal/rendering.rb:30:in `process_action'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:108:in `block in run_callbacks'
 | /home/foreman/git/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
 | /home/foreman/git/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
 | /home/foreman/git/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
 | /home/foreman/gems/gems/audited-4.7.0/lib/audited/sweeper.rb:14:in `around'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
 | /home/foreman/gems/gems/audited-4.7.0/lib/audited/sweeper.rb:14:in `around'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:135:in `run_callbacks'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/abstract_controller/callbacks.rb:19:in `process_action'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_controller/metal/rescue.rb:20:in `process_action'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/notifications.rb:166:in `block in instrument'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/notifications.rb:166:in `instrument'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_controller/metal/params_wrapper.rb:252:in `process_action'
 | /home/foreman/gems/gems/activerecord-5.1.4/lib/active_record/railties/controller_runtime.rb:22:in `process_action'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/abstract_controller/base.rb:124:in `process'
 | /home/foreman/gems/gems/actionview-5.1.4/lib/action_view/rendering.rb:30:in `process'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_controller/metal.rb:189:in `dispatch'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_controller/metal.rb:253:in `dispatch'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_dispatch/routing/route_set.rb:49:in `dispatch'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_dispatch/routing/route_set.rb:31:in `serve'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_dispatch/journey/router.rb:50:in `block in serve'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_dispatch/journey/router.rb:33:in `each'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_dispatch/journey/router.rb:33:in `serve'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_dispatch/routing/route_set.rb:834:in `call'
 | /home/foreman/gems/gems/apipie-rails-0.5.6/lib/apipie/static_dispatcher.rb:65:in `call'
 | /home/foreman/gems/gems/apipie-rails-0.5.6/lib/apipie/extractor/recorder.rb:136:in `call'
 | /home/foreman/git/foreman/lib/middleware/telemetry.rb:9:in `call'
 | /home/foreman/gems/gems/apipie-rails-0.5.6/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 | /home/foreman/git/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call'
 | /home/foreman/gems/gems/rack-2.0.4/lib/rack/etag.rb:25:in `call'
 | /home/foreman/gems/gems/rack-2.0.4/lib/rack/conditional_get.rb:38:in `call'
 | /home/foreman/gems/gems/rack-2.0.4/lib/rack/head.rb:12:in `call'
 | /home/foreman/git/foreman/lib/middleware/logging_context.rb:23:in `call'
 | /home/foreman/gems/gems/rack-2.0.4/lib/rack/session/abstract/id.rb:232:in `context'
 | /home/foreman/gems/gems/rack-2.0.4/lib/rack/session/abstract/id.rb:226:in `call'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/cookies.rb:613:in `call'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/callbacks.rb:26:in `block in call'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:97:in `run_callbacks'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/callbacks.rb:24:in `call'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/debug_exceptions.rb:59:in `call'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
 | /home/foreman/gems/gems/railties-5.1.4/lib/rails/rack/logger.rb:36:in `call_app'
 | /home/foreman/gems/gems/railties-5.1.4/lib/rails/rack/logger.rb:26:in `call'
 | /home/foreman/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/request_id.rb:25:in `call'
 | /home/foreman/gems/gems/rack-2.0.4/lib/rack/method_override.rb:22:in `call'
 | /home/foreman/gems/gems/rack-2.0.4/lib/rack/runtime.rb:22:in `call'
 | /home/foreman/gems/gems/activesupport-5.1.4/lib/active_support/cache/strategy/local_cache_middleware.rb:27:in `call'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/executor.rb:12:in `call'
 | /home/foreman/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/static.rb:125:in `call'
 | /home/foreman/gems/gems/rack-2.0.4/lib/rack/sendfile.rb:111:in `call'
 | /home/foreman/gems/gems/secure_headers-3.7.3/lib/secure_headers/middleware.rb:12:in `call'
 | /home/foreman/gems/gems/railties-5.1.4/lib/rails/engine.rb:522:in `call'
 | /home/foreman/gems/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `public_send'
 | /home/foreman/gems/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `method_missing'
 | /home/foreman/gems/gems/rack-2.0.4/lib/rack/urlmap.rb:68:in `block in call'
 | /home/foreman/gems/gems/rack-2.0.4/lib/rack/urlmap.rb:53:in `each'
 | /home/foreman/gems/gems/rack-2.0.4/lib/rack/urlmap.rb:53:in `call'
 | /root/.gem/gems/passenger-5.0.7/lib/phusion_passenger/rack/thread_handler_extension.rb:89:in `process_request'
 | /root/.gem/gems/passenger-5.0.7/lib/phusion_passenger/request_handler/thread_handler.rb:155:in `accept_and_process_next_request'
 | /root/.gem/gems/passenger-5.0.7/lib/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'
 | /root/.gem/gems/passenger-5.0.7/lib/phusion_passenger/request_handler.rb:414:in `block (3 levels) in start_threads'
 | /root/.gem/gems/passenger-5.0.7/lib/phusion_passenger/utils.rb:111:in `block in create_thread_and_abort_on_exception'
 | /home/foreman/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2018-03-18T13:11:52 [W|app|d4163] Rolling back due to a problem: [#<Orchestration::Task:0x007f64375e1558 @name="Remove DHCP Settings for heath-horstead.sat.lab.tlv.example.com", @id="dhcp_remove_52:54:00:86:dc:8e", @status="failed", @priority=5, @action=[#<Nic::Managed id: 248150, mac: "52:54:00:86:dc:8e", ip: "10.35.27.9", type: "Nic::Managed", name: "heath-horstead.sat.lab.tlv.example.com", host_id: 84890, subnet_id: 1, domain_id: 6, attrs: {}, created_at: "2018-02-25 09:49:26", updated_at: "2018-02-25 09:49:26", provider: nil, username: nil, password: nil, virtual: false, link: true, identifier: "", tag: "", attached_to: "", managed: true, mode: "balance-rr", attached_devices: "", bond_options: "", primary: true, provision: true, compute_attributes: {"bridge"=>"br180", "model"=>"virtio", "type"=>"bridge"}, execution: true, ip6: "", subnet6_id: nil>, :del_dhcp], @created=1521378711.7660596, @timestamp=2018-03-18 13:11:52 UTC>]
2018-03-18T13:11:52 [I|app|d4163] Add DNS PTR record for 10.35.27.9/heath-horstead.sat.lab.tlv.example.com
2018-03-18T13:11:52 [I|app|d4163] Add DNS A record for heath-horstead.sat.lab.tlv.example.com/10.35.27.9
2018-03-18T13:11:52 [I|app|d4163] Processed 2 tasks from queue 'Host::Managed Main', completed 0/6
2018-03-18T13:11:52 [E|app|d4163] Task 'Remove IPv4 DNS record for heath-horstead.sat.lab.tlv.example.com' *rollbacked*
2018-03-18T13:11:52 [E|app|d4163] Task 'Remove Reverse IPv4 DNS record for heath-horstead.sat.lab.tlv.example.com' *rollbacked*
2018-03-18T13:11:52 [E|app|d4163] Task 'Remove DHCP Settings for heath-horstead.sat.lab.tlv.example.com' *failed*
2018-03-18T13:11:52 [E|app|d4163] Task 'Delete TFTP PXEGrub2 config for heath-horstead.sat.lab.tlv.example.com' *canceled*
2018-03-18T13:11:52 [E|app|d4163] Task 'Delete TFTP PXELinux config for heath-horstead.sat.lab.tlv.example.com' *canceled*
2018-03-18T13:11:52 [E|app|d4163] Task 'Delete TFTP PXEGrub config for heath-horstead.sat.lab.tlv.example.com' *canceled*
2018-03-18T13:11:52 [I|app|d4163] Failed to save: Remove DHCP Settings for heath-horstead.sat.lab.tlv.example.com task failed with the following error: ERF12-0635 [ProxyAPI::ProxyException]: Unable to delete DHCP entry for 52:54:00:86:dc:8e ([RestClient::NotFound]: 404 Not Found) for proxy http://h01.sat.lab.tlv.example.com:8443/dhcp
2018-03-18T13:11:52 [I|app|d4163] Redirected to https://fdev.sat.lab.tlv.example.com/hosts
2018-03-18T13:11:52 [I|app|d4163] Completed 302 Found in 1220ms (ActiveRecord: 178.9ms)


Related issues 2 (1 open1 closed)

Related to Foreman - Refactor #19295: Use new proxy API for getting information from DHCPClosedMarek Hulán04/18/2017Actions
Has duplicate Foreman - Bug #19321: Unable to delete a host due to DHCP orchestration failure with older smart proxyNew04/19/2017Actions
Actions #1

Updated by Ohad Levy about 6 years ago

  • Related to Refactor #19295: Use new proxy API for getting information from DHCP added
Actions #2

Updated by Ohad Levy about 6 years ago

  • Has duplicate Bug #19321: Unable to delete a host due to DHCP orchestration failure with older smart proxy added
Actions #3

Updated by Ohad Levy about 6 years ago

  • Status changed from New to Duplicate
Actions

Also available in: Atom PDF