Bug #26182
Updated by Lovro Galesic about 5 years ago
Hey guys! Have this weird problem and I can't figure out if that's expected behaviour or just my config being weird... A simple explanation: I have a registry a docker host registered as a compute resource and foreman with foreman-docker. As my docker host has the socket exposed without a password, I created the compute resource without the password. I added the registry and the password to access it to Foreman and I can browse all the images there without a problem. If I try to deploy a new container using the private registry, I get auth errors from the registry as there is no X-Registry-Auth header with the login details. If I add user/pass/mail combo to the compute resource definition, it can pull images from the private repo without a problem (using the details found in the header), but Foreman can't do other stuff any more as it's getting Fog's "Error connecting with the compute resource: Docker::Error::AuthenticationError" After I remove the user/pass details from the compute resource, I can run the created container, but again can't pull images from the private registry. So sounds like instead of passing eg. registry username/password for image pull Foreman is always passing the compute resource username/password for all auth purposes? The setup is: *Foreman server:* * Ubuntu 16.04 * Foreman 1.20.2 * Foreman-docker 4.1.0-1 *Docker host:* * Ubuntu 16.04 * Docker version 18.09.2, build 6247962 * Docker socket listening on TCP (no auth) *Docker Registry:* * An internal registry we are using through Nexus3 * Basic auth needed to pull images With no compute resource user/pass (dockerd debug): <pre><code class="ruby"> Feb 27 16:44:24 infrasvc01 dockerd[21809]: time="2019-02-27T16:44:24.629877909Z" level=debug msg="Calling POST /v1.16/images/create?fromImage=registry.domain.tld%3A443%2Finfra%2Finfra-dnsresolver%3A1.0.0" Feb 27 16:44:24 infrasvc01 dockerd[21809]: time="2019-02-27T16:44:24.632494087Z" level=debug msg="hostDir: /etc/docker/certs.d/registry.domain.tld:443" Feb 27 16:44:24 infrasvc01 dockerd[21809]: time="2019-02-27T16:44:24.632836950Z" level=debug msg="Trying to pull registry.domain.tld:443/infra/infra-dnsresolver from https://registry..domain.tld:443 v2" Feb 27 16:44:24 infrasvc01 dockerd[21809]: time="2019-02-27T16:44:24.643275574Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry.domain.tld:443/v2/infra/infra-dnsresolver/manifests/1.0.0: no basic auth credentials" Feb 27 16:44:24 infrasvc01 dockerd[21809]: time="2019-02-27T16:44:24.643730667Z" level=error msg="Handler for POST /v1.16/images/create returned error: Get https://registry.domain.tld:443/v2/infra/infra-dnsresolver/manifests/1.0.0: no basic auth credentials" </code></pre> After I set the password for the compute resource: <pre><code class="ruby"> Feb 27 17:14:46 infrasvc01 dockerd[21809]: time="2019-02-27T17:14:46.388363679Z" level=debug msg="Calling POST /v1.16/images/create?fromImage=registry.domain.tld%3A443%2Finfra%2Finfra-dnsresolver%3A1.0.0" Feb 27 17:14:46 infrasvc01 dockerd[21809]: time="2019-02-27T17:14:46.415146062Z" level=debug msg="hostDir: /etc/docker/certs.d/registry.domain.tld:443" Feb 27 17:14:46 infrasvc01 dockerd[21809]: time="2019-02-27T17:14:46.415626813Z" level=debug msg="Trying to pull registry.domain.tld:443/infra/infra-dnsresolver registry.xcl.ie:443/infra/infra-dnsresolver from https://registry.domain.tld:443 v2" Feb 27 17:14:46 infrasvc01 dockerd[21809]: time="2019-02-27T17:14:46.440741382Z" level=debug msg="Pulling ref from V2 registry: registry.domain.tld:443/infra/infra-dnsresolver:1.0.0" Feb 27 17:14:46 infrasvc01 dockerd[21809]: time="2019-02-27T17:14:46.441798415Z" level=debug msg="pulling blob \"sha256:3b37166ec61459e76e33282dda08f2a9cd698ca7e3d6bc44e6a6e7580cdeff8e\"" Feb 27 17:14:46 infrasvc01 dockerd[21809]: time="2019-02-27T17:14:46.442334753Z" level=debug msg="pulling blob \"sha256:504facff238fde83f1ca8f9f54520b4219c5b8f80be9616ddc52d31448a044bd\"" </code></pre> But because the password is set for the compute resource, but it's not expecting auth I get: <pre><code class="ruby"> Fog::Errors::Fogdocker::AuthenticationError Docker::Error::AuthenticationError /usr/share/foreman/vendor/ruby/2.3.0/gems/fog-1.42.1/lib/fog/fogdocker/compute.rb:47:in `rescue in initialize' /usr/share/foreman/vendor/ruby/2.3.0/gems/fog-1.42.1/lib/fog/fogdocker/compute.rb:37:in `initialize' /usr/share/foreman/vendor/ruby/2.3.0/gems/fog-core-1.45.0/lib/fog/core/service.rb:115:in `new' /usr/share/foreman/vendor/ruby/2.3.0/gems/fog-core-1.45.0/lib/fog/core/service.rb:115:in `new' /usr/share/foreman/vendor/ruby/2.3.0/gems/fog-core-1.45.0/lib/fog/core/services_mixin.rb:16:in `new' /usr/share/foreman/vendor/ruby/2.3.0/gems/fog-core-1.45.0/lib/fog/compute.rb:54:in `new' /usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_docker-4.1.0/app/models/foreman_docker/docker.rb:163:in `client' /usr/share/foreman/app/models/compute_resource.rb:181:in `find_vm_by_uuid' /usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_docker-4.1.0/app/models/service/containers.rb:109:in `run_container' /usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_docker-4.1.0/app/models/service/containers.rb:11:in `block in start_container!' /usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `block in transaction' /usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction' /usr/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize' /usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction' /usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `transaction' /usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.2.1/lib/active_record/transactions.rb:212:in `transaction' /usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_docker-4.1.0/app/models/service/containers.rb:8:in `start_container!' /usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_docker-4.1.0/app/controllers/containers/steps_controller.rb:76:in `create_container' /usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_docker-4.1.0/app/controllers/containers/steps_controller.rb:22:in `update' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks' /usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks' /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks' /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks' /usr/share/foreman/vendor/ruby/2.3.0/gems/audited-4.8.0/lib/audited/sweeper.rb:14:in `around' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks' /usr/share/foreman/vendor/ruby/2.3.0/gems/audited-4.8.0/lib/audited/sweeper.rb:14:in `around' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action' /usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/apipie-rails-0.5.15/lib/apipie/static_dispatcher.rb:66:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/apipie-rails-0.5.15/lib/apipie/extractor/recorder.rb:137:in `call' /usr/share/foreman/lib/middleware/telemetry.rb:9:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/apipie-rails-0.5.15/lib/apipie/middleware/checksum_in_headers.rb:27:in `call' /usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/tempfile_reaper.rb:15:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/etag.rb:25:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/conditional_get.rb:38:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/head.rb:12:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call' /usr/share/foreman/lib/middleware/logging_context.rb:23:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:232:in `context' /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app' /usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/method_override.rb:22:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/runtime.rb:22:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/sendfile.rb:111:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call' /usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send' /usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing' /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/urlmap.rb:68:in `block in call' /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `each' /usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `call' /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:160:in `accept_and_process_next_request' /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:113:in `main_loop' /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads' /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' /usr/share/foreman/vendor/ruby/2.3.0/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context' </code></pre>