Plugins » Docker

Hey guys!

Have this weird problem and I can't figure out if that's expected behaviour or just my config being weird...

A simple explanation:
I have a registry a docker host registered as a compute resource and foreman with foreman-docker.
As my docker host has the socket exposed without a password, I created the compute resource without the password.
I added the registry and the password to access it to Foreman and I can browse all the images there without a problem.

If I try to deploy a new container using the private registry, I get auth errors from the registry as there is no X-Registry-Auth header with the login details.
If I add user/pass/mail combo to the compute resource definition, it can pull images from the private repo without a problem (using the details found in the header), but Foreman can't do other stuff any more as it's getting Fog's "Error connecting with the compute resource: Docker::Error::AuthenticationError"

After I remove the user/pass details from the compute resource, I can run the created container, but again can't pull images from the private registry.

So sounds like instead of passing eg. registry username/password for image pull Foreman is always passing the compute resource username/password for all auth purposes?

The setup is:

• Ubuntu 16.04
• Foreman 1.20.2
• Foreman-docker 4.1.0-1

• Ubuntu 16.04
• Docker version 18.09.2, build 6247962
• Docker socket listening on TCP (no auth)

• An internal registry we are using through Nexus3
• Basic auth needed to pull images

With no compute resource user/pass (dockerd debug):

Feb 27 16:44:24 infrasvc01 dockerd[21809]: time="2019-02-27T16:44:24.629877909Z" level=debug msg="Calling POST /v1.16/images/create?fromImage=registry.domain.tld%3A443%2Finfra%2Finfra-dnsresolver%3A1.0.0" 
Feb 27 16:44:24 infrasvc01 dockerd[21809]: time="2019-02-27T16:44:24.632494087Z" level=debug msg="hostDir: /etc/docker/certs.d/registry.domain.tld:443" 
Feb 27 16:44:24 infrasvc01 dockerd[21809]: time="2019-02-27T16:44:24.632836950Z" level=debug msg="Trying to pull registry.domain.tld:443/infra/infra-dnsresolver from https://registry..domain.tld:443 v2" 
Feb 27 16:44:24 infrasvc01 dockerd[21809]: time="2019-02-27T16:44:24.643275574Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry.domain.tld:443/v2/infra/infra-dnsresolver/manifests/1.0.0: no basic auth credentials" 
Feb 27 16:44:24 infrasvc01 dockerd[21809]: time="2019-02-27T16:44:24.643730667Z" level=error msg="Handler for POST /v1.16/images/create returned error: Get https://registry.domain.tld:443/v2/infra/infra-dnsresolver/manifests/1.0.0: no basic auth credentials" 

After I set the password for the compute resource:

Feb 27 17:14:46 infrasvc01 dockerd[21809]: time="2019-02-27T17:14:46.388363679Z" level=debug msg="Calling POST /v1.16/images/create?fromImage=registry.domain.tld%3A443%2Finfra%2Finfra-dnsresolver%3A1.0.0" 
Feb 27 17:14:46 infrasvc01 dockerd[21809]: time="2019-02-27T17:14:46.415146062Z" level=debug msg="hostDir: /etc/docker/certs.d/registry.domain.tld:443" 
Feb 27 17:14:46 infrasvc01 dockerd[21809]: time="2019-02-27T17:14:46.415626813Z" level=debug msg="Trying to pull registry.domain.tld:443/infra/infra-dnsresolver from https://registry.domain.tld:443 v2" 
Feb 27 17:14:46 infrasvc01 dockerd[21809]: time="2019-02-27T17:14:46.440741382Z" level=debug msg="Pulling ref from V2 registry: registry.domain.tld:443/infra/infra-dnsresolver:1.0.0" 
Feb 27 17:14:46 infrasvc01 dockerd[21809]: time="2019-02-27T17:14:46.441798415Z" level=debug msg="pulling blob \"sha256:3b37166ec61459e76e33282dda08f2a9cd698ca7e3d6bc44e6a6e7580cdeff8e\"" 
Feb 27 17:14:46 infrasvc01 dockerd[21809]: time="2019-02-27T17:14:46.442334753Z" level=debug msg="pulling blob \"sha256:504facff238fde83f1ca8f9f54520b4219c5b8f80be9616ddc52d31448a044bd\"" 

But because the password is set for the compute resource, but it's not expecting auth I get:

Fog::Errors::Fogdocker::AuthenticationError
Docker::Error::AuthenticationError
/usr/share/foreman/vendor/ruby/2.3.0/gems/fog-1.42.1/lib/fog/fogdocker/compute.rb:47:in `rescue in initialize'
/usr/share/foreman/vendor/ruby/2.3.0/gems/fog-1.42.1/lib/fog/fogdocker/compute.rb:37:in `initialize'
/usr/share/foreman/vendor/ruby/2.3.0/gems/fog-core-1.45.0/lib/fog/core/service.rb:115:in `new'
/usr/share/foreman/vendor/ruby/2.3.0/gems/fog-core-1.45.0/lib/fog/core/service.rb:115:in `new'
/usr/share/foreman/vendor/ruby/2.3.0/gems/fog-core-1.45.0/lib/fog/core/services_mixin.rb:16:in `new'
/usr/share/foreman/vendor/ruby/2.3.0/gems/fog-core-1.45.0/lib/fog/compute.rb:54:in `new'
/usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_docker-4.1.0/app/models/foreman_docker/docker.rb:163:in `client'
/usr/share/foreman/app/models/compute_resource.rb:181:in `find_vm_by_uuid'
/usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_docker-4.1.0/app/models/service/containers.rb:109:in `run_container'
/usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_docker-4.1.0/app/models/service/containers.rb:11:in `block in start_container!'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `block in transaction'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'
/usr/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `transaction'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.2.1/lib/active_record/transactions.rb:212:in `transaction'
/usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_docker-4.1.0/app/models/service/containers.rb:8:in `start_container!'
/usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_docker-4.1.0/app/controllers/containers/steps_controller.rb:76:in `create_container'
/usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_docker-4.1.0/app/controllers/containers/steps_controller.rb:22:in `update'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/vendor/ruby/2.3.0/gems/audited-4.8.0/lib/audited/sweeper.rb:14:in `around'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/vendor/ruby/2.3.0/gems/audited-4.8.0/lib/audited/sweeper.rb:14:in `around'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/apipie-rails-0.5.15/lib/apipie/static_dispatcher.rb:66:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/apipie-rails-0.5.15/lib/apipie/extractor/recorder.rb:137:in `call'
/usr/share/foreman/lib/middleware/telemetry.rb:9:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/apipie-rails-0.5.15/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/tempfile_reaper.rb:15:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/etag.rb:25:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/conditional_get.rb:38:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/head.rb:12:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/usr/share/foreman/lib/middleware/logging_context.rb:23:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:232:in `context'
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/method_override.rb:22:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/runtime.rb:22:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/sendfile.rb:111:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/urlmap.rb:68:in `block in call'
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `each'
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `call'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:160:in `accept_and_process_next_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:113:in `main_loop'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads'
/usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'
/usr/share/foreman/vendor/ruby/2.3.0/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'