Bug #11982
openDocker pull unable to find image
Description
One of my consumers pushed a new image into their repository. For some reason we are unable to retrieve said image.
Here is some information from the current
[root@box ~]# docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE <none> <none> 12621274fbc3 5 weeks ago 595.6 MB cc1_prereqs cc1_prereqs 85c719651f54 5 weeks ago 595.6 MB katello.net:5000/org-dockah-dockah latest 85c719651f54 5 weeks ago 595.6 MB katello.net:5000/org-dockah-dockah cc1_prereqs 85c719651f54 5 weeks ago 595.6 MB katello.net:5000/org-library-dockah-dockah-dockah cc1_prereqs 85c719651f54 5 weeks ago 595.6 MB katello.net:5000/org-library-dockah-dockah-dockah latest 85c719651f54 5 weeks ago 595.6 MB ..snip..
When we run a pull:
[root@infraqe-jenkins-n02 ~]# docker pull katello.net:5000/org-dockah-dockah:cc1_prereqs Pulling repository katello.net:5000/org-dockah-dockah 14a6fa0550f9: Download complete FATA[0001] could not find image: no such id: 14a6fa0550f98b8ba3d64ae47cc206d625a66ce6cbcc931509ddfbbbf72d9566 [root@infraqe-jenkins-n02 ~]# docker pull katello.net:5000/org-library-dockah-dockah-dockah:cc1_prereqs Pulling repository katello.net:5000/org-library-dockah-dockah-dockah 14a6fa0550f9: Download complete FATA[0001] could not find image: no such id: 14a6fa0550f98b8ba3d64ae47cc206d625a66ce6cbcc931509ddfbbbf72d9566
And looking at the logs in crane_access_ssl.log and foreman-ssl_access_ssl.log (the only files that seem to be updated during these transactions):
==> ./httpd/crane_access_ssl.log <== 10.14.238.212 - - [28/Sep/2015:14:35:01 +0000] "GET /v2/ HTTP/1.1" 404 200 "-" "docker/1.6.2 go/go1.4.2 kernel/2.6.32-431.29.2.el6.x86_64 os/linux arch/amd64" 10.14.238.212 - - [28/Sep/2015:14:35:01 +0000] "GET /v1/_ping HTTP/1.1" 200 4 "-" "docker/1.6.2 go/go1.4.2 kernel/2.6.32-431.29.2.el6.x86_64 os/linux arch/amd64" 10.14.238.212 - - [28/Sep/2015:14:35:01 +0000] "GET /v1/_ping HTTP/1.1" 200 4 "-" "docker/1.6.2 go/go1.4.2 kernel/2.6.32-431.29.2.el6.x86_64 os/linux arch/amd64" 10.14.238.212 - - [28/Sep/2015:14:35:02 +0000] "GET /v1/repositories/org-dockah-dockah/images HTTP/1.1" 200 2660 "-" "docker/1.6.2 go/go1.4.2 kernel/2.6.32-431.29.2.el6.x86_64 os/linux arch/amd64" 10.14.238.212 - - [28/Sep/2015:14:35:02 +0000] "GET /v1/repositories/library/org-dockah-dockah/tags HTTP/1.1" 200 83 "-" "docker/1.6.2 go/go1.4.2 kernel/2.6.32-431.29.2.el6.x86_64 os/linux arch/amd64" 10.14.238.212 - - [28/Sep/2015:14:35:02 +0000] "GET /v1/images/14a6fa0550f98b8ba3d64ae47cc206d625a66ce6cbcc931509ddfbbbf72d9566/ancestry HTTP/1.1" 302 303 "-" "docker/1.6.2 go/go1.4.2 kernel/2.6.32-431.29.2.el6.x86_64 os/linux arch/amd64" ==> ./httpd/foreman-ssl_access_ssl.log <== 10.14.238.212 - - [28/Sep/2015:14:35:03 +0000] "GET /pulp/docker/org-library-dockah-dockah-dockah/14a6fa0550f98b8ba3d64ae47cc206d625a66ce6cbcc931509ddfbbbf72d9566/ancestry HTTP/1.1" 200 2 "https://katello.net:5000/v1/images/14a6fa0550f98b8ba3d64ae47cc206d625a66ce6cbcc931509ddfbbbf72d9566/ancestry" "docker/1.6.2 go/go1.4.2 kernel/2.6.32-431.29.2.el6.x86_64 os/linux arch/amd64" 10.23.251.116 - - [28/Sep/2015:14:35:02 +0000] "POST /api/hosts/facts HTTP/1.1" 201 918 "-" "-" 10.23.251.116 - - [28/Sep/2015:14:35:03 +0000] "GET /node/katello.net?format=yml HTTP/1.1" 200 519 "-" "-" 10.23.251.116 - - [28/Sep/2015:14:35:04 +0000] "POST /api/reports HTTP/1.1" 201 520 "-" "-" ==> ./httpd/crane_access_ssl.log <== 10.14.238.212 - - [28/Sep/2015:14:35:07 +0000] "GET /v2/ HTTP/1.1" 404 200 "-" "docker/1.6.2 go/go1.4.2 kernel/2.6.32-431.29.2.el6.x86_64 os/linux arch/amd64" 10.14.238.212 - - [28/Sep/2015:14:35:08 +0000] "GET /v1/_ping HTTP/1.1" 200 4 "-" "docker/1.6.2 go/go1.4.2 kernel/2.6.32-431.29.2.el6.x86_64 os/linux arch/amd64" 10.14.238.212 - - [28/Sep/2015:14:35:08 +0000] "GET /v1/_ping HTTP/1.1" 200 4 "-" "docker/1.6.2 go/go1.4.2 kernel/2.6.32-431.29.2.el6.x86_64 os/linux arch/amd64" 10.14.238.212 - - [28/Sep/2015:14:35:08 +0000] "GET /v1/repositories/org-library-dockah-dockah-dockah/images HTTP/1.1" 200 2736 "-" "docker/1.6.2 go/go1.4.2 kernel/2.6.32-431.29.2.el6.x86_64 os/linux arch/amd64" 10.14.238.212 - - [28/Sep/2015:14:35:08 +0000] "GET /v1/repositories/library/org-library-dockah-dockah-dockah/tags HTTP/1.1" 200 83 "-" "docker/1.6.2 go/go1.4.2 kernel/2.6.32-431.29.2.el6.x86_64 os/linux arch/amd64" 10.14.238.212 - - [28/Sep/2015:14:35:09 +0000] "GET /v1/images/14a6fa0550f98b8ba3d64ae47cc206d625a66ce6cbcc931509ddfbbbf72d9566/ancestry HTTP/1.1" 302 303 "-" "docker/1.6.2 go/go1.4.2 kernel/2.6.32-431.29.2.el6.x86_64 os/linux arch/amd64" ==> ./httpd/foreman-ssl_access_ssl.log <== 10.14.238.212 - - [28/Sep/2015:14:35:09 +0000] "GET /pulp/docker/org-library-dockah-dockah-dockah/14a6fa0550f98b8ba3d64ae47cc206d625a66ce6cbcc931509ddfbbbf72d9566/ancestry HTTP/1.1" 200 2 "https://katello.net:5000/v1/images/14a6fa0550f98b8ba3d64ae47cc206d625a66ce6cbcc931509ddfbbbf72d9566/ancestry" "docker/1.6.2 go/go1.4.2 kernel/2.6.32-431.29.2.el6.x86_64 os/linux arch/amd64"
I thought maybe the problem was that it needed to be promoted to the library, even though that first set is outside the library. Which seemed counter-intuitive based on the content life cycle elsewhere in Katello.
I wouldn't figure out how to pull a view of the images in the repo using the hammer cli so i'm paraphrasing here from the webui:
Image ID tag ..snip.. d0ff3bc98d4296f2a6eec9f8c9ed48c37f51c8d82100992125cac3b850bed18d a6c203c9a8f0181e804c6d0b07dc7ce3bd4f60bc4ff50359f23cbb1fc2d2d175 85bcec70619504ed38ec9741b273fab618c2f69c639c1c0891e88a14a5d01796 973cbf29c390db8a50636b73036d48ba23923a80c8e46c84a42017118132330f 14a6fa0550f98b8ba3d64ae47cc206d625a66ce6cbcc931509ddfbbbf72d9566 cc1_prereqs 1124621907d8b2320628deeddd5fd1ce33ca87da404509283c5ccafe66d13cd3 a246075f963f83cdc5197132cea846e8f4cb369be5b921a19b761245daf214a7 0c6d61c6e28c4d29c027077143b0f7e3470bdb8c763015c4ec6fa05bd9f87293
As you can see the image id and the tag match what we saw in the output above.