Project

General

Profile

Actions
Copy link

Bug #32178

closed

deb repo - apt install - File has unexpected size - http pipeline

Added by Rossen G about 3 years ago. Updated about 3 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
-
Category:
Repositories
Target version:
Katello Recycle Bin
Difficulty:
Triaged:
No
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:
Katello 3.18.1, Katello 3.18.2
Red Hat JIRA:

Description

I'm getting unexpected size error on ubuntu 18.04 when trying to install from a deb repository.
The problem seems to be caused by apt pipelining requests.
Only happens on katello deb repos, and when there are many packages to download as seen below.
I have ubuntu mirror in the pub directory, not through katello.

My guess is this is issue with pulp, as httpd proxies the requests to it, right?

# apt install -y ...
<snip>
Get:29 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 libxi6 amd64 2:1.7.9-1 [29.2 kB]
Get:30 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 libxrender1 amd64 1:0.9.10-1 [18.7 kB]
Get:31 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic-updates/main amd64 x11-common all 1:7.7+19ubuntu7.1 [22.5 kB]
Get:32 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 libxtst6 amd64 2:1.2.3-1 [12.8 kB]
Get:33 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic-updates/universe amd64 openjdk-8-jre-headless amd64 8u282-b08-0ubuntu1~18.04 [28.2 MB]
Get:34 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic-updates/main amd64 ca-certificates-java all 20180516ubuntu1~18.04.1 [12.2 kB]
Get:35 http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Scylla/scylladb-enterprise-2020_1_bionic default/all amd64 scylla-enterprise-conf amd64 2020.1.7-0.20210304.e42189e09-1 [11.8 kB]
Get:36 http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Scylla/scylladb-enterprise-2020_1_bionic default/all amd64 scylla-enterprise-python3 amd64 3.7.7-0.20210304.e42189e09-1 [20.0 MB]
Err:36 http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Scylla/scylladb-enterprise-2020_1_bionic default/all amd64 scylla-enterprise-python3 amd64 3.7.7-0.20210304.e42189e09-1
  File has unexpected size (20004486 != 86464). Mirror sync in progress? [IP: 10.1.1.8 80]
  Hashes of expected file:
   - SHA256:51a6c7c50ebd72be856b02a9fbe8ac39739e3f0c36bcf6a7c5f1d2d83092286c
   - SHA1:02cf7a581beeaf1b067600c72c268dc0b9e713bd [weak]
   - MD5Sum:aa1e11ed2ba92ca3da8a9785cdfae2ed [weak]
Get:37 http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Scylla/scylladb-enterprise-2020_1_bionic default/all amd64 scylla-enterprise-server amd64 2020.1.7-0.20210304.e42189e09-1 [35.7 MB]
Err:37 http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Scylla/scylladb-enterprise-2020_1_bionic default/all amd64 scylla-enterprise-server amd64 2020.1.7-0.20210304.e42189e09-1
  File has unexpected size (35725644 != 86464). Mirror sync in progress? [IP: 10.1.1.8 80]
  Hashes of expected file:
   - SHA256:74cbe6f7ba359869f84bf671941e9e7d60b4c302de836f35e73f12b37fe57c46
   - SHA1:ba106ac7ae29c5821a9d3c01d6f601352081d6e3 [weak]
   - MD5Sum:83f1ad8d23dcbf950096c28b88059442 [weak]
Get:38 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic-updates/universe amd64 sntp amd64 1:4.2.8p10+dfsg-5ubuntu7.3 [86.5 kB]
Fetched 38.1 MB in 1s (53.8 MB/s)
E: Failed to fetch http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Scylla/scylladb-enterprise-2020_1_bionic/pool/all/s/scylla-enterprise-python3/scylla-enterprise-python3_3.7.7-0.20210304.e42189e09-1_amd64.deb  File has unexpected size (20004486 != 86464). Mirror sync in progress? [IP: 10.1.1.8 80]
   Hashes of expected file:
    - SHA256:51a6c7c50ebd72be856b02a9fbe8ac39739e3f0c36bcf6a7c5f1d2d83092286c
    - SHA1:02cf7a581beeaf1b067600c72c268dc0b9e713bd [weak]
    - MD5Sum:aa1e11ed2ba92ca3da8a9785cdfae2ed [weak]
E: Failed to fetch http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Scylla/scylladb-enterprise-2020_1_bionic/pool/all/s/scylla-enterprise-server/scylla-enterprise-server_2020.1.7-0.20210304.e42189e09-1_amd64.deb  File has unexpected size (35725644 != 86464). Mirror sync in progress? [IP: 10.1.1.8 80]
   Hashes of expected file:
    - SHA256:74cbe6f7ba359869f84bf671941e9e7d60b4c302de836f35e73f12b37fe57c46
    - SHA1:ba106ac7ae29c5821a9d3c01d6f601352081d6e3 [weak]
    - MD5Sum:83f1ad8d23dcbf950096c28b88059442 [weak]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

# apt -oAcquire::http::Pipeline-Depth=0 install ...
# install completes with issue

Files

apt_unexpected_size.png View apt_unexpected_size.png 183 KB Rossen G, 03/24/2021 11:46 PM
Actions
Copy link
 #1

Updated by Rossen G about 3 years ago

Correcting a typo:

# apt -oAcquire::http::Pipeline-Depth=0 install ...
# install completes without any issue
Actions
Copy link
 #2

Updated by Ian Ballou about 3 years ago

Hi Rossen P,

Do you know if you're running Pulp 2 or Pulp 3 for debian content? One way of knowing is if this is a fresh Katello install for 3.18.1 or if you upgraded from an older version.

You can also check on the "Services" tab on your primary smart proxy to see if debian content is under Pulp or Pulpcore. Also, let us know if it's under Pulpcore but says it's overridden to use Pulp 2.

Actions
Copy link
 #3

Updated by Ian Ballou about 3 years ago

  • Status changed from New to Need more information
Actions
Copy link
 #4

Updated by Ian Ballou about 3 years ago

Edit: I just noticed on another bug of yours that you are on Pulp 3, so don't worry about that bit.

Can you tell us what repositories you are trying (with URLs) so we can look into reproducing the issue?

Actions
Copy link
 #5

Updated by Rossen G about 3 years ago

  • Found in Releases Katello 3.18.2 added

I believe I'm using pulp3. This one can be hard to reproduce, but I can do it reliably.
It seems that I have to first hit package downloads on the mirror in /pub/, and then hit the katello managed repo.
Only then I get the error. Works as expected when I disable pipeling in apt.

Let me know if I can collect more debug info, if you cant repro.

I updated to `3.18.2` this morning, so here is the same issue with Saltstack repo this time:

root@testvm:~# cat /etc/apt/sources.list
deb [arch=amd64] http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu/ bionic main restricted universe multiverse
deb [arch=amd64] http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu/ bionic-updates main restricted universe multiverse
deb [arch=amd64] http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu/ bionic-security main restricted universe multiverse

deb [trusted=true] http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest/ default all

root@testvm:~# apt update
Hit:1 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:3 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic-security InRelease
Ign:4 http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest default InRelease
Hit:5 http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest default Release
Ign:6 http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest default Release.gpg
Reading package lists... Done
Building dependency tree
Reading state information... Done
9 packages can be upgraded. Run 'apt list --upgradable' to see them.

root@vm-test-rpopov-ubuntu:~# apt install -y salt-minion salt-master salt-api salt-ssh salt-syndic salt-cloud
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  dctrl-tools git git-man ieee-data libcurl3-gnutls liberror-perl libnorm1 libpgm-5.2-0 libsodium23 libzmq5 lsb-release patch python3-apt python3-asn1crypto python3-certifi python3-cffi-backend python3-chardet python3-cherrypy3
  python3-croniter python3-crypto python3-cryptography python3-dateutil python3-distro python3-git python3-gitdb python3-gnupg python3-idna python3-jinja2 python3-libcloud python3-lockfile python3-markupsafe python3-msgpack
  python3-netaddr python3-openssl python3-pkg-resources python3-psutil python3-pycryptodome python3-repoze.lru python3-requests python3-routes python3-simplejson python3-six python3-smmap python3-systemd python3-tz
  python3-urllib3 python3-webob python3-yaml python3-zmq salt-common
Suggested packages:
  debtags git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk gitweb git-cvs git-mediawiki git-svn lsb diffutils-doc python3-apt-dbg python-apt-doc python-crypto-doc python-cryptography-doc
  python3-cryptography-vectors python-git-doc python-jinja2-doc python-lockfile-doc ipython3 python-netaddr-docs python-openssl-doc python3-openssl-dbg python3-setuptools python-psutil-doc python3-socks python3-paste
  python3-nose python-webob-doc python3-botocore ifupdown python3-pycurl python3-twisted python3-augeas
The following NEW packages will be installed:
  dctrl-tools git git-man ieee-data libcurl3-gnutls liberror-perl libnorm1 libpgm-5.2-0 libsodium23 libzmq5 lsb-release patch python3-apt python3-asn1crypto python3-certifi python3-cffi-backend python3-chardet python3-cherrypy3
  python3-croniter python3-crypto python3-cryptography python3-dateutil python3-distro python3-git python3-gitdb python3-gnupg python3-idna python3-jinja2 python3-libcloud python3-lockfile python3-markupsafe python3-msgpack
  python3-netaddr python3-openssl python3-pkg-resources python3-psutil python3-pycryptodome python3-repoze.lru python3-requests python3-routes python3-simplejson python3-six python3-smmap python3-systemd python3-tz
  python3-urllib3 python3-webob python3-yaml python3-zmq salt-api salt-cloud salt-common salt-master salt-minion salt-ssh salt-syndic
0 upgraded, 56 newly installed, 0 to remove and 9 not upgraded.
Need to get 16.2 MB of archives.
After this operation, 125 MB of additional disk space will be used.
Get:1 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 lsb-release all 9.20170808ubuntu1 [11.0 kB]
Get:2 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 python3-yaml amd64 3.12-1build2 [109 kB]
Get:3 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-apt amd64 1.6.5ubuntu0.5 [149 kB]
Get:4 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 dctrl-tools amd64 2.24-2build1 [60.9 kB]
Get:5 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic-updates/main amd64 libcurl3-gnutls amd64 7.58.0-2ubuntu3.12 [212 kB]
Get:6 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 liberror-perl all 0.17025-1 [22.8 kB]
Get:7 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic-updates/main amd64 git-man all 1:2.17.1-1ubuntu0.8 [804 kB]
Get:8 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic-updates/main amd64 git amd64 1:2.17.1-1ubuntu0.8 [3,916 kB]
Get:9 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 ieee-data all 20180204.1 [1,539 kB]
Get:10 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/universe amd64 libnorm1 amd64 1.5r6+dfsg1-6 [224 kB]
Get:11 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/universe amd64 libpgm-5.2-0 amd64 5.2.122~dfsg-2 [157 kB]
Get:12 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 libsodium23 amd64 1.0.16-2 [143 kB]
Get:13 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic-updates/universe amd64 libzmq5 amd64 4.2.5-1ubuntu0.2 [221 kB]
Get:14 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic-updates/main amd64 patch amd64 2.7.6-2ubuntu1.1 [102 kB]
Get:15 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB]
Get:16 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 python3-certifi all 2018.1.18-2 [144 kB]
Get:17 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB]
Get:18 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 python3-pkg-resources all 39.0.1-2 [98.8 kB]
Get:19 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 python3-chardet all 3.0.4-1 [80.3 kB]
Get:20 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 python3-six all 1.11.0-2 [11.4 kB]
Get:21 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB]
Get:22 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB]
Get:23 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB]
Get:24 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/universe amd64 python3-croniter all 0.3.12-2 [12.0 kB]
Get:25 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 python3-crypto amd64 2.6.1-8ubuntu2 [244 kB]
Get:26 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 python3-idna all 2.6-1 [32.5 kB]
Get:27 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB]
Get:28 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/universe amd64 python3-smmap all 2.0.3-1 [20.1 kB]
Get:29 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/universe amd64 python3-gitdb all 2.0.3-1 [46.6 kB]
Get:30 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/universe amd64 python3-git all 2.1.8-1 [301 kB]
Get:31 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-gnupg all 0.4.1-1ubuntu1.18.04.1 [17.1 kB]
Get:32 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB]
Get:33 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB]
Get:34 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-urllib3 all 1.22-1ubuntu0.18.04.2 [86.2 kB]
Get:35 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-requests all 2.18.4-2ubuntu0.1 [58.3 kB]
Get:36 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/universe amd64 python3-lockfile all 1:0.12.2-2 [14.7 kB]
Get:37 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB]
Get:38 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/universe amd64 python3-libcloud all 2.2.1-1 [1,221 kB]
Get:39 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 python3-msgpack amd64 0.5.6-1 [73.5 kB]
Get:40 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 python3-netaddr all 0.7.19-1 [213 kB]
Get:41 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB]
Get:42 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-psutil amd64 5.4.2-1ubuntu0.1 [136 kB]
Get:43 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pycryptodome amd64 3.4.7-1ubuntu1 [4,760 kB]
Get:44 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB]
Get:45 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB]
Get:46 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 python3-systemd amd64 234-1build1 [36.1 kB]
Get:47 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB]
Get:48 http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest default/all amd64 python3-zmq amd64 17.1.2-3ubuntu1 [254 kB]
Err:48 http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest default/all amd64 python3-zmq amd64 17.1.2-3ubuntu1
  File has unexpected size (254232 != 11196). Mirror sync in progress? [IP: 10.1.1.8 80]
  Hashes of expected file:
   - SHA256:91b1b2d62af0ca94a29bd8fe56f7c889adfac97881f81130946e623635d6891e
   - SHA1:079fff9fca7ea935b94dc41e8ab88cfad23ba92b [weak]
   - MD5Sum:ed612d722841ba1503286d49f4c0b240 [weak]
Get:49 http://foreman.example.com/pub/mirror/archive.ubuntu.com/ubuntu bionic/universe amd64 python3-distro all 1.0.1-2 [11.2 kB]
Get:50 http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest default/all all salt-common all 3002.6+ds-1 [7,097 kB]
Get:51 http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest default/all all salt-master all 3002.6+ds-1 [40.8 kB]
Get:52 http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest default/all all salt-api all 3002.6+ds-1 [13.1 kB]
Get:53 http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest default/all all salt-cloud all 3002.6+ds-1 [14.4 kB]
Get:54 http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest default/all all salt-minion all 3002.6+ds-1 [28.5 kB]
Get:55 http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest default/all all salt-ssh all 3002.6+ds-1 [14.1 kB]
Get:56 http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest default/all all salt-syndic all 3002.6+ds-1 [13.3 kB]
Fetched 23.4 MB in 0s (46.9 MB/s)
E: Failed to fetch http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest/pool/all/p/pyzmq/python3-zmq_17.1.2-3ubuntu1_amd64.deb  File has unexpected size (254232 != 11196). Mirror sync in progress? [IP: 10.1.1.8 80]
   Hashes of expected file:
    - SHA256:91b1b2d62af0ca94a29bd8fe56f7c889adfac97881f81130946e623635d6891e
    - SHA1:079fff9fca7ea935b94dc41e8ab88cfad23ba92b [weak]
    - MD5Sum:ed612d722841ba1503286d49f4c0b240 [weak]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

# On foreman server
# I've verified the failing package is correct on the file system, and in the Release file.

[root@foreman ~]# sha256sum /var/lib/pulp/media/artifact/91/b1b2d62af0ca94a29bd8fe56f7c889adfac97881f81130946e623635d6891e
91b1b2d62af0ca94a29bd8fe56f7c889adfac97881f81130946e623635d6891e  /var/lib/pulp/media/artifact/91/b1b2d62af0ca94a29bd8fe56f7c889adfac97881f81130946e623635d6891e
[root@foreman ~]# wc -c /var/lib/pulp/media/artifact/91/b1b2d62af0ca94a29bd8fe56f7c889adfac97881f81130946e623635d6891e
254232 /var/lib/pulp/media/artifact/91/b1b2d62af0ca94a29bd8fe56f7c889adfac97881f81130946e623635d6891e

[root@foreman ~]# hammer repository show --id 24
ID:                 24
Name:               ubuntu_18.04_salt_latest
Label:              ubuntu_18_04_salt_latest
Description:
Organization:       MyOrg
Red Hat Repository: no
Content Type:       deb
Mirror on Sync:     yes
URL:                https://repo.saltproject.io/py3/ubuntu/18.04/amd64/latest/
Publish Via HTTP:   yes
Published At:       http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest/
Relative Path:      MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest
HTTP Proxy:
    HTTP Proxy Policy: global_default_http_proxy
Product:
    ID:   2
    Name: Saltstack
GPG Key:
    ID:   2
    Name: Saltstack
Sync:
    Status:         Success
    Last Sync Date: about 19 hours
Created:            2021/03/12 21:54:45
Updated:            2021/03/19 23:33:38
Content Counts:

[root@foreman ~]# tail -20 /var/log/httpd/foreman_access.log
10.1.1.120 - - [24/Mar/2021:21:43:13 +0000] "GET /pub/mirror/archive.ubuntu.com/ubuntu/pool/main/p/python-msgpack/python3-msgpack_0.5.6-1_amd64.deb HTTP/1.1" 200 73544 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.2)" 
10.1.1.120 - - [24/Mar/2021:21:43:13 +0000] "GET /pub/mirror/archive.ubuntu.com/ubuntu/pool/main/p/python-netaddr/python3-netaddr_0.7.19-1_all.deb HTTP/1.1" 200 212860 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.2)" 
10.1.1.120 - - [24/Mar/2021:21:43:13 +0000] "GET /pub/mirror/archive.ubuntu.com/ubuntu/pool/main/p/pyopenssl/python3-openssl_17.5.0-1ubuntu1_all.deb HTTP/1.1" 200 41452 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.2)" 
10.1.1.120 - - [24/Mar/2021:21:43:13 +0000] "GET /pub/mirror/archive.ubuntu.com/ubuntu/pool/main/p/python-psutil/python3-psutil_5.4.2-1ubuntu0.1_amd64.deb HTTP/1.1" 200 135840 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.2)" 
10.1.1.120 - - [24/Mar/2021:21:43:13 +0000] "GET /pub/mirror/archive.ubuntu.com/ubuntu/pool/universe/p/pycryptodome/python3-pycryptodome_3.4.7-1ubuntu1_amd64.deb HTTP/1.1" 200 4760028 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.2)" 
10.1.1.120 - - [24/Mar/2021:21:43:13 +0000] "GET /pub/mirror/archive.ubuntu.com/ubuntu/pool/universe/p/python-repoze.lru/python3-repoze.lru_0.7-1_all.deb HTTP/1.1" 200 11740 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.2)" 
10.1.1.120 - - [24/Mar/2021:21:43:13 +0000] "GET /pub/mirror/archive.ubuntu.com/ubuntu/pool/universe/r/routes/python3-routes_2.4.1-1_all.deb HTTP/1.1" 200 28440 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.2)" 
10.1.1.120 - - [24/Mar/2021:21:43:13 +0000] "GET /pub/mirror/archive.ubuntu.com/ubuntu/pool/main/p/python-systemd/python3-systemd_234-1build1_amd64.deb HTTP/1.1" 200 36112 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.2)" 
10.1.1.120 - - [24/Mar/2021:21:43:13 +0000] "GET /pub/mirror/archive.ubuntu.com/ubuntu/pool/main/p/python-webob/python3-webob_1.7.3-2fakesync1_all.deb HTTP/1.1" 200 64324 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.2)" 
10.1.1.120 - - [24/Mar/2021:21:43:13 +0000] "GET /pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest/pool/all/p/pyzmq/python3-zmq_17.1.2-3ubuntu1_amd64.deb HTTP/1.1" 200 254232 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.2)" 
10.1.1.120 - - [24/Mar/2021:21:43:13 +0000] "GET /pub/mirror/archive.ubuntu.com/ubuntu/pool/universe/p/python-distro/python3-distro_1.0.1-2_all.deb HTTP/1.1" 200 11196 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.2)" 
10.1.1.120 - - [24/Mar/2021:21:43:13 +0000] "GET /pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest/pool/all/s/salt/salt-common_3002.6%2bds-1_all.deb HTTP/1.1" 200 5734400 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.2)" 
10.1.1.120 - - [24/Mar/2021:21:43:13 +0000] "GET /pub/mirror/archive.ubuntu.com/ubuntu/pool/universe/p/python-distro/python3-distro_1.0.1-2_all.deb HTTP/1.1" 200 11196 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.2)" 
10.1.1.120 - - [24/Mar/2021:21:43:13 +0000] "GET /pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest/pool/all/s/salt/salt-common_3002.6%2bds-1_all.deb HTTP/1.1" 200 7096600 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.2)" 
10.1.1.120 - - [24/Mar/2021:21:43:13 +0000] "GET /pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest/pool/all/s/salt/salt-master_3002.6%2bds-1_all.deb HTTP/1.1" 200 40776 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.2)" 
10.1.1.120 - - [24/Mar/2021:21:43:13 +0000] "GET /pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest/pool/all/s/salt/salt-api_3002.6%2bds-1_all.deb HTTP/1.1" 200 13108 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.2)" 
10.1.1.120 - - [24/Mar/2021:21:43:13 +0000] "GET /pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest/pool/all/s/salt/salt-cloud_3002.6%2bds-1_all.deb HTTP/1.1" 200 14376 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.2)" 
10.1.1.120 - - [24/Mar/2021:21:43:14 +0000] "GET /pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest/pool/all/s/salt/salt-minion_3002.6%2bds-1_all.deb HTTP/1.1" 200 28528 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.2)" 
10.1.1.120 - - [24/Mar/2021:21:43:14 +0000] "GET /pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest/pool/all/s/salt/salt-ssh_3002.6%2bds-1_all.deb HTTP/1.1" 200 14120 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.2)" 
10.1.1.120 - - [24/Mar/2021:21:43:14 +0000] "GET /pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest/pool/all/s/salt/salt-syndic_3002.6%2bds-1_all.deb HTTP/1.1" 200 13332 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.2)" 

[root@foreman ~]# grep 'python3-zmq' /var/log/messages
Mar 24 21:43:13 foreman pulpcore-content: [24/Mar/2021:21:43:13 +0000] "GET /pulp/content/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest/pool/all/p/pyzmq/python3-zmq_17.1.2-3ubuntu1_amd64.deb HTTP/1.1" 200 254454 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.2)"
Actions
Copy link
 #6

Updated by Rossen G about 3 years ago

I captured a tcpdump from the client, and it appears that client is getting the correct data in the correct order.
Apt appears to mismatching response and metadata somehow...
Explains why on the original log, all mismatches are against the same expected size of 86464.

Actions
Copy link
 #7

Updated by Rossen G about 3 years ago

  • Status changed from Need more information to New
Actions
Copy link
 #8

Updated by Rossen G about 3 years ago

Opened a bug for apt for Ubuntu: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1921626

Managed to reproduce the issue with bionic, focal, and hirsute.

Actions
Copy link
 #9

Updated by Rossen G about 3 years ago

Turns out apt relies on "Size" field from "Packages" file to match pipeline responses correctly.
The file produced by pulp doesn't have "Size" field for packages.

root@aptbug:~# curl -sS https://repo.saltproject.io/py3/ubuntu/18.04/amd64/latest/dists/bionic/main/binary-amd64/Packages | grep -c ^Size
14
root@aptbug:~# curl -sS http://foreman.example.com/pulp/deb/MyOrg/Library/custom/Saltstack/ubuntu_18_04_salt_latest/dists/default/all/binary-amd64/Packages | grep -c ^Size
0

https://salsa.debian.org/apt-team/apt/-/blob/0d25ce3d466ecddea02d171981f011f7dbf95e08/methods/basehttp.cc#L711-731

Actions
Copy link
 #10

Updated by Justin Sherrill about 3 years ago

  • Status changed from New to Rejected
  • Target version set to Katello Recycle Bin

I've opened a pulp issue for this: https://pulp.plan.io/issues/8512

i'm not sure there's anything that katello can do.

I'm gonna close this issue, but we can reopen if that turns out not to be the case

Actions
Copy link
 #11

Updated by Rossen G about 3 years ago

Thanks. I already raised it at https://pulp.plan.io/issues/8506

Actions
Copy link

Also available in: Atom PDF