Plugins » Katello

Due to a programming error, while testing upload for debian-packages into pulpcore debian repositories, I had the problem, that the first action (Actions::Pulp3::Repository::UploadFile) succeeded, but the next action (Actions::Pulp3::Repository::SaveArtifact) failed and therefore was skipped.

This left me with the situation that after fixing the code-error in the 2nd step, I got failures in the first step, if I tried to upload the same package again.
This then gave me a Checksum-Error from pulp: {'non_field_errors': [ErrorDetail(string='sha512 checksum must be unique.', code='unique')]}

I suspect that pulp still knows about the package from when I successfully uploaded it in the first try and therefore prevents me from re-uploading it a second time.

So I would expect: if the UploadFile-action succeeds and the SaveArtifact does not, the Task should rather be paused, so one can check why it did not work and try again, if it was a temporary failure. This should work, because the pulpcore task-href put out by UploadFile is used as input for the next action and therefore should be reusable.

Dynflow-console output for the 1st try (failure during SaveArtifact):¶

4: Actions::Pulp3::Repository::UploadFile (success) [ 1.84s / 0.84s ]¶

Queue: default

Started at: 2020-10-28 20:36:09 UTC

Ended at: 2020-10-28 20:36:11 UTC

Real time: 1.84s

Execution time (excluding suspended state): 0.84s

Input:

---
repository_id: 4
smart_proxy_id: 1
file: "/home/vagrant/foreman/tmp/uploads/RackMultipart20201028-15806-1xk3a17.deb" 
remote_user: admin
remote_cp_user: admin
locale: de
current_request_id: 
current_timezone: Europe/Berlin
current_user_id: 4
current_organization_id: 1
current_location_id: 2

Output:

---
pulp_tasks:
- pulp_href: "/pulp/api/v3/tasks/f127be17-16c5-46ad-9622-b4ce58ec9ee2/" 
  pulp_created: '2020-10-28T20:36:09.980+00:00'
  state: completed
  name: pulpcore.app.tasks.upload.commit
  started_at: '2020-10-28T20:36:10.507+00:00'
  finished_at: '2020-10-28T20:36:10.870+00:00'
  worker: "/pulp/api/v3/workers/a83fe795-c780-419e-bff7-79a452696fc6/" 
  child_tasks: []
  progress_reports: []
  created_resources:
  - "/pulp/api/v3/artifacts/e270a837-58d2-4678-ae92-4a5777dcc512/" 
  reserved_resources_record:
  - "/pulp/api/v3/uploads/fc1c6ac6-b587-4932-be45-42b73adba8f6/" 
task_groups: []
poll_attempts:
  total: 2
  failed: 0

7: Actions::Pulp3::Repository::SaveArtifact (skipped) [ 0.13s / 0.06s ]¶

Queue: default

Started at: 2020-10-28 20:36:11 UTC

Ended at: 2020-10-28 20:36:11 UTC

Real time: 0.13s

Execution time (excluding suspended state): 0.06s

Input:

---
repository_id: 4
smart_proxy_id: 1
tasks:
- pulp_href: "/pulp/api/v3/tasks/f127be17-16c5-46ad-9622-b4ce58ec9ee2/" 
  pulp_created: '2020-10-28T20:36:09.980+00:00'
  state: completed
  name: pulpcore.app.tasks.upload.commit
  started_at: '2020-10-28T20:36:10.507+00:00'
  finished_at: '2020-10-28T20:36:10.870+00:00'
  worker: "/pulp/api/v3/workers/a83fe795-c780-419e-bff7-79a452696fc6/" 
  child_tasks: []
  progress_reports: []
  created_resources:
  - "/pulp/api/v3/artifacts/e270a837-58d2-4678-ae92-4a5777dcc512/" 
  reserved_resources_record:
  - "/pulp/api/v3/uploads/fc1c6ac6-b587-4932-be45-42b73adba8f6/" 
unit_type_id: deb
options:
  file_name: frozen-bubble_2.212-9+b1_amd64.deb
remote_user: admin
remote_cp_user: admin
locale: de
current_request_id: 
current_timezone: Europe/Berlin
current_user_id: 4
current_organization_id: 1
current_location_id: 2

Output:

--- {}

Error:

[...]

Dynflow-console output for the 2nd try (Failure in UploadFile):¶

4: Actions::Pulp3::Repository::UploadFile (skipped) [ 0.99s / 0.39s ]¶

Queue: default

Started at: 2020-10-28 20:54:38 UTC

Ended at: 2020-10-28 20:54:39 UTC

Real time: 0.99s

Execution time (excluding suspended state): 0.39s

Input:

---
repository_id: 4
smart_proxy_id: 1
file: "/home/vagrant/foreman/tmp/uploads/RackMultipart20201028-15807-h94cwb.deb" 
remote_user: admin
remote_cp_user: admin
locale: de
current_request_id: 
current_timezone: Europe/Berlin
current_user_id: 4
current_organization_id: 1
current_location_id: 2

Output:

---
pulp_tasks:
- pulp_href: "/pulp/api/v3/tasks/390c3dfc-e302-4bc6-a6c8-64c15a467c46/" 
  pulp_created: '2020-10-28T20:54:38.784+00:00'
  state: failed
  name: pulpcore.app.tasks.upload.commit
  started_at: '2020-10-28T20:54:39.026+00:00'
  finished_at: '2020-10-28T20:54:39.106+00:00'
  error:
    traceback: |2
        File "/usr/lib/python3.6/site-packages/rq/worker.py", line 883, in perform_job
          rv = job.perform()
        File "/usr/lib/python3.6/site-packages/rq/job.py", line 657, in perform
          self._result = self._execute()
        File "/usr/lib/python3.6/site-packages/rq/job.py", line 663, in _execute
          return self.func(*self.args, **self.kwargs)
        File "/usr/local/lib/python3.6/site-packages/pulpcore/app/tasks/upload.py", line 28, in commit
          serializer.is_valid(raise_exception=True)
        File "/usr/lib/python3.6/site-packages/rest_framework/serializers.py", line 243, in is_valid
          raise ValidationError(self.errors)
    description: "{'non_field_errors': [ErrorDetail(string='sha256 checksum must be
      unique.', code='unique')]}" 
  worker: "/pulp/api/v3/workers/9776f49b-3bee-4456-b532-bcc6b3ed1309/" 
  child_tasks: []
  progress_reports: []
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/uploads/a355814e-9351-4eb6-8d45-1522871760d4/" 
task_groups: []
poll_attempts:
  total: 1
  failed: 1

Error:

Katello::Errors::Pulp3Error

{'non_field_errors': [ErrorDetail(string='sha256 checksum must be unique.', code='unique')]}