This is a tricky one:

We had the problem, that REX-Jobs were successful on the host, but reported as failed, with a strange log-message:

ArgumentError (invalid %-encoding (http%3A%2F%2F...

From the Backtrace it was visible, that this happened during Rails.cache.delete_matched()

Running a foreman-rake tmp:cache:clear fixed the issue for a short time, but the error re-appeared.

I was able to reproduce it with the following code-snippet:

a = "http://thisisaverylonghostname.de/pulp/content/Organize/installation_media/installation_media/custom/installation_media/Ubuntu20_04_1//dists/focal/main/installer-amd64/current/legacy-images/netboot/ubuntu-installer/amd64/initrd.gz" 
Rails.cache.write a, 'hello'
Rails.cache.delete_matched(/hello/)

This leads to the following Error:

Traceback (most recent call last):
        2: from lib/tasks/console.rake:5:in `block in <top (required)>'
        1: from (irb):4
ArgumentError (invalid %-encoding (http%3A%2F%2Fthisisaverylonghostname.de%2Fpulp%2Fcontent%2FOrganize%2Finstallation_media%2Finstallation_media%2Fcustom%2Finstallation_media%2FUbuntu20_04_1%2F%2Fdists%2Ffocal%2Fmain%2Finstaller-amd64%2Fcurrent%2Flegacy-images%/2Fnetboot%2Fubuntu-installer%2Famd64%2Finitrd.gz))

This IMHO is a generic problem with the FileStore-cache in rails: https://github.com/rails/rails/issues/49690

But it looks like we can mitigate it by adding a trailing / to the cache-path, since then, only the second part will be decoded.