This happens on latest stable Foreman/Katello or even develop. Typical use case is when you have RHEL 7 Server and Workstation, they are both associated to RedHat 7 - this association is done by Katello.

It is possible to reproduce this with Foreman - changing installation media with two different kernel/initram images back and forth for a single OS makes smart-proxy to corrupt the files eventually.

This is a proposal to change naming scheme from os-version to something more stable and more effective (causing less download attempts or whole downloads when installation media is changed).

Ideally to solve the issue with "-c" flag in wget, the sha1sum should be calculated from contents instead of the URL. That would mean downloading it to temporary filename first, calculating it, renaming, symlinking.

Note this is an attempt to fix two issues:

• wget -c corrupting files
• Anaconda trying to load incomplete initramdisk (when network is slow)

Both should be fixed without changing proxy API contracts.

I don't see how creating per-host symlinks in the original proposal will prevent sending of not fully downloaded files. Other than that I largely agree with the assessment of the problem. The underlying problem seems to be that the combination of os-medium isn't unique. The most straightforward fix would be to introduce unique, non-mutable ids for both os and medium and use them to generate filenames for kernel files.

A better fix would involve fixing the relationship between os and medium objects: why is medium is a many-to-many with os? When is this valid?

Either way I do not think the fix belongs on smart-proxy, as this is a problem with how Foreman handles oses and media.

Dmitri Dolguikh wrote:

I don't see how creating per-host symlinks in the original proposal will prevent sending of not fully downloaded files. Other than that I largely agree with the assessment of the problem. The underlying problem seems to be that the combination of os-medium isn't unique. The most straightforward fix would be to introduce unique, non-mutable ids for both os and medium and use them to generate filenames for kernel files.

I agree, totally valid solution. I was trying to find something that does not require changes in both Foreman and Smart Proxy. I was hoping for something that does not even change the TFTP contract - we would reuse the input that Foreman sends (filename, source URL) and just make a symlink afterwards. Symlinks are indeed not solving the issue, it's basically dropping redownloading of content what does this.

A better fix would involve fixing the relationship between os and medium objects: why is medium is a many-to-many with os? When is this valid?

While I agree this is a bit of over-engineering, a valid use case would be mirrors in all your datacenters and during provisioning of a host or hostgroup you would select the closest mirror manually.

One important thing we need to keep in mind - when Katello is installed, Installation Media is not not used, Katello takes over and provides it's own source. When user select RHEL 7 Server or Workstation "flavor", installation media URL changes as well. So this is kinda many-to-many behavior of one individual OS (RHEL7). Even if we drop SQL relationship, Katello will still do many-to-many behavior.

Either way I do not think the fix belongs on smart-proxy, as this is a problem with how Foreman handles oses and media.

While I agree on refactoring Foreman part, if you look at current TFTP Smart Proxy API it is broken. You give it a filename and URL and it corrupts it every time you change the URL. You expect it to work, not to corrupt files at least. This is unexpected and very difficult error to track for some (random behaviour in the installer). I prefer fixing it, we will still have this API for legacy purposes for some time.

Now maybe we can figure out a different way, I don't insist on symlinks, this was just an idea. There was a PR to rewrite wget into pure Ruby (#3034) which was IMHO little bit of an overkill. Maybe there is something in the middle, I can imagine more simple pure Ruby implementation with correct handling of changed content (e.g. when URL changes and filename remains the same). One major problem of pure Ruby implementation I still have is that installation media can be anything, from ancient Debian mirror with super old Apache httpd without Content-Length support or even last modified headers to super-duper modern Pulp with everything in. For this reason I tend to find a solution based on wget or curl with symlinks ("temporary files").

If a many-to-many relationship between os and media/source is a requirement, then information identifying the specific os-source pair has to be provided to smart-proxy (alongside with url and filename) in order for it to retrieve correct boot files. And this information has to come from foreman, it cannot be automatically generated (think hashsum) on the proxy. I don't think smart-proxy api has to be extended for this, as os-source pair id can be encoded in file paths.

So instead of RedHat-7.3 naming (OS name, version) we would do something like OS name, version media name. The problem with this is installation media is empty when Katello is installed, we can make Katello aware of this. This will end up with encoding Organization, Environment and Content View with Product Name into the filename because it needs to be unique. So I expect this to end up with something really crazy like:

MyOrganization-Library_RHEL_ATOMIC_CCV1_Red_Hat_Enterprise_Linux_Atomic_Host_7_4-RedHat-7.4.img

or

MyOrganization-Library_RHEL_SOE_CV_Red_Hat_Enterprise_Linux_Server_7_4-RedHat-7.4.img

Without Katello it will be little bit nicer incorporating only the medium name:

Debian_Main_Mirror-Debian-9.img

One more argument why I incorporated symlinks into this is that I would like to introduce more of a "atomicity" to avoid situations when Anaconda trying to load a init RAM disk which is still being downloaded. If we download into temporary file and then rename (or symlink it), then Anaconda will fail early which will be much more easier to troubleshoot than random freeze or error. But this can be as simple as a temporary download location with rename.

We could use media ID instead of name, Katello could use Content View ID as well.

I briefly looked into Foreman core codebase and this will require decent changes and refactoring, kernel/initram methods should have been moved from OperatingSystem class to Host where it belongs.

I also thing getting the responsibility from the operating system to something host specific is what we need to move the things forward, as currently, you don't have a way where to get the medium_id or content view id and we ran out of the hacks we can do without some decent change (and yes, the original code has been around for a while so it will require some effort to put it out or all the places, including the templates.

I'm thinking about having host.pxe_files, which could actually return different implementations based on what the host needs at the moment. So there could be OperatingSystem::PxeFilesProvider, as well as Katello::ContentView::PxeFilesProvider, and every provider would be able to provide the relative paths, as well as the urls where to get the files from. From the templates or orchestration, it would be transparent enough (just using @host.pxe_files.kernel_path or @host.pxe_files.kernel_url). Thoughts?

I second using IDs (probably generated, not db ids) instead of names for os-source pair matching.