Plugins » Discovery

PXE without discovery works fine. Bare-metal without discovery works fine. Happens even with no hostgroup selected. Only occurs on Discovered hosts.

Okay, you say it causes "render failures", can you specify in more detail? Error, exception, do you have backtrace?

If provisioned host does not let you in using the password you entered, I suggest to rule out web browser password remembering option - try with a browser which has this feature turned off. We recently fixed this in core, discovery might break it.

Lukas Zapletal wrote:

Okay, you say it causes "render failures", can you specify in more detail? Error, exception, do you have backtrace?

If provisioned host does not let you in using the password you entered, I suggest to rule out web browser password remembering option - try with a browser which has this feature turned off. We recently fixed this in core, discovery might break it.

We cant even create a host, let alone attempt to access it. Even if used in Incognito, without any type of auto-fill, it doesnt matter because the password field is greyed out on all Discovered hosts:

Here is any other host that does not use Discovery:

On ALL hosts that DO NOT use Discovery, the Password field is open and editable. If we leave it blank, it uses the Global password from Settings, templates render, everything is happy. This works just fine.
On ALL hosts that DO use Discovery, the Password field is greyed out. You can edit it if you click on the pencil. And you MUST edit it, or templates will fail to render, because the password field is passed to them, then the password is encoded. The encoding fails. I can get you a render failure stacktrace, but it would be basically pointless, as that is just a side effect of the original issue. The render issue is simply an illegal character, which makes sense because the global password in settings is not passed to the Host. The original issue being, that on ALL hosts that use Discovery, the password field is greyed out and thus does not use the main Settings password.

I will file a patch shortly, test it. I found some old code around root passwords, no clue if this change helps or not in this case.

The render error is related to our templates that use powershell for windows, and xml for windows. They will both fail to render the password encoding, because something is wrong with the password field. On our system it is ONLY Discovered hosts that are doing it. If we manually click the pencil button, and manually edit the password, and type it in, it works fine. Its only when we DO NOT touch the password field. On any host that is NOT a Discovered host, the password field is editable. We just dont enter one, and the template uses the root_pass (global settings password). But for some reason (as shown in the above screenshots) Discovered Hosts password field is greyed out.

Here is an example of one of those templates:

<%#
kind: provision
name: Windows Unattend
oses:
- Windows Server 2012
- Windows Server 2012 R2
- Windows Server 2016

This template accepts the following parameters:
- windowsLicenseKey: ABCDE-ABCDE-ABCDE-ABCDE-ABCDE # Valid Windows license key
- windowsLicenseOwner: Company, INC # Legal owner of the Windows license key
%>

<?xml version="1.0" encoding="utf-8"?>
<unattend xmlns="urn:schemas-microsoft-com:unattend">
    <servicing></servicing>
    <settings pass="offlineServicing">
        <component name="Microsoft-Windows-LUA-Settings" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <EnableLUA>false</EnableLUA>
        </component>
    </settings>    
    <settings pass="oobeSystem">
        <component name="Microsoft-Windows-International-Core" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <InputLocale>en-US</InputLocale>
            <SystemLocale>en-US</SystemLocale>
            <UILanguage>en-US</UILanguage>
            <UILanguageFallback>en-US</UILanguageFallback>
            <UserLocale>en-US</UserLocale>
        </component>
        <component name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <UserAccounts>
                <AdministratorPassword>
                    <Value><%= Base64.encode64(Encoding::Converter.new("UTF-8", "UTF-16LE",:undef => nil).convert(Base64.decode64(root_pass)+"AdministratorPassword")).delete!("\n").chomp %></Value>
                    <PlainText>false</PlainText>
                </AdministratorPassword>                
            </UserAccounts>

            <TimeZone>Central Standard Time</TimeZone>
            <% if @host.params['windowsLicenseOwner'] -%>
                <RegisteredOrganization><%= @host.params['windowsLicenseOwner'] %></RegisteredOrganization>
                <RegisteredOwner><%= @host.params['windowsLicenseOwner'] %></RegisteredOwner>
            <% end -%>
            <OOBE>
                <HideEULAPage>true</HideEULAPage>
                <NetworkLocation>Work</NetworkLocation>
                <ProtectYourPC>1</ProtectYourPC>
                <SkipUserOOBE>true</SkipUserOOBE>
                <SkipMachineOOBE>true</SkipMachineOOBE>
            </OOBE>
            <ShowWindowsLive>false</ShowWindowsLive>            
            <% if @host.params['windowsLicenseKey'] -%>
                <ProductKey><%= @host.params['windowsLicenseKey'].to_s %></ProductKey>
            <% end -%>  
        </component>
    </settings>
    <settings pass="specialize">
        <component name="Microsoft-Windows-International-Core" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <InputLocale>en-US</InputLocale>
            <SystemLocale>en-US</SystemLocale>
            <UILanguage>en-US</UILanguage>
            <UILanguageFallback>en-US</UILanguageFallback>
            <UserLocale>en-US</UserLocale>
        </component>
        <component name="Microsoft-Windows-IE-ESC" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <IEHardenAdmin>false</IEHardenAdmin>
        </component>
        <component name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <ComputerName><%= @host.shortname %></ComputerName>
            <% if @host.params['windowsLicenseOwner'] -%>
                <RegisteredOrganization><%= @host.params['windowsLicenseOwner'] %></RegisteredOrganization>
                <RegisteredOwner><%= @host.params['windowsLicenseOwner'] %></RegisteredOwner>
            <% end -%>
            <TimeZone>Central Standard Time</TimeZone>
        </component>
        <component name="Networking-MPSSVC-Svc" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <DomainProfile_EnableFirewall>false</DomainProfile_EnableFirewall>
            <PrivateProfile_EnableFirewall>false</PrivateProfile_EnableFirewall>
            <PublicProfile_EnableFirewall>false</PublicProfile_EnableFirewall>
        </component>
        <component name="Microsoft-Windows-ServerManager-SvrMgrNc" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <DoNotOpenServerManagerAtLogon>true</DoNotOpenServerManagerAtLogon>
        </component>
        <component name="Microsoft-Windows-TerminalServices-LocalSessionManager" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <fDenyTSConnections>false</fDenyTSConnections>
        </component>
        <component name="Microsoft-Windows-TerminalServices-RDP-WinStationExtensions" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <SecurityLayer>1</SecurityLayer>
            <UserAuthentication>0</UserAuthentication>
        </component>
    </settings>
    <settings pass="offlineServicing">
        <component name="Microsoft-Windows-PnpCustomizationsNonWinPE" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <DriverPaths>
                <!--<PathAndCredentials wcm:action="add" wcm:keyValue="1">

                    <Path>c:\drivers</Path>
                </PathAndCredentials>-->
            </DriverPaths>
        </component>
    </settings>
</unattend>

You can see the section:

            <UserAccounts>
                <AdministratorPassword>
                    <Value><%= Base64.encode64(Encoding::Converter.new("UTF-8", "UTF-16LE",:undef => nil).convert(Base64.decode64(root_pass)+"AdministratorPassword")).delete!("\n").chomp %></Value>
                    <PlainText>false</PlainText>
                </AdministratorPassword>                
            </UserAccounts>

Which calls on

(Base64.decode64(root_pass)+"AdministratorPassword")

But, because ALL Discovered hosts (in our environment) have an issue with the password field, it fails to render the template. ANY and ALL other hosts that do not use discovery, work fine. Again, it is only Discovered hosts.

Is there a way for us to take a Discovered Host, and submit the provisioning, and then somehow see what its setting in the password field? Maybe we can try to figure out what the password field is doing on Discovered hosts vs. any other host.

I attempted the change on 3 different hosts. The issue still resides across all 3.
Previously, the password field would be editable (not greyed out). Since this issue occurred, I have noticed the password field is always greyed out on Discovered hosts. If that helps anything.

I marked this as High, because it would be very easy for anyone that is deploying Windows, ESX or XEN to hit this error. It is not a highly customized template. It is a basic windows XML template that anyone deploying windows will use. The decoding is also used in ESX and XEN templates, so it is not that rare. I apologize for using capital letters. It is very common in my business and culture to use capitalized letters in order to enhance readability and to exaggerate understandability of important information, on complicated technical tickets. It was only done to help point out importance and not to yell or shout at you. I also apologize that my company requires passwords not be in plaintext, and that I am not one of the standard 9/10 Foreman users.

Thank you for your help on this. I greatly appreciate it and am sincerely apologetic for previous communications.

Can you provision a host (which fails) and then do foreman-rake console:

> Setting[:root_pass]
=> "$1$aOdKUP5S$TxOzhcCOQkl/Vc7S1r8rQ0" 

> Hostgroup.all.pluck(:name, :root_pass)
=> [["CentOS 7.0", "$5$2EbzMzmA$rMRyr/7c6kSYJt3E1CT0NljQPsAUPhfJQTA6RdLBIzA"],
 ["CentOS 7.1", "$5$QdijIbxm$WMpHq4t0d5ymrpe15cXq6IlHD3U2aj9AFpJxYa2U811"],
 ["CentOS 7.2", "$5$sPVMbuOD$AefEQ1TLVZ9.xn9lK/2KUmPXJjxqTu8m7NvVqP24MZ0"],
 ["CentOS 7.3", "$5$b4gtVVC7$GnEnB251iIqNtR2QqHChm8g38nFnFkz1HmtidB.VQ/3"],
 ["RHEL 7.0", "$5$PNaoXOuk$HdZSqzz9g.UnvHYtOQ2VY6XbWgNAtQ4el391MYFAkMD"],
 ["RHEL 7.1", "$5$9R+XKnJH$3fuiA5ZvwWvsxsOwv2sg6qrngRh/WTlFaXGkkHMYyM5"],
 ["RHEL 7.2", "$5$mKLaqDC0$RXyYlwoYyxKdOfodEIeRvVvq99s0FzhGv69EWxfabe7"],
 ["RHEL 7.3", "$5$1gOeJeBi$Ue60Tm5XIDYtHM2ZY/m/tG0VPFi5OUn9JglLK3qZz7."]]

> Host.all.pluck(:name, :root_pass)
=> [["joan-jacoby.home.lan", "$5$b4gtVVC7$GnEnB251iIqNtR2QqHChm8g38nFnFkz1HmtidB.VQ/3"],
 ["carl-magpusao.home.lan", "$5$2EbzMzmA$rMRyr/7c6kSYJt3E1CT0NljQPsAUPhfJQTA6RdLBIzA"]]

Feel free to edit the output and only show relevant host/hostgroup, also replace real passwords with some strings in a consistent way so I can compare which are the same. But it would be much better if you can test this on an instance with some test passwords you can paste unmodified so it's absolutely clear what ends up in the database.

Please repeat before and after applying the patch, so we can compare if it had any effect on how root password is stored.

Here is one workaround you can use - instead using root password (which has some code logic in Foreman core built-in - the format of root password is assumed to follow Linux PAM format), you can simply store a dummy unused password there and then you can use host/hostgroup parapeter to store your real password and use that in your templates.

Thanks Lukas. Here you are. Its interesting, the data doesnt change. In the console, we can see the password is indeed getting set correctly. I did not include hostgroups because for the moment we are provisioning these all without a hostgroup, to take away that possibility. So, if the output is the same, maybe the disconnect is somewhere on how the templates read that field? Visually, the only difference I see in the GUI, is that on Discovered hosts, the password field is greyed out. On any other host it is not greyed out and is editable.

Before patch:

[*PROD* root@puppet 1 /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_discovery-10.0.0/app/models/host]# foreman-rake console
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rdoc-4.2.0/lib/rdoc/task.rb:326: warning: already initialized constant Rake::RDocTask
/opt/rh/rh-ruby22/root/usr/share/ruby/rdoc/task.rb:326: warning: previous definition of RDocTask was here
Loading production environment (Rails 4.2.5.1)
Failed to load console gems, starting anyway
irb(main):001:0>
irb(main):002:0*
irb(main):003:0* Setting[:root_pass]
=> "GlobalPassword" 
irb(main):004:0>
irb(main):005:0*
irb(main):006:0* Host.all.pluck(:name, :root_pass)
=> ["cmlb8ca3a6e8a50.lab.beer.town", "GlobalPassword"]]
irb(main):007:0>

After patch:

[*PROD* root@puppet 0 ~]# foreman-rake console
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rdoc-4.2.0/lib/rdoc/task.rb:326: warning: already initialized constant Rake::RDocTask
/opt/rh/rh-ruby22/root/usr/share/ruby/rdoc/task.rb:326: warning: previous definition of RDocTask was here
Loading production environment (Rails 4.2.5.1)
Failed to load console gems, starting anyway
irb(main):001:0>
irb(main):002:0*
irb(main):003:0* Setting[:root_pass]
=> "GlobalPassword" 
irb(main):004:0>
irb(main):005:0*
irb(main):006:0* Host.all.pluck(:name, :root_pass)
=> ["cmlb8ca3a6e8a50.lab.beer.town", "GlobalPassword"]

Thanks for the update Lukas. Interesting find. Can I ask why this only fails for discovered hosts? Is it Foreman Discovery that assumes Linux provisioning? Because the issue does not occur on virtual machines or bare-metal hosts that do not use Discovery.

It fails with discovery IMHO because of bug #22581 - discovery currently does not set password from hostgroup and automatically falls back to global root password. You haven't confirmed tho, but that's my only explanation.

Anyway, can you remove the base64 decoding and try to provision a host with a template which prints the password "as is". Can you tell which password is used? Is it the one you put into Discovery form, Hostgroup or the global password? Try again with and without the patch I created.

Anyway, I suggest to avoid using root password, until we do the RFE I filed (disable Linux encryption for root passwords). That's the only way you can get it working IMHO.

Thanks again Lukas. Yes, I figured that as a work around I will probably create a hostgroup called "Discovery Deployment" and add the password to the host group. Or else use a global parameter and pass it.

So, I changed the decode lines in the templates to use the plaintext password. I am able to see in the template that the Global Password is showing correctly!!
I am able to render all the templates (although they render correctly, the windows xml will fail during installation, as it requires the password to be encoded, but I've got work arounds for that..

I did not try with and without the patch, as I don't believe that will have any effect on it, now that we are able to prove that it is indeed getting the root_pass. I am happy to wait for the RFE for the disabling of Linux encryption.

If you really need me to test the patch, I can, however it will be a few days.

Let me know if I can do anything else to help. Really appreciate the assistance.

Any idea if this will ever get resolved? Its still prevalent in 1.19. Cuases lots of headaches on Windows hosts. Many of my weekly failures are due to this specific issue.

Hello, as I've said above, you cannot use root_pass for non-UNIX OS, just use some custom parameter named "windows_pass" or whatever. We encrypt it with UNIX crypt functions and this will not work for Windows.

Ah ok. Sorry, I thought based on the comment below that it was in the works to be resolved at some point.

Anyway, I suggest to avoid using root password, until we do the RFE I filed (disable Linux encryption for root passwords). That's the only way you can get it working IMHO.

I'll use a Global Param, although it will be available for all to see I suppose.