Project

General

Profile

Bug #3541

ec2 provisioning failure in setSSHProvision (when no CA available?)

Added by David Schmitt over 5 years ago. Updated 11 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Compute resources
Target version:
Difficulty:
Triaged:
Bugzilla link:
Team Backlog:
Fixed in Releases:
Found in Releases:

Description

Hi,

I've a fresh 1.3 foreman connected to aws. Deploying new machines works flawlessly, except for this failure:

Rolling back due to a problem: [Configure instance c06-test3.example.com via SSH   2003    failed  [#<Host::Managed id: 7, name: "c06-test3.example.com", ip: "54.211.196.93", last_compile: nil, last_freshcheck: nil, last_report: nil, updated_at: "2013-10-29 14:19:45", source_file_id: nil, created_at: "2013-10-29 14:19:45", mac: nil, root_pass: nil, serial: nil, puppet_status: 0, domain_id: 1, architecture_id: 1, operatingsystem_id: 1, environment_id: 11, subnet_id: nil, ptable_id: 1, medium_id: nil, build: true, comment: "", disk: "", installed_at: nil, model_id: nil, hostgroup_id: nil, owner_id: nil, owner_type: nil, enabled: true, puppet_ca_proxy_id: nil, managed: true, use_image: nil, image_file: nil, uuid: "i-953e16f0", compute_resource_id: 1, puppet_proxy_id: 1, certname: nil, image_id: 1, organization_id: nil, location_id: 1, type: "Host::Managed">, :setSSHProvision]]
ActiveRecord::Rollback

The finish script has already run successfully (checked by manually connecting to the new instance). setSSHProvision seems to fail on the "respond_to?(:initialize_puppetca,true) && initialize_puppetca && delAutosign if puppetca?" step. This system has no CA proxy configured and the test image/script has never tried to connect to the puppetmaster.

There were multiple factors which combined into making this an ugly issue to debug, I've no idea how easy those are to fix or which of those should be fixed:

  • no default finish template for EC2: this would have helped establishing the fact that a successful puppet run is required to finish the provisioning
  • output of the finish script not displayed: echoing the output somewhere on the foreman side would have easily established the success of the script without having to connect to the instance (which required extracting the key from the database)
    • related: unclear output from the ssh provisioner on default log levels. Has the following succeeded?
      SSH connection established to 54.205.211.252 - executing template
      negotiating protocol version
      got KEXINIT from server
      sending KEXINIT
      negotiating algorithms
      could not connect to ssh-agent
      channel_open_confirmation: 0 0 0 32768
      sending channel request "exec" 
      channel_window_adjust: 0 +2097152
      channel_success: 0
      channel_data: 0 1b
      channel_data: 0 1b
      channel_data: 0 1b
      channel_eof: 0
      channel_request: 0 exit-status false
      channel_close: 0
      closing remaining channels (0 open)
      

      (yes, it has, but it doesn't say so)
  • no error from
    respond_to?(:initialize_puppetca,true) && initialize_puppetca && delAutosign if puppetca?
    in setSSHProvision: this can fail the provisioning, but neither "respond_to?(:initialize_puppetca,true) == false" nor "not puppetca?" would report any specific error.
  • The "Rolling back due to a problem" message does not point to the source location actually causing the error: this would have helped with debugging * related: "Configure instance .* failed" only matches .mo and .js files, also not helping in locating the failing code: indicating the separate fields of this message ("[Configure instance c06-test3.example.com via SSH] [prio:2003] failed") would have helped with debugging

I've added a simple "true" after the initialize_puppetca line and thus was able to successfully provision to EC2.


Related issues

Related to Foreman - Bug #2693: unattended installation without Puppet CA is failingClosed2013-06-21
Related to Foreman - Bug #3222: Disabling UUID certificates leaves UUID certname in place on newly created hostsClosed2013-10-08
Related to Foreman - Bug #13770: SSH provisioning fails when no puppetca is assigned to the hostDuplicate2016-02-17
Related to Foreman - Tracker #14002: Orchestration build around ActiveRecord hooks trackerNew2016-03-02

Related to Foreman - Bug #14004: After_commit failure doesn't handle the rollback for the actions performed from the pre_commit phase and active record objectsNew2016-03-02
Has duplicate Foreman - Bug #9414: The SSH config step of provisioning assumes a Puppet CA proxy is specified, fails otherwiseDuplicate2015-02-17

Associated revisions

Revision a4e1e08c (diff)
Added by Ivan Necas over 2 years ago

Fixes #3541,#13769 - return true on success of ssh orchestration

Strange things start happening otherwise.

Revision 20134e11 (diff)
Added by Ivan Necas over 2 years ago

Fixes #3541,#13769 - return true on success of ssh orchestration

Strange things start happening otherwise.

(cherry picked from commit a4e1e08ceb388e9749fceee8ae4487ab3c76c105)

History

#1 Updated by Dominic Cleal over 5 years ago

  • Category set to Compute resources

#2 Updated by Dominic Cleal over 5 years ago

Just as a note, I think this is the same problem as #2693 but I didn't fix it in the SSH part as I didn't realise the same code existed. We should review and combine these, as #3222 probably isn't addressed for SSH provisioning either.

#3 Updated by Dominic Cleal over 5 years ago

  • Related to Bug #2693: unattended installation without Puppet CA is failing added

#4 Updated by Dominic Cleal over 5 years ago

  • Related to Bug #3222: Disabling UUID certificates leaves UUID certname in place on newly created hosts added

#5 Updated by Dominic Cleal over 3 years ago

  • Related to Bug #13770: SSH provisioning fails when no puppetca is assigned to the host added

#6 Updated by Dominic Cleal over 3 years ago

  • Has duplicate Bug #9414: The SSH config step of provisioning assumes a Puppet CA proxy is specified, fails otherwise added

#7 Updated by Ivan Necas over 3 years ago

  • Related to Tracker #14002: Orchestration build around ActiveRecord hooks tracker added

#8 Updated by Ivan Necas over 3 years ago

  • Related to Bug #14004: After_commit failure doesn't handle the rollback for the actions performed from the pre_commit phase and active record objects added

#9 Updated by The Foreman Bot over 3 years ago

  • Status changed from New to Ready For Testing
  • Assignee set to Ivan Necas
  • Pull request https://github.com/theforeman/foreman/pull/3269 added

#10 Updated by Dominic Cleal over 2 years ago

  • Status changed from Ready For Testing to New
  • Assignee deleted (Ivan Necas)
  • Pull request deleted (https://github.com/theforeman/foreman/pull/3269)

Issue likely still present, but PR closed due to inactivity.

#11 Updated by The Foreman Bot over 2 years ago

  • Status changed from New to Ready For Testing
  • Assignee set to Ivan Necas
  • Pull request https://github.com/theforeman/foreman/pull/4262 added

#12 Updated by Ivan Necas over 2 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100

#13 Updated by Dominic Cleal over 2 years ago

  • Legacy Backlogs Release (now unused) set to 221

Also available in: Atom PDF