Bug #7524
closed
Adding libvirt compute resource ersults in error
Description
Adding a new compute libvirt resource results in an error if testing the connection to the hypervisor:
Warning!
Error making a connection to libvirt URI qemu+ssh://root@coshv1.localdomain/system: Call to virConnectOpen failed: Cannot find 'ssh' in path: No such file or directory
Full trace
Fog::Errors::Error
Error making a connection to libvirt URI qemu+ssh://root@coshv1.localdomain/system: Call to virConnectOpen failed: Cannot find 'ssh' in path: No such file or directory
app/models/compute_resources/foreman/model/libvirt.rb:147:in `client'
app/models/compute_resource.rb:133:in `vms'
app/controllers/compute_resources_controller.rb:47:in `associate'
app/models/concerns/foreman/thread_session.rb:33:in `clear_thread'
lib/middleware/catch_json_parse_errors.rb:9:in `call'
The ssh key exchange with the hypervisor works fine for the foreman user from the command line:
[root@cosdpl1 ~]su foreman -s /bin/bash
bash-4.2$ ssh root@coshv1.localdomain
Last login: Thu Sep 18 11:11:23 2014 from 141.73.53.249
[root@coshv1 ~]# exit
logout
Connection to coshv1.localdomain closed.
bash-4.2$ virsh -c qemu+ssh://root@coshv1.localdomain/system
Welcome to virsh, the virtualization interactive terminal.
Type: 'help' for help with commands
'quit' to quit
virsh # exit
bash-4.2$ exit
exit
[root@cosdpl1 ~]#
If the compute resource will be created anyway, ignoring the connection error, there is no interaction possible with is compute resource.
For example, pushing the "Assosiate VMs" button results in the same error.
This is an extract of the /var/logs/foreman/prdoduction.log in debug mode:
Operation FAILED: Error making a connection to libvirt URI qemu+ssh://root@coshv1.localdomain/system:
Call to virConnectOpen failed: Cannot find 'ssh' in path: No such file or directory
/opt/rh/ruby193/root/usr/share/gems/gems/fog-1.23.0/lib/fog/libvirt/compute.rb:105:in `rescue in initialize'
/opt/rh/ruby193/root/usr/share/gems/gems/fog-1.23.0/lib/fog/libvirt/compute.rb:90:in `initialize'
/opt/rh/ruby193/root/usr/share/gems/gems/fog-core-1.23.0/lib/fog/core/service.rb:115:in `new'
/opt/rh/ruby193/root/usr/share/gems/gems/fog-core-1.23.0/lib/fog/core/service.rb:115:in `new'
/opt/rh/ruby193/root/usr/share/gems/gems/fog-core-1.23.0/lib/fog/compute.rb:58:in `new'
/usr/share/foreman/app/models/compute_resources/foreman/model/libvirt.rb:147:in `client'
/usr/share/foreman/app/models/compute_resource.rb:133:in `vms'
/usr/share/foreman/app/controllers/compute_resources_controller.rb:47:in `associate'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:167:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rendering.rb:10:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:18:in `block in process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:549:in `block (3 levels) in run_733772365948061383__process_action__1382470231606620647__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in conditional_callback_around_6836'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:326:in `around'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:310:in `_callback_around_1817'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_6836'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:537:in `block (2 levels) in _run_733772365948061383__process_action__1382470231606620647__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in conditional_callback_around_6835'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:33:in `clear_thread'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_6835'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:426:in `block in _run_733772365948061383__process_action__1382470231606620647__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in conditional_callback_around_6834'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:326:in `around'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:310:in `_callback_around_13'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_6834'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:414:in `_run_733772365948061383__process_action__1382470231606620647__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:17:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rescue.rb:29:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `block in instrument'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `instrument'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:29:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/params_wrapper.rb:207:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:121:in `process'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:45:in `process'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:203:in `dispatch'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:246:in `block in action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `dispatch'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:36:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call'
/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:600:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/apipie-rails-0.2.5/lib/apipie/extractor/recorder.rb:97:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/apipie-rails-0.2.5/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/etag.rb:23:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/conditionalget.rb:35:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/head.rb:14:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/params_parser.rb:21:in `call'
/usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:9:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/flash.rb:242:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:205:in `context'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:200:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/cookies.rb:339:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/query_cache.rb:64:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:473:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `_run__1288008719218470782__call__4330123796480538868__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:26:in `call_app'
/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:16:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/request_id.rb:22:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/methodoverride.rb:21:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/runtime.rb:17:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/lock.rb:15:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:143:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:155:in `invalidate'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:71:in `call!'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/engine.rb:479:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/application.rb:223:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/railtie/configurable.rb:30:in `method_missing'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/builder.rb:134:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/urlmap.rb:64:in `block in call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/urlmap.rb:49:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/urlmap.rb:49:in `call'
/usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/rack/thread_handler_extension.rb:77:in `process_request'
/usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:140:in `accept_and_process_next_request'
/usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:108:in `main_loop'
/usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler.rb:441:in `block (3 levels) in start_threads'
Rendered common/500.html.erb within layouts/application (3.2ms)
Rendered layouts/base.html.erb (1.2ms)
Completed 500 Internal Server Error in 16ms (Views: 5.6ms | ActiveRecord: 1.0ms)
Updated by Dirk Mayer over 9 years ago
Operating systems used:
Foreman: RHEL 7
KVM Hypervisor: RHEL 7
Updated by Lukas Zapletal over 9 years ago
- Status changed from New to Need more information
Hello,
can you check if your Foreman instance has ssh client installed and Foreman user can execute it?
Also, the libvirt server must have sshd installed, enabled, firewall set properly.
Updated by Dirk Mayer over 9 years ago
hello,
here is the requested output of the path variable:
Foreman instance:
[root@cosdpl1 ~]# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
[root@cosdpl1 ~]# ssh root@coshv1.localdomain
Last login: Thu Sep 18 11:14:57 2014 from 141.73.53.249
Hypervisor instance:
[root@coshv1 ~]# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
[root@coshv1 ~]# exit
logout
Connection to coshv1.localdomain closed.
Updated by Dirk Mayer over 9 years ago
I I found out that selinux is part of the issue. if i disable selinux with "setenforce 0" on he foreman server, it is possible to add a compute resource and the ssh key authentification out of foremen works. I suppose it has to do something with selinux permission flags on the "/usr/share/foreman" and/or "/usr/share/foreman/.ssh" folder.
How do the correct permissions have to look like ? do I have to apply the "home_root_t", "user_home_dir_t" and "ssh_home_t" flags recursively to the "/usr/share/foreman" folder ?
Updated by Dominic Cleal over 9 years ago
- Project changed from Foreman to SELinux
- Category changed from Compute resources - libvirt to Compute resources
- Status changed from Need more information to New
Updated by Lukas Zapletal over 9 years ago
Hey,
can you confirm this fixes the issue:
chcon -R system_u:object_r:ssh_home_t:s0 /usr/share/foreman/.ssh
Updated by Dirk Mayer over 9 years ago
Hey,
no this does not fix the issue ! Maybe the "user_home_dir_t" flag of the parent directory is missing ?
Updated by Lukas Zapletal over 9 years ago
Would you mind sharing the denial you get after you executed the above command and set Enforcing again?
I doubt context of the parent directory matters... Thank you very much for your help with this!
Updated by Dirk Mayer over 9 years ago
Hello,
I did some testing and I became aware of the following fact:
Changing the permission with "chcon -R system_u:object_r:ssh_home_t:s0 /usr/share/foreman/.ssh" solves the problem as long as I change the selinunx policy via commandline and without rebooting.
But as soon as I enforce selinux in the config file and perform a reboot after that, the error occurs again after the reboot although the selinux folder permissions are applied:
Output from command line after clean reboot:
[root@cosdpl1 ~]# getenforce
Enforcing
[root@cosdpl1 ~]# cat /etc/selinux/config
- This file controls the state of SELinux on the system.
- SELINUX= can take one of these three values:
- enforcing - SELinux security policy is enforced.
- permissive - SELinux prints warnings instead of enforcing.
- disabled - No SELinux policy is loaded.
#SELINUX=Permissive
SELINUX=enforcing - SELINUXTYPE= can take one of these two values:
- targeted - Targeted processes are protected,
- minimum - Modification of targeted policy. Only selected processes are protected.
- mls - Multi Level Security protection.
SELINUXTYPE=targeted
[root@cosdpl1 ~]# ls alZ /usr/share/foreman/.ssh/. foreman foreman system_u:object_r:ssh_home_t:s0 id_rsa
drwxr-xr-x. foreman foreman system_u:object_r:ssh_home_t:s0 .
drwxr-xr-x. root root system_u:object_r:usr_t:s0 ..
-rw------rw-r--r-. foreman foreman system_u:object_r:ssh_home_t:s0 id_rsa.pubrw-r--r-. foreman foreman system_u:object_r:ssh_home_t:s0 known_hosts
Error message in foreman after e.g. pushing the "Associate VMs" button of a compute resource:
Warning!
Error making a connection to libvirt URI qemu+ssh://root@coshv1.localdomain/system: Call to virConnectOpen failed: Cannot find 'ssh' in path: No such file or directory
Fog::Errors::Error
Error making a connection to libvirt URI qemu+ssh://root@coshv1.localdomain/system: Call to virConnectOpen failed: Cannot find 'ssh' in path: No such file or directory
app/models/compute_resources/foreman/model/libvirt.rb:147:in `client'
app/models/compute_resource.rb:133:in `vms'
app/controllers/compute_resources_controller.rb:47:in `associate'
app/models/concerns/foreman/thread_session.rb:33:in `clear_thread'
lib/middleware/catch_json_parse_errors.rb:9:in `call'
I hope this helps analyzing the problem!
Updated by Lukas Zapletal over 9 years ago
- Status changed from New to Duplicate
Ok this is dupe: http://projects.theforeman.org/issues/7719
I will have the patch this week. Thanks for the report!
Updated by Lukas Zapletal over 9 years ago
- Has duplicate Bug #7719: Selinux prevents console from starting/connecting added