Opensack Issue & Solution: "HypervisorUnavailable: Connection to the hypervisor is broken on host"

Openstack Juno

Controller OS: Redhat 6.5 

Computer node: 1 Redhat 6.5 + 4 Ubuntu14.04

Issue: 

nova-compute can’t start,  and "HypervisorUnavailable: Connection to the hypervisor is broken on host: quasarsdnve1.ibm.com" will be observed. 

2015-08-22 03:08:44.803 37239 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
2015-08-22 03:08:44.820 37239 INFO nova.virt.driver [-] Loading compute driver 'libvirt.LibvirtDriver'
2015-08-22 03:08:44.840 37239 INFO oslo.messaging._drivers.impl_rabbit [req-4152b72a-6be8-4e8e-8764-60e46a992324 ] Connecting to AMQP server on 10.140.33.216:5671
2015-08-22 03:08:44.858 37239 INFO oslo.messaging._drivers.impl_rabbit [req-4152b72a-6be8-4e8e-8764-60e46a992324 ] Connected to AMQP server on 10.140.33.216:5671
2015-08-22 03:08:44.862 37239 INFO oslo.messaging._drivers.impl_rabbit [req-4152b72a-6be8-4e8e-8764-60e46a992324 ] Connecting to AMQP server on 10.140.33.216:5671
2015-08-22 03:08:44.878 37239 INFO oslo.messaging._drivers.impl_rabbit [req-4152b72a-6be8-4e8e-8764-60e46a992324 ] Connected to AMQP server on 10.140.33.216:5671
2015-08-22 03:08:44.900 37239 AUDIT nova.service [-] Starting compute node (version 2014.2.3)
2015-08-22 03:08:44.906 37239 ERROR nova.virt.libvirt.driver [-] Connection to libvirt failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied
2015-08-22 03:08:44.906 37239 TRACE nova.virt.libvirt.driver Traceback (most recent call last):
2015-08-22 03:08:44.906 37239 TRACE nova.virt.libvirt.driver   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 840, in _connect
2015-08-22 03:08:44.906 37239 TRACE nova.virt.libvirt.driver     libvirt.openAuth, uri, auth, flags)
2015-08-22 03:08:44.906 37239 TRACE nova.virt.libvirt.driver   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call
2015-08-22 03:08:44.906 37239 TRACE nova.virt.libvirt.driver     rv = execute(f,*args,**kwargs)
2015-08-22 03:08:44.906 37239 TRACE nova.virt.libvirt.driver   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker
2015-08-22 03:08:44.906 37239 TRACE nova.virt.libvirt.driver     rv = meth(*args,**kwargs)
2015-08-22 03:08:44.906 37239 TRACE nova.virt.libvirt.driver   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 105, in openAuth
2015-08-22 03:08:44.906 37239 TRACE nova.virt.libvirt.driver     if ret is None:raise libvirtError('virConnectOpenAuth() failed')
<span style="color:#ff0000;">2015-08-22 03:08:44.906 37239 TRACE nova.virt.libvirt.driver libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied
2015-08-22 03:08:44.906 37239 TRACE nova.virt.libvirt.driver</span>
2015-08-22 03:08:44.937 37239 ERROR nova.openstack.common.threadgroup [-] Connection to the hypervisor is broken on host: quasarsdnve1.ibm.com
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 125, in wait
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup     x.wait()
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 47, in wait
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup     return self.thread.wait()
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wait
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup     return self._exit_event.wait()
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup     return hubs.get_hub().switch()
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py", line 490, in run_service
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup     service.start()
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 164, in start
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup     self.manager.init_host()
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1149, in init_host
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup     self.driver.init_host(host=self.host)
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 697, in init_host
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup     self._do_quality_warnings()
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 680, in _do_quality_warnings
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup     caps = self._get_host_capabilities()
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3300, in _get_host_capabilities
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup     xmlstr = self._conn.getCapabilities()
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 776, in _get_connection
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup     wrapped_conn = self._get_new_connection()
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 729, in _get_new_connection
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup     wrapped_conn = self._connect(self.uri(), self.read_only)
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 849, in _connect
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup     raise exception.HypervisorUnavailable(host=CONF.host)
<span style="color:#cc0000;">2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup HypervisorUnavailable: Connection to the hypervisor is broken on host: quasarsdnve1.ibm.com</span>
2015-08-22 03:08:44.937 37239 TRACE nova.openstack.common.threadgroup

Solution: 

After checking the nova-compute.log on compute node,  i found that nova can't luanch libvirt driver because of the user permission.  Since user nova is not in user group libvirtd, add it then restart nova-compute again.  It works fine. 

root@quasarsdnve1:~# id nova
uid=107(nova) gid=112(nova) groups=112(nova),999(docker)
root@quasarsdnve1:~# usermod -G libvirtd nova
root@quasarsdnve1:~# id nova
uid=107(nova) gid=112(nova) groups=112(nova),113(libvirtd)

root@quasarsdnve1:~# service nova-compute restart
stop: Unknown instance:
nova-compute start/running, process 7156
root@quasarsdnve1:~#
root@quasarsdnve1:~# service nova-compute status
nova-compute start/running, process 7156