Project

General

Profile

Actions
Copy link

Bug #8478

closed

MongoDB fails to start with a RHEL 7 system

Added by Og Maciel over 9 years ago. Updated almost 6 years ago.

Status:
Closed
Priority:
High
Assignee:
Ivan Necas
Category:
Installer
Target version:
Katello 2.2
Difficulty:
Triaged:
Yes
Bugzilla link:
Pull request:
https://github.com/Katello/katello-installer/pull/144
Fixed in Releases:
Found in Releases:
Red Hat JIRA:

Description

Our automation against nightly on a RHEL 7 system failed today with the following error:

[10.8.29.189] out: [ INFO 2014-11-21 02:04:31 verbose]  Class[Mongodb::Server::Service]: Scheduling refresh of Service[mongodb]
[10.8.29.189] out: [ERROR 2014-11-21 02:06:01 verbose]  Could not start Service[mongodb]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait mongod start' returned 1: Redirecting to /bin/systemctl start  mongod.service
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] Job for mongod.service failed. See 'systemctl status mongod.service' and 'journalctl -xn' for details.
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/util/execution.rb:203:in `execute'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/provider.rb:115:in `execute'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/provider/service/service.rb:25:in `texecute'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/provider/service/service.rb:39:in `ucommand'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/provider/service/base.rb:65:in `start'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/type/service.rb:83:in `block (3 levels) in <module:Puppet>'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/property.rb:197:in `call_valuemethod'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/property.rb:498:in `set'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/property.rb:581:in `sync'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/type/service.rb:94:in `sync'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:204:in `sync'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in `sync_if_needed'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:in `perform_changes'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:204:in `apply'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:217:in `eval_resource'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `call'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels) in evaluate'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/util.rb:327:in `block in thinmark'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/benchmark.rb:296:in `realtime'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/util.rb:326:in `thinmark'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in evaluate'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/configurer.rb:118:in `block in apply_catalog'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/benchmark.rb:296:in `realtime'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/configurer.rb:117:in `apply_catalog'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/configurer.rb:224:in `run_internal'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/configurer.rb:132:in `block in run'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/context.rb:64:in `override'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet.rb:244:in `override'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/configurer.rb:131:in `run'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/application/apply.rb:297:in `apply_catalog'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/application/apply.rb:231:in `block in main'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/context.rb:64:in `override'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet.rb:244:in `override'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/application/apply.rb:193:in `main'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/application/apply.rb:154:in `run_command'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/util.rb:488:in `exit_on_fail'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `run'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/bin/puppet:8:in `<main>'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] Wrapped exception:
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait mongod start' returned 1: Redirecting to /bin/systemctl start  mongod.service

systemctl status mongod.service shows:

# systemctl status mongod.service
mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/usr/lib/systemd/system/mongod.service; disabled)
   Active: failed (Result: timeout) since Fri 2014-11-21 02:06:01 EST; 5h 28min ago

Nov 21 02:04:31 qe-foreman-rhel7.usersys.redhat.com systemd[1]: Starting High-performance, schema-free document-oriented database...
Nov 21 02:04:31 qe-foreman-rhel7.usersys.redhat.com mongod[24992]: about to fork child process, waiting until server is ready for connections.
Nov 21 02:04:31 qe-foreman-rhel7.usersys.redhat.com mongod[24992]: forked process: 24994
Nov 21 02:04:31 qe-foreman-rhel7.usersys.redhat.com mongod[24992]: child process started successfully, parent exiting
Nov 21 02:04:31 qe-foreman-rhel7.usersys.redhat.com systemd[1]: PID file /var/run/mongodb/mongod.pid not readable (yet?) after start.
Nov 21 02:06:01 qe-foreman-rhel7.usersys.redhat.com systemd[1]: mongod.service operation timed out. Terminating.
Nov 21 02:06:01 qe-foreman-rhel7.usersys.redhat.com systemd[1]: Failed to start High-performance, schema-free document-oriented database.
Nov 21 02:06:01 qe-foreman-rhel7.usersys.redhat.com systemd[1]: Unit mongod.service entered failed state.

Trying to run the installer command again, 

katello-installer -v -d --foreman-admin-password="changeme" --foreman-initial-organization="Default_Organization" --foreman-initial-location="Default_Location"
also failed again to start mongodb. Logs are attached.

Environment:
  • candlepin-0.9.34-1.el7.noarch
  • candlepin-common-1.0.14-1.el7.noarch
  • candlepin-selinux-0.9.34-1.el7.noarch
  • candlepin-tomcat-0.9.34-1.el7.noarch
  • elasticsearch-0.90.10-7.el7.noarch
  • foreman-1.8.0-0.develop.201411201326giteaefbec.el7.noarch
  • foreman-compute-1.8.0-0.develop.201411201326giteaefbec.el7.noarch
  • foreman-gce-1.8.0-0.develop.201411201326giteaefbec.el7.noarch
  • foreman-libvirt-1.8.0-0.develop.201411201326giteaefbec.el7.noarch
  • foreman-ovirt-1.8.0-0.develop.201411201326giteaefbec.el7.noarch
  • foreman-postgresql-1.8.0-0.develop.201411201326giteaefbec.el7.noarch
  • foreman-proxy-1.8.0-0.develop.201411201507gitfc67f91.el7.noarch
  • foreman-release-1.8.0-0.develop.201411201326giteaefbec.el7.noarch
  • foreman-selinux-1.8.0-0.develop.201410280941git10de1c5.el7.noarch
  • foreman-vmware-1.8.0-0.develop.201411201326giteaefbec.el7.noarch
  • katello-2.1.0-1.201411061509gitb0b8f43.el7.noarch
  • katello-certs-tools-2.0.1-1.el7.noarch
  • katello-default-ca-1.0-1.noarch
  • katello-installer-2.1.0-1.201411172212git9d6f424.el7.noarch
  • katello-repos-2.1.1-1.el7.noarch
  • katello-server-ca-1.0-1.noarch
  • openldap-2.4.39-3.el7.x86_64
  • pulp-docker-plugins-0.2.1-0.2.beta.el7.noarch
  • pulp-katello-0.3-3.el7.noarch
  • pulp-nodes-common-2.5.0-0.7.beta.el7.noarch
  • pulp-nodes-parent-2.5.0-0.7.beta.el7.noarch
  • pulp-puppet-plugins-2.5.0-0.7.beta.el7.noarch
  • pulp-puppet-tools-2.5.0-0.7.beta.el7.noarch
  • pulp-rpm-plugins-2.5.0-0.7.beta.el7.noarch
  • pulp-selinux-2.5.0-0.7.beta.el7.noarch
  • pulp-server-2.5.0-0.7.beta.el7.noarch
  • python-ldap-2.4.6-6.el7.x86_64
  • ruby193-rubygem-ldap_fluff-0.3.3-1.el7.noarch
  • ruby193-rubygem-net-ldap-0.3.1-2.el7.noarch
  • ruby193-rubygem-runcible-1.3.0-1.el7.noarch
  • rubygem-hammer_cli-0.1.3-1.201411181453git0c80831.el7.noarch
  • rubygem-hammer_cli_foreman-0.1.3-1.201411121216git9381fc5.el7.noarch
  • rubygem-hammer_cli_foreman_tasks-0.0.3-2.201409091410git163c264.git.0.988ca80.el7.noarch
  • rubygem-hammer_cli_import-0.10.4-1.el7.noarch
  • rubygem-hammer_cli_katello-0.0.6-1.201411202238git7fa4778.git.0.a5a9308.el7.noarch

Files

foreman-debug-pZotY.tar.xz foreman-debug-pZotY.tar.xz 225 KB Og Maciel, 11/21/2014 07:42 AM

Related issues 3 (0 open — 3 closed)

Has duplicate Katello - Bug #8700: EL7 mongod.service typo in unit file Duplicate12/14/2014Actions
Has duplicate Katello - Bug #9163: katello-installer failed on rhel7 with error: Could not enable postgresql: Execution of '/sbin/chkconfig --add postgresql' returned 1:Closed01/29/2015Actions
Has duplicate Katello - Bug #9202: katello-installer should use systemd instead of chkconfig for all services on rhel7Duplicatedustin tsang02/03/2015Actions
Actions
Copy link
 #1

Updated by Ivan Necas over 9 years ago

Looks like an issue with epel update on mongodb, searching for things to fix. For now, the workaround should be downgrading to previous version:

yum downgrade https://kojipkgs.fedoraproject.org//packages/mongodb/2.4.9/3.el7/x86_64/mongodb-2.4.9-3.el7.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/mongodb/2.4.9/3.el7/x86_64/mongodb-server-2.4.9-3.el7.x86_64.rpm

Actions
Copy link
 #2

Updated by Ivan Necas over 9 years ago

  • Assignee set to Ivan Necas
  • Target version set to 61
Actions
Copy link
 #3

Updated by Ivan Necas over 9 years ago

  • Status changed from New to Assigned
Actions
Copy link
 #4

Updated by Ivan Necas over 9 years ago

The proper fix will involve sending some updates to upstream puppet mongo module and them updating katello-installer accordingly.

The temporary workarounds are either downagrade of mongodb packages:

yum downgrade https://kojipkgs.fedoraproject.org//packages/mongodb/2.4.9/3.el7/x86_64/mongodb-2.4.9-3.el7.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/mongodb/2.4.9/3.el7/x86_64/mongodb-server-2.4.9-3.el7.x86_64.rpm

or updating the path to the mongodb pid file locally in the installer:

https://github.com/iNecas/katello-installer/commit/445521c2617b24cd39bcc4ae34502cf415ca5cb9

I will continue with working on a proper solution next week.

Actions
Copy link
 #5

Updated by Ivan Necas over 9 years ago

  • Status changed from Assigned to Ready For Testing
Actions
Copy link
 #6

Updated by Eric Helms over 9 years ago

  • Target version changed from 61 to 62
Actions
Copy link
 #7

Updated by The Foreman Bot over 9 years ago

  • Pull request https://github.com/Katello/katello-installer/pull/144 added
  • Pull request deleted ()
Actions
Copy link
 #8

Updated by Eric Helms over 9 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100
Actions
Copy link
 #9

Updated by Eric Helms over 9 years ago

  • Triaged changed from No to Yes
Actions
Copy link
 #10

Updated by Florian Koch over 9 years ago

  • Has duplicate Bug #8700: EL7 mongod.service typo in unit file added
Actions
Copy link
 #11

Updated by dustin tsang about 9 years ago

  • Has duplicate Bug #9163: katello-installer failed on rhel7 with error: Could not enable postgresql: Execution of '/sbin/chkconfig --add postgresql' returned 1: added
Actions
Copy link
 #12

Updated by dustin tsang about 9 years ago

  • Has duplicate Bug #9202: katello-installer should use systemd instead of chkconfig for all services on rhel7 added
Actions
Copy link
 #13

Updated by Eric Helms about 9 years ago

  • translation missing: en.field_release set to 23
Actions
Copy link

Also available in: Atom PDF