Project

General

Profile

Actions
Copy link

Bug #8478

closed

MongoDB fails to start with a RHEL 7 system

Added by Og Maciel almost 10 years ago. Updated over 6 years ago.

Status:
Closed
Priority:
High
Assignee:
Ivan Necas
Category:
Installer
Target version:
Katello 2.2
Difficulty:
Triaged:
Yes
Bugzilla link:
Pull request:
https://github.com/Katello/katello-installer/pull/144
Fixed in Releases:
Found in Releases:
Red Hat JIRA:

Description

Our automation against nightly on a RHEL 7 system failed today with the following error:

[10.8.29.189] out: [ INFO 2014-11-21 02:04:31 verbose]  Class[Mongodb::Server::Service]: Scheduling refresh of Service[mongodb]
[10.8.29.189] out: [ERROR 2014-11-21 02:06:01 verbose]  Could not start Service[mongodb]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait mongod start' returned 1: Redirecting to /bin/systemctl start  mongod.service
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] Job for mongod.service failed. See 'systemctl status mongod.service' and 'journalctl -xn' for details.
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/util/execution.rb:203:in `execute'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/provider.rb:115:in `execute'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/provider/service/service.rb:25:in `texecute'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/provider/service/service.rb:39:in `ucommand'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/provider/service/base.rb:65:in `start'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/type/service.rb:83:in `block (3 levels) in <module:Puppet>'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/property.rb:197:in `call_valuemethod'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/property.rb:498:in `set'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/property.rb:581:in `sync'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/type/service.rb:94:in `sync'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:204:in `sync'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in `sync_if_needed'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:in `perform_changes'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:204:in `apply'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:217:in `eval_resource'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `call'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels) in evaluate'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/util.rb:327:in `block in thinmark'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/benchmark.rb:296:in `realtime'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/util.rb:326:in `thinmark'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in evaluate'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/configurer.rb:118:in `block in apply_catalog'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/benchmark.rb:296:in `realtime'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/configurer.rb:117:in `apply_catalog'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/configurer.rb:224:in `run_internal'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/configurer.rb:132:in `block in run'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/context.rb:64:in `override'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet.rb:244:in `override'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/configurer.rb:131:in `run'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/application/apply.rb:297:in `apply_catalog'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/application/apply.rb:231:in `block in main'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/context.rb:64:in `override'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet.rb:244:in `override'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/application/apply.rb:193:in `main'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/application/apply.rb:154:in `run_command'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/util.rb:488:in `exit_on_fail'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `run'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] /usr/bin/puppet:8:in `<main>'
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] Wrapped exception:
[10.8.29.189] out: [ INFO 2014-11-21 02:06:01 verbose] Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait mongod start' returned 1: Redirecting to /bin/systemctl start  mongod.service

systemctl status mongod.service shows:

# systemctl status mongod.service
mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/usr/lib/systemd/system/mongod.service; disabled)
   Active: failed (Result: timeout) since Fri 2014-11-21 02:06:01 EST; 5h 28min ago

Nov 21 02:04:31 qe-foreman-rhel7.usersys.redhat.com systemd[1]: Starting High-performance, schema-free document-oriented database...
Nov 21 02:04:31 qe-foreman-rhel7.usersys.redhat.com mongod[24992]: about to fork child process, waiting until server is ready for connections.
Nov 21 02:04:31 qe-foreman-rhel7.usersys.redhat.com mongod[24992]: forked process: 24994
Nov 21 02:04:31 qe-foreman-rhel7.usersys.redhat.com mongod[24992]: child process started successfully, parent exiting
Nov 21 02:04:31 qe-foreman-rhel7.usersys.redhat.com systemd[1]: PID file /var/run/mongodb/mongod.pid not readable (yet?) after start.
Nov 21 02:06:01 qe-foreman-rhel7.usersys.redhat.com systemd[1]: mongod.service operation timed out. Terminating.
Nov 21 02:06:01 qe-foreman-rhel7.usersys.redhat.com systemd[1]: Failed to start High-performance, schema-free document-oriented database.
Nov 21 02:06:01 qe-foreman-rhel7.usersys.redhat.com systemd[1]: Unit mongod.service entered failed state.

Trying to run the installer command again, 

katello-installer -v -d --foreman-admin-password="changeme" --foreman-initial-organization="Default_Organization" --foreman-initial-location="Default_Location"
also failed again to start mongodb. Logs are attached.

Environment:
  • candlepin-0.9.34-1.el7.noarch
  • candlepin-common-1.0.14-1.el7.noarch
  • candlepin-selinux-0.9.34-1.el7.noarch
  • candlepin-tomcat-0.9.34-1.el7.noarch
  • elasticsearch-0.90.10-7.el7.noarch
  • foreman-1.8.0-0.develop.201411201326giteaefbec.el7.noarch
  • foreman-compute-1.8.0-0.develop.201411201326giteaefbec.el7.noarch
  • foreman-gce-1.8.0-0.develop.201411201326giteaefbec.el7.noarch
  • foreman-libvirt-1.8.0-0.develop.201411201326giteaefbec.el7.noarch
  • foreman-ovirt-1.8.0-0.develop.201411201326giteaefbec.el7.noarch
  • foreman-postgresql-1.8.0-0.develop.201411201326giteaefbec.el7.noarch
  • foreman-proxy-1.8.0-0.develop.201411201507gitfc67f91.el7.noarch
  • foreman-release-1.8.0-0.develop.201411201326giteaefbec.el7.noarch
  • foreman-selinux-1.8.0-0.develop.201410280941git10de1c5.el7.noarch
  • foreman-vmware-1.8.0-0.develop.201411201326giteaefbec.el7.noarch
  • katello-2.1.0-1.201411061509gitb0b8f43.el7.noarch
  • katello-certs-tools-2.0.1-1.el7.noarch
  • katello-default-ca-1.0-1.noarch
  • katello-installer-2.1.0-1.201411172212git9d6f424.el7.noarch
  • katello-repos-2.1.1-1.el7.noarch
  • katello-server-ca-1.0-1.noarch
  • openldap-2.4.39-3.el7.x86_64
  • pulp-docker-plugins-0.2.1-0.2.beta.el7.noarch
  • pulp-katello-0.3-3.el7.noarch
  • pulp-nodes-common-2.5.0-0.7.beta.el7.noarch
  • pulp-nodes-parent-2.5.0-0.7.beta.el7.noarch
  • pulp-puppet-plugins-2.5.0-0.7.beta.el7.noarch
  • pulp-puppet-tools-2.5.0-0.7.beta.el7.noarch
  • pulp-rpm-plugins-2.5.0-0.7.beta.el7.noarch
  • pulp-selinux-2.5.0-0.7.beta.el7.noarch
  • pulp-server-2.5.0-0.7.beta.el7.noarch
  • python-ldap-2.4.6-6.el7.x86_64
  • ruby193-rubygem-ldap_fluff-0.3.3-1.el7.noarch
  • ruby193-rubygem-net-ldap-0.3.1-2.el7.noarch
  • ruby193-rubygem-runcible-1.3.0-1.el7.noarch
  • rubygem-hammer_cli-0.1.3-1.201411181453git0c80831.el7.noarch
  • rubygem-hammer_cli_foreman-0.1.3-1.201411121216git9381fc5.el7.noarch
  • rubygem-hammer_cli_foreman_tasks-0.0.3-2.201409091410git163c264.git.0.988ca80.el7.noarch
  • rubygem-hammer_cli_import-0.10.4-1.el7.noarch
  • rubygem-hammer_cli_katello-0.0.6-1.201411202238git7fa4778.git.0.a5a9308.el7.noarch

Files

foreman-debug-pZotY.tar.xz foreman-debug-pZotY.tar.xz 225 KB Og Maciel, 11/21/2014 07:42 AM

Related issues 3 (0 open — 3 closed)

Has duplicate Katello - Bug #8700: EL7 mongod.service typo in unit file Duplicate12/14/2014Actions
Has duplicate Katello - Bug #9163: katello-installer failed on rhel7 with error: Could not enable postgresql: Execution of '/sbin/chkconfig --add postgresql' returned 1:Closed01/29/2015Actions
Has duplicate Katello - Bug #9202: katello-installer should use systemd instead of chkconfig for all services on rhel7Duplicatedustin tsang02/03/2015Actions
Actions
Copy link
 #1

Updated by Ivan Necas almost 10 years ago

Looks like an issue with epel update on mongodb, searching for things to fix. For now, the workaround should be downgrading to previous version:

yum downgrade https://kojipkgs.fedoraproject.org//packages/mongodb/2.4.9/3.el7/x86_64/mongodb-2.4.9-3.el7.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/mongodb/2.4.9/3.el7/x86_64/mongodb-server-2.4.9-3.el7.x86_64.rpm

Actions
Copy link
 #2

Updated by Ivan Necas almost 10 years ago

  • Assignee set to Ivan Necas
  • Target version set to 61
Actions
Copy link
 #3

Updated by Ivan Necas almost 10 years ago

  • Status changed from New to Assigned
Actions
Copy link
 #4

Updated by Ivan Necas almost 10 years ago

The proper fix will involve sending some updates to upstream puppet mongo module and them updating katello-installer accordingly.

The temporary workarounds are either downagrade of mongodb packages:

yum downgrade https://kojipkgs.fedoraproject.org//packages/mongodb/2.4.9/3.el7/x86_64/mongodb-2.4.9-3.el7.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/mongodb/2.4.9/3.el7/x86_64/mongodb-server-2.4.9-3.el7.x86_64.rpm

or updating the path to the mongodb pid file locally in the installer:

https://github.com/iNecas/katello-installer/commit/445521c2617b24cd39bcc4ae34502cf415ca5cb9

I will continue with working on a proper solution next week.

Actions
Copy link
 #5

Updated by Ivan Necas almost 10 years ago

  • Status changed from Assigned to Ready For Testing
Actions
Copy link
 #6

Updated by Eric Helms almost 10 years ago

  • Target version changed from 61 to 62
Actions
Copy link
 #7

Updated by The Foreman Bot almost 10 years ago

  • Pull request https://github.com/Katello/katello-installer/pull/144 added
  • Pull request deleted ()
Actions
Copy link
 #8

Updated by Eric Helms almost 10 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100
Actions
Copy link
 #9

Updated by Eric Helms almost 10 years ago

  • Triaged changed from No to Yes
Actions
Copy link
 #10

Updated by Florian Koch almost 10 years ago

  • Has duplicate Bug #8700: EL7 mongod.service typo in unit file added
Actions
Copy link
 #11

Updated by dustin tsang almost 10 years ago

  • Has duplicate Bug #9163: katello-installer failed on rhel7 with error: Could not enable postgresql: Execution of '/sbin/chkconfig --add postgresql' returned 1: added
Actions
Copy link
 #12

Updated by dustin tsang almost 10 years ago

  • Has duplicate Bug #9202: katello-installer should use systemd instead of chkconfig for all services on rhel7 added
Actions
Copy link
 #13

Updated by Eric Helms over 9 years ago

  • Translation missing: en.field_release set to 23
Actions
Copy link

Also available in: Atom PDF