Project

General

Profile

Actions

Bug #8995

closed

Document space requirement for mongodb

Added by Duncan Innes almost 10 years ago. Updated over 6 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
Documentation
Target version:
Difficulty:
easy
Triaged:
Yes
Fixed in Releases:
Found in Releases:

Description

Possibly an upstream issue, but mongod does not start during katello-installer on CentOS7.

I'm installing Katello 2.1

Output:

[root@foreman ~]# katello-installer
Could not start Service[mongodb]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait mongod start' returned 1: Redirecting to /bin/systemctl start mongod.service
/Stage[main]/Mongodb::Server::Service/Service[mongodb]/ensure: change from stopped to running failed: Could not start Service[mongodb]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait mongod start' returned 1: Redirecting to /bin/systemctl start mongod.service
/Stage[main]/Apache::Service/Service[httpd]: Failed to call refresh: Could not start Service[httpd]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait httpd start' returned 1: Redirecting to /bin/systemctl start httpd.service
/Stage[main]/Apache::Service/Service[httpd]: Could not start Service[httpd]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait httpd start' returned 1: Redirecting to /bin/systemctl start httpd.service
/Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
/Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
/Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman.rhel7.local]: Failed to call refresh: Could not load data from https://foreman.rhel7.local
/Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman.rhel7.local]: Could not load data from https://foreman.rhel7.local
Installing Done [100%] [............................................................................................]
Something went wrong! Check the log for ERROR-level output
The full log is at /var/log/katello-installer/katello-installer.log
[root@foreman ~]#

When I try starting mongod manually, it also fails:

[root@foreman ~]# systemctl start mongod.service
Job for mongod.service failed. See 'systemctl status mongod.service' and 'journalctl -xn' for details.
[root@foreman ~]# systemctl status mongod.service
mongod.service - High-performance, schema-free document-oriented database
Loaded: loaded (/usr/lib/systemd/system/mongod.service; disabled)
Active: failed (Result: exit-code) since Fri 2015-01-16 11:53:14 GMT; 10s ago
Process: 32023 ExecStart=/usr/bin/mongod $OPTIONS run (code=exited, status=100)

Jan 16 11:53:14 foreman.rhel7.local systemd1: Starting High-performance, schema-free document-oriented database...
Jan 16 11:53:14 foreman.rhel7.local mongod32023: about to fork child process, waiting until server is ready for connections.
Jan 16 11:53:14 foreman.rhel7.local mongod32023: forked process: 32025
Jan 16 11:53:14 foreman.rhel7.local mongod32023: ERROR: child process failed, exited with error number 100
Jan 16 11:53:14 foreman.rhel7.local systemd1: mongod.service: control process exited, code=exited status=100
Jan 16 11:53:14 foreman.rhel7.local systemd1: Failed to start High-performance, schema-free document-oriented database.
Jan 16 11:53:14 foreman.rhel7.local systemd1: Unit mongod.service entered failed state.
[root@foreman ~]#

I'm following the install notes at: http://www.katello.org/docs/2.1/installation/index.html rather than the katello-deploy method in order to test the installation method. I have fixed the yum repositories to make sure I am getting the right software:

[root@foreman ~]# yum repolist
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile * base: mirror.as29550.net * epel: mirror.bytemark.co.uk * extras: mirror.bytemark.co.uk * updates: mirror.bytemark.co.uk
repo id repo name status
base/7/x86_64 CentOS-7 - Base 8,465
epel/x86_64 Extra Packages for Enterprise Linux 7 - x86_64 7,031
extras/7/x86_64 CentOS-7 - Extras 103
foreman/x86_64 Foreman 1.7 308
foreman-plugins/x86_64 Foreman plugins 1.7 158
katello/7/x86_64 Katello Stable 62
katello-candlepin/7/x86_64 An open source entitlement management system. 6
katello-pulp/7/x86_64 Pulp Community Releases 41
puppetlabs-deps/x86_64 Puppet Labs Dependencies El 7 - x86_64 17
puppetlabs-products/x86_64 Puppet Labs Products El 7 - x86_64 123
rhscl-ruby193-epel-7-x86_64 Ruby193 - epel-7-x86_64 405
rhscl-v8314-epel-7-x86_64 V8314 - epel-7-x86_64 23
updates/7/x86_64 CentOS-7 - Updates 1,568
repolist: 18,310
[root@foreman ~]#

Actions #1

Updated by Duncan Innes almost 10 years ago

I'm aware of Issue #8478 and have tried both the workarounds listed in http://projects.theforeman.org/issues/8478#note-4

Neither of these appear to work for me however.

My test machine is rebuilt clean from PXE boot for every test, so I don't think I've got any old edits hanging around.

Actions #2

Updated by Eric Helms almost 10 years ago

  • Triaged changed from No to Yes
Actions #3

Updated by Justin Sherrill almost 10 years ago

apologies if i missed it, but you can confirm the exact version of mongodb-server you are running?

Actions #4

Updated by Duncan Innes almost 10 years ago

Justin Sherrill wrote:

apologies if i missed it, but you can confirm the exact version of mongodb-server you are running?

Just reinstalled the test box again and I currently have:

[root@foreman ~]# rpm -qa | grep mongo
mongodb-server-2.6.5-2.el7.x86_64
python-pymongo-2.5.2-2.el7.x86_64
mongodb-2.6.5-2.el7.x86_64
You have mail in /var/spool/mail/root
[root@foreman ~]#

Going to run katello-installer now (no flags given - using defaults all round). Will post mongo versions after the install as well.

Actions #5

Updated by Duncan Innes almost 10 years ago

Getting the same version of mongodb after installation now too:

[root@foreman ~]# rpm -qa | grep mongo
mongodb-server-2.6.5-2.el7.x86_64
python-pymongo-2.5.2-2.el7.x86_64
mongodb-2.6.5-2.el7.x86_64
[root@foreman ~]#

The repositories available to the machine are:

[root@foreman ~]# yum repolist
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile * base: mirrors.coreix.net * epel: mirror.bytemark.co.uk * extras: mirrors.coreix.net * updates: mirrors.coreix.net
repo id repo name status
base/7/x86_64 CentOS-7 - Base 8,465
epel/x86_64 Extra Packages for Enterprise Linux 7 - x86_64 7,143
extras/7/x86_64 CentOS-7 - Extras 104
foreman/x86_64 Foreman 1.7 324
foreman-plugins/x86_64 Foreman plugins 1.7 160
katello/7/x86_64 Katello Stable 63
katello-candlepin/7/x86_64 An open source entitlement management system. 6
katello-pulp/7/x86_64 Pulp Community Releases 41
puppetlabs-deps/x86_64 Puppet Labs Dependencies El 7 - x86_64 17
puppetlabs-products/x86_64 Puppet Labs Products El 7 - x86_64 125
rhscl-ruby193-epel-7-x86_64 Ruby193 - epel-7-x86_64 405
rhscl-v8314-epel-7-x86_64 V8314 - epel-7-x86_64 23
updates/7/x86_64 CentOS-7 - Updates 1,615
repolist: 18,491
[root@foreman ~]#

Actions #6

Updated by Justin Sherrill almost 10 years ago

Hey Duncan,

Can you upload a foreman-debug to the rsync server (instructions given at the end of the command)? and then paste here the filename.

-Justin

Actions #7

Updated by Duncan Innes almost 10 years ago

Justin Sherrill wrote:

Hey Duncan,

Can you upload a foreman-debug to the rsync server (instructions given at the end of the command)? and then paste here the filename.

-Justin

Try foreman-debug-HDOMh.tar.xz

I opted for y at the end of the command to rsync it for me. I then tried to do it manually afterwards, but keep getting errors.

If that fails, I renamed it to foreman-debug-Duncan.tar.xz and managed to upload that one successfully.

D

Actions #8

Updated by Eric Helms almost 10 years ago

  • Target version set to 66
Actions #9

Updated by Justin Sherrill almost 10 years ago

  • Target version deleted (66)

Debugging with a user on IRC with a similar issue, it ended up that /var/run/mongodb didn't exist. Can you see if this is your case too?

Actions #10

Updated by Duncan Innes almost 10 years ago

Left 'Katello' laptop at work, so can't check until Monday. But will test that as soon as I can Monday. Will try to get on IRC too if possible.

Actions #11

Updated by Duncan Innes almost 10 years ago

Eric,

The mongodb test is passed now, but there are other failures in my default install. I don't remember the Katello packages being at version 2.2 last week (could be wrong). Should I close this and open new bugs to track the issue I find?

Duncan

Actions #12

Updated by Duncan Innes almost 10 years ago

Right, getting confused about what changes have been going on in the last week.

I checked Bug #9163 which shows install issues similar to what I now find myself facing. But downgrading puppet as suggest in that issue now gets me back to facing the issue I reported here.

I'm looking at /var/run/mogodb at the moment though - so that exists in my current install, although I can't figure if it got there because of the install that failed due to the high version number of puppet.

Actions #13

Updated by Duncan Innes almost 10 years ago

OK - after getting a bit misdirected by the arrival of 2.2 into the nightlies, I've amended my OS install to fetch the Katello 2.1 repo RPM:

http://fedorapeople.org/groups/katello/releases/yum/2.1/katello/RHEL/7Server/x86_64/katello-repos-latest.rpm

The install guide needs to be updated to reflect this.

Running another install now to see what the result is.

D

Actions #14

Updated by Duncan Innes almost 10 years ago

I'm still getting a failure at the mongodb point:

[root@foreman ~]# katello-installer
Could not start Service[mongodb]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait mongod start' returned 1: Redirecting to /bin/systemctl start mongod.service
/Stage[main]/Mongodb::Server::Service/Service[mongodb]/ensure: change from stopped to running failed: Could not start Service[mongodb]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait mongod start' returned 1: Redirecting to /bin/systemctl start mongod.service
/Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman.rhel7.local]: Failed to call refresh: Could not load data from https://foreman.rhel7.local
/Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman.rhel7.local]: Could not load data from https://foreman.rhel7.local
Installing Done [100%] [........................................]
Something went wrong! Check the log for ERROR-level output
The full log is at /var/log/katello-installer/katello-installer.log
[root@foreman ~]#

With attempts to start mongodb producing:

[root@foreman mongodb]# systemctl restart mongod.service
Job for mongod.service failed. See 'systemctl status mongod.service' and 'journalctl -xn' for details.
[root@foreman mongodb]# systemctl status mongod.service
mongod.service - High-performance, schema-free document-oriented database
Loaded: loaded (/usr/lib/systemd/system/mongod.service; disabled)
Active: failed (Result: exit-code) since Mon 2015-02-02 16:39:24 GMT; 5s ago
Process: 8612 ExecStart=/usr/bin/mongod $OPTIONS run (code=exited, status=100)

Feb 02 16:39:24 foreman.rhel7.local systemd1: Starting High-performance, schema-free document-oriented database...
Feb 02 16:39:24 foreman.rhel7.local mongod8612: about to fork child process, waiting until server is ready for con...ions.
Feb 02 16:39:24 foreman.rhel7.local mongod8612: forked process: 8614
Feb 02 16:39:24 foreman.rhel7.local mongod8612: ERROR: child process failed, exited with error number 100
Feb 02 16:39:24 foreman.rhel7.local systemd1: mongod.service: control process exited, code=exited status=100
Feb 02 16:39:24 foreman.rhel7.local systemd1: Failed to start High-performance, schema-free document-oriented database.
Feb 02 16:39:24 foreman.rhel7.local systemd1: Unit mongod.service entered failed state.
Hint: Some lines were ellipsized, use -l to show in full.
[root@foreman mongodb]# rpm -qa | grep mongod
mongodb-server-2.6.5-2.el7.x86_64
mongodb-2.6.5-2.el7.x86_64
[root@foreman mongodb]# ls -laFs /var/run/mongodb/
total 0
0 drwxr-xr-x. 2 mongodb root 40 Feb 2 16:39 ./
0 drwxr-xr-x. 33 root root 1060 Feb 2 16:28 ../
[root@foreman mongodb]#

Actions #15

Updated by Duncan Innes almost 10 years ago

Have uploaded: foreman-debug-64zYP.tar.xz to the debug-incoming folder.

Actions #16

Updated by Duncan Innes almost 10 years ago

Have just rebuilt test server from scratch and tried to enable/start mongod.service before any attempt at katello-install:

[root@foreman ~]# rpm -qa | grep mongo
mongodb-server-2.6.5-2.el7.x86_64
python-pymongo-2.5.2-2.el7.x86_64
mongodb-2.6.5-2.el7.x86_64
[root@foreman ~]# systemctl status mongod.service
mongod.service - High-performance, schema-free document-oriented database
Loaded: loaded (/usr/lib/systemd/system/mongod.service; disabled)
Active: inactive (dead)

[root@foreman ~]# systemctl enable mongod.service
ln -s '/usr/lib/systemd/system/mongod.service' '/etc/systemd/system/multi-user.target.wants/mongod.service'
[root@foreman ~]# systemctl status mongod.service
mongod.service - High-performance, schema-free document-oriented database
Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled)
Active: inactive (dead)

[root@foreman ~]# systemctl start mongod.service
Job for mongod.service failed. See 'systemctl status mongod.service' and 'journalctl -xn' for details.
[root@foreman ~]# systemctl status mongod.service
mongod.service - High-performance, schema-free document-oriented database
Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled)
Active: failed (Result: exit-code) since Tue 2015-02-03 10:43:49 GMT; 4s ago
Process: 1147 ExecStart=/usr/bin/mongod $OPTIONS run (code=exited, status=100)

Feb 03 10:43:49 foreman.rhel7.local mongod1147: about to fork child process, waiting until server is ready for connections.
Feb 03 10:43:49 foreman.rhel7.local mongod1147: forked process: 1149
Feb 03 10:43:49 foreman.rhel7.local mongod1147: ERROR: child process failed, exited with error number 100
Feb 03 10:43:49 foreman.rhel7.local systemd1: mongod.service: control process exited, code=exited status=100
Feb 03 10:43:49 foreman.rhel7.local systemd1: Failed to start High-performance, schema-free document-oriented database.
Feb 03 10:43:49 foreman.rhel7.local systemd1: Unit mongod.service entered failed state.
[root@foreman ~]#

Is this an issue with mongod rather than Katello?

Actions #17

Updated by Duncan Innes almost 10 years ago

Looks like it's me being a bit dense and the Hardware Requirements perhaps needing a bit of clarification.

Reason: We split the disk into various partitions:

[root@foreman ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_root-lv_root 20G 2.2G 18G 11% /
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 8.4M 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/mapper/vg_root-lv_tmp 1014M 33M 982M 4% /tmp
/dev/mapper/vg_root-lv_home 1014M 33M 982M 4% /home
/dev/mapper/vg_root-lv_var 2.0G 245M 1.8G 13% /var
/dev/mapper/vg_root-lv_pulp 32G 33M 32G 1% /var/lib/pulp
/dev/vda2 509M 104M 405M 21% /boot
[root@foreman ~]#

Which seems to cover the 30Gb for /var/lib/pulp and the 20Gb for root that is called for. We run into problems here though because the /var provides too little space for /var/lib/mongodb (which I've now rectified)

I now run into http startup problems, but I'll log those elsewhere.

Do you want me to close this an open a new one asking for clarification in the Hardware Requirements section of the Install notes? Or deal with that in this ticket?

Actions #18

Updated by Justin Sherrill almost 10 years ago

  • Category changed from Installer to Documentation
  • Difficulty set to easy

ahh, we can deal with it in this issue. Changing category and title.

Actions #19

Updated by Justin Sherrill almost 10 years ago

  • Subject changed from katello-installer: mongod service failing to Document space requirement for mongodb
Actions #20

Updated by Stephen Benjamin almost 10 years ago

  • Status changed from New to Closed
  • Pull request https://github.com/Katello/katello.org/pull/113 added
  • Pull request deleted ()
Actions

Also available in: Atom PDF