Bug #8995
closedDocument space requirement for mongodb
Description
Possibly an upstream issue, but mongod does not start during katello-installer on CentOS7.
I'm installing Katello 2.1
Output:
[root@foreman ~]# katello-installer
Could not start Service[mongodb]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait mongod start' returned 1: Redirecting to /bin/systemctl start mongod.service
/Stage[main]/Mongodb::Server::Service/Service[mongodb]/ensure: change from stopped to running failed: Could not start Service[mongodb]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait mongod start' returned 1: Redirecting to /bin/systemctl start mongod.service
/Stage[main]/Apache::Service/Service[httpd]: Failed to call refresh: Could not start Service[httpd]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait httpd start' returned 1: Redirecting to /bin/systemctl start httpd.service
/Stage[main]/Apache::Service/Service[httpd]: Could not start Service[httpd]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait httpd start' returned 1: Redirecting to /bin/systemctl start httpd.service
/Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
/Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
/Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman.rhel7.local]: Failed to call refresh: Could not load data from https://foreman.rhel7.local
/Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman.rhel7.local]: Could not load data from https://foreman.rhel7.local
Installing Done [100%] [............................................................................................]
Something went wrong! Check the log for ERROR-level output
The full log is at /var/log/katello-installer/katello-installer.log
[root@foreman ~]#
When I try starting mongod manually, it also fails:
[root@foreman ~]# systemctl start mongod.service
Job for mongod.service failed. See 'systemctl status mongod.service' and 'journalctl -xn' for details.
[root@foreman ~]# systemctl status mongod.service
mongod.service - High-performance, schema-free document-oriented database
Loaded: loaded (/usr/lib/systemd/system/mongod.service; disabled)
Active: failed (Result: exit-code) since Fri 2015-01-16 11:53:14 GMT; 10s ago
Process: 32023 ExecStart=/usr/bin/mongod $OPTIONS run (code=exited, status=100)
Jan 16 11:53:14 foreman.rhel7.local systemd1: Starting High-performance, schema-free document-oriented database...
Jan 16 11:53:14 foreman.rhel7.local mongod32023: about to fork child process, waiting until server is ready for connections.
Jan 16 11:53:14 foreman.rhel7.local mongod32023: forked process: 32025
Jan 16 11:53:14 foreman.rhel7.local mongod32023: ERROR: child process failed, exited with error number 100
Jan 16 11:53:14 foreman.rhel7.local systemd1: mongod.service: control process exited, code=exited status=100
Jan 16 11:53:14 foreman.rhel7.local systemd1: Failed to start High-performance, schema-free document-oriented database.
Jan 16 11:53:14 foreman.rhel7.local systemd1: Unit mongod.service entered failed state.
[root@foreman ~]#
I'm following the install notes at: http://www.katello.org/docs/2.1/installation/index.html rather than the katello-deploy method in order to test the installation method. I have fixed the yum repositories to make sure I am getting the right software:
[root@foreman ~]# yum repolist
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: mirror.as29550.net
* epel: mirror.bytemark.co.uk
* extras: mirror.bytemark.co.uk
* updates: mirror.bytemark.co.uk
repo id repo name status
base/7/x86_64 CentOS-7 - Base 8,465
epel/x86_64 Extra Packages for Enterprise Linux 7 - x86_64 7,031
extras/7/x86_64 CentOS-7 - Extras 103
foreman/x86_64 Foreman 1.7 308
foreman-plugins/x86_64 Foreman plugins 1.7 158
katello/7/x86_64 Katello Stable 62
katello-candlepin/7/x86_64 An open source entitlement management system. 6
katello-pulp/7/x86_64 Pulp Community Releases 41
puppetlabs-deps/x86_64 Puppet Labs Dependencies El 7 - x86_64 17
puppetlabs-products/x86_64 Puppet Labs Products El 7 - x86_64 123
rhscl-ruby193-epel-7-x86_64 Ruby193 - epel-7-x86_64 405
rhscl-v8314-epel-7-x86_64 V8314 - epel-7-x86_64 23
updates/7/x86_64 CentOS-7 - Updates 1,568
repolist: 18,310
[root@foreman ~]#
Updated by Duncan Innes almost 10 years ago
I'm aware of Issue #8478 and have tried both the workarounds listed in http://projects.theforeman.org/issues/8478#note-4
Neither of these appear to work for me however.
My test machine is rebuilt clean from PXE boot for every test, so I don't think I've got any old edits hanging around.
Updated by Justin Sherrill almost 10 years ago
apologies if i missed it, but you can confirm the exact version of mongodb-server you are running?
Updated by Duncan Innes almost 10 years ago
Justin Sherrill wrote:
apologies if i missed it, but you can confirm the exact version of mongodb-server you are running?
Just reinstalled the test box again and I currently have:
[root@foreman ~]# rpm -qa | grep mongo
mongodb-server-2.6.5-2.el7.x86_64
python-pymongo-2.5.2-2.el7.x86_64
mongodb-2.6.5-2.el7.x86_64
You have mail in /var/spool/mail/root
[root@foreman ~]#
Going to run katello-installer now (no flags given - using defaults all round). Will post mongo versions after the install as well.
Updated by Duncan Innes almost 10 years ago
Getting the same version of mongodb after installation now too:
[root@foreman ~]# rpm -qa | grep mongo
mongodb-server-2.6.5-2.el7.x86_64
python-pymongo-2.5.2-2.el7.x86_64
mongodb-2.6.5-2.el7.x86_64
[root@foreman ~]#
The repositories available to the machine are:
[root@foreman ~]# yum repolist
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: mirrors.coreix.net
* epel: mirror.bytemark.co.uk
* extras: mirrors.coreix.net
* updates: mirrors.coreix.net
repo id repo name status
base/7/x86_64 CentOS-7 - Base 8,465
epel/x86_64 Extra Packages for Enterprise Linux 7 - x86_64 7,143
extras/7/x86_64 CentOS-7 - Extras 104
foreman/x86_64 Foreman 1.7 324
foreman-plugins/x86_64 Foreman plugins 1.7 160
katello/7/x86_64 Katello Stable 63
katello-candlepin/7/x86_64 An open source entitlement management system. 6
katello-pulp/7/x86_64 Pulp Community Releases 41
puppetlabs-deps/x86_64 Puppet Labs Dependencies El 7 - x86_64 17
puppetlabs-products/x86_64 Puppet Labs Products El 7 - x86_64 125
rhscl-ruby193-epel-7-x86_64 Ruby193 - epel-7-x86_64 405
rhscl-v8314-epel-7-x86_64 V8314 - epel-7-x86_64 23
updates/7/x86_64 CentOS-7 - Updates 1,615
repolist: 18,491
[root@foreman ~]#
Updated by Justin Sherrill almost 10 years ago
Hey Duncan,
Can you upload a foreman-debug to the rsync server (instructions given at the end of the command)? and then paste here the filename.
-Justin
Updated by Duncan Innes almost 10 years ago
Justin Sherrill wrote:
Hey Duncan,
Can you upload a foreman-debug to the rsync server (instructions given at the end of the command)? and then paste here the filename.
-Justin
Try foreman-debug-HDOMh.tar.xz
I opted for y at the end of the command to rsync it for me. I then tried to do it manually afterwards, but keep getting errors.
If that fails, I renamed it to foreman-debug-Duncan.tar.xz and managed to upload that one successfully.
D
Updated by Justin Sherrill almost 10 years ago
- Target version deleted (
66)
Debugging with a user on IRC with a similar issue, it ended up that /var/run/mongodb didn't exist. Can you see if this is your case too?
Updated by Duncan Innes almost 10 years ago
Left 'Katello' laptop at work, so can't check until Monday. But will test that as soon as I can Monday. Will try to get on IRC too if possible.
Updated by Duncan Innes almost 10 years ago
Eric,
The mongodb test is passed now, but there are other failures in my default install. I don't remember the Katello packages being at version 2.2 last week (could be wrong). Should I close this and open new bugs to track the issue I find?
Duncan
Updated by Duncan Innes almost 10 years ago
Right, getting confused about what changes have been going on in the last week.
I checked Bug #9163 which shows install issues similar to what I now find myself facing. But downgrading puppet as suggest in that issue now gets me back to facing the issue I reported here.
I'm looking at /var/run/mogodb at the moment though - so that exists in my current install, although I can't figure if it got there because of the install that failed due to the high version number of puppet.
Updated by Duncan Innes almost 10 years ago
OK - after getting a bit misdirected by the arrival of 2.2 into the nightlies, I've amended my OS install to fetch the Katello 2.1 repo RPM:
The install guide needs to be updated to reflect this.
Running another install now to see what the result is.
D
Updated by Duncan Innes almost 10 years ago
I'm still getting a failure at the mongodb point:
[root@foreman ~]# katello-installer
Could not start Service[mongodb]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait mongod start' returned 1: Redirecting to /bin/systemctl start mongod.service
/Stage[main]/Mongodb::Server::Service/Service[mongodb]/ensure: change from stopped to running failed: Could not start Service[mongodb]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait mongod start' returned 1: Redirecting to /bin/systemctl start mongod.service
/Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman.rhel7.local]: Failed to call refresh: Could not load data from https://foreman.rhel7.local
/Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman.rhel7.local]: Could not load data from https://foreman.rhel7.local
Installing Done [100%] [........................................]
Something went wrong! Check the log for ERROR-level output
The full log is at /var/log/katello-installer/katello-installer.log
[root@foreman ~]#
With attempts to start mongodb producing:
[root@foreman mongodb]# systemctl restart mongod.service
Job for mongod.service failed. See 'systemctl status mongod.service' and 'journalctl -xn' for details.
[root@foreman mongodb]# systemctl status mongod.service
mongod.service - High-performance, schema-free document-oriented database
Loaded: loaded (/usr/lib/systemd/system/mongod.service; disabled)
Active: failed (Result: exit-code) since Mon 2015-02-02 16:39:24 GMT; 5s ago
Process: 8612 ExecStart=/usr/bin/mongod $OPTIONS run (code=exited, status=100)
Feb 02 16:39:24 foreman.rhel7.local systemd1: Starting High-performance, schema-free document-oriented database...
Feb 02 16:39:24 foreman.rhel7.local mongod8612: about to fork child process, waiting until server is ready for con...ions.
Feb 02 16:39:24 foreman.rhel7.local mongod8612: forked process: 8614
Feb 02 16:39:24 foreman.rhel7.local mongod8612: ERROR: child process failed, exited with error number 100
Feb 02 16:39:24 foreman.rhel7.local systemd1: mongod.service: control process exited, code=exited status=100
Feb 02 16:39:24 foreman.rhel7.local systemd1: Failed to start High-performance, schema-free document-oriented database.
Feb 02 16:39:24 foreman.rhel7.local systemd1: Unit mongod.service entered failed state.
Hint: Some lines were ellipsized, use -l to show in full.
[root@foreman mongodb]# rpm -qa | grep mongod
mongodb-server-2.6.5-2.el7.x86_64
mongodb-2.6.5-2.el7.x86_64
[root@foreman mongodb]# ls -laFs /var/run/mongodb/
total 0
0 drwxr-xr-x. 2 mongodb root 40 Feb 2 16:39 ./
0 drwxr-xr-x. 33 root root 1060 Feb 2 16:28 ../
[root@foreman mongodb]#
Updated by Duncan Innes almost 10 years ago
Have uploaded: foreman-debug-64zYP.tar.xz to the debug-incoming folder.
Updated by Duncan Innes almost 10 years ago
Have just rebuilt test server from scratch and tried to enable/start mongod.service before any attempt at katello-install:
[root@foreman ~]# rpm -qa | grep mongo
mongodb-server-2.6.5-2.el7.x86_64
python-pymongo-2.5.2-2.el7.x86_64
mongodb-2.6.5-2.el7.x86_64
[root@foreman ~]# systemctl status mongod.service
mongod.service - High-performance, schema-free document-oriented database
Loaded: loaded (/usr/lib/systemd/system/mongod.service; disabled)
Active: inactive (dead)
[root@foreman ~]# systemctl enable mongod.service
ln -s '/usr/lib/systemd/system/mongod.service' '/etc/systemd/system/multi-user.target.wants/mongod.service'
[root@foreman ~]# systemctl status mongod.service
mongod.service - High-performance, schema-free document-oriented database
Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled)
Active: inactive (dead)
[root@foreman ~]# systemctl start mongod.service
Job for mongod.service failed. See 'systemctl status mongod.service' and 'journalctl -xn' for details.
[root@foreman ~]# systemctl status mongod.service
mongod.service - High-performance, schema-free document-oriented database
Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled)
Active: failed (Result: exit-code) since Tue 2015-02-03 10:43:49 GMT; 4s ago
Process: 1147 ExecStart=/usr/bin/mongod $OPTIONS run (code=exited, status=100)
Feb 03 10:43:49 foreman.rhel7.local mongod1147: about to fork child process, waiting until server is ready for connections.
Feb 03 10:43:49 foreman.rhel7.local mongod1147: forked process: 1149
Feb 03 10:43:49 foreman.rhel7.local mongod1147: ERROR: child process failed, exited with error number 100
Feb 03 10:43:49 foreman.rhel7.local systemd1: mongod.service: control process exited, code=exited status=100
Feb 03 10:43:49 foreman.rhel7.local systemd1: Failed to start High-performance, schema-free document-oriented database.
Feb 03 10:43:49 foreman.rhel7.local systemd1: Unit mongod.service entered failed state.
[root@foreman ~]#
Is this an issue with mongod rather than Katello?
Updated by Duncan Innes almost 10 years ago
Looks like it's me being a bit dense and the Hardware Requirements perhaps needing a bit of clarification.
Reason: We split the disk into various partitions:
[root@foreman ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_root-lv_root 20G 2.2G 18G 11% /
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 8.4M 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/mapper/vg_root-lv_tmp 1014M 33M 982M 4% /tmp
/dev/mapper/vg_root-lv_home 1014M 33M 982M 4% /home
/dev/mapper/vg_root-lv_var 2.0G 245M 1.8G 13% /var
/dev/mapper/vg_root-lv_pulp 32G 33M 32G 1% /var/lib/pulp
/dev/vda2 509M 104M 405M 21% /boot
[root@foreman ~]#
Which seems to cover the 30Gb for /var/lib/pulp and the 20Gb for root that is called for. We run into problems here though because the /var provides too little space for /var/lib/mongodb (which I've now rectified)
I now run into http startup problems, but I'll log those elsewhere.
Do you want me to close this an open a new one asking for clarification in the Hardware Requirements section of the Install notes? Or deal with that in this ticket?
Updated by Justin Sherrill almost 10 years ago
- Category changed from Installer to Documentation
- Difficulty set to easy
ahh, we can deal with it in this issue. Changing category and title.
Updated by Justin Sherrill almost 10 years ago
- Subject changed from katello-installer: mongod service failing to Document space requirement for mongodb
Updated by Stephen Benjamin almost 10 years ago
- Status changed from New to Closed
- Pull request https://github.com/Katello/katello.org/pull/113 added
- Pull request deleted (
)