Project

General

Profile

Actions

Bug #21616

closed

/etc/puppet/rack queue became full with "/usr/bin/mkpasswd -H MD5" processes

Added by Shod Dan over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Difficulty:
Triaged:
Fixed in Releases:
Found in Releases:

Description

After start, foreman creates strange endless waiting processes from one to five, aka "/usr/bin/mkpasswd -H MD5".

lsof output:

lsof -p 21746
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
mkpasswd 21746 puppet cwd DIR 254,0 4096 274859 /etc/puppet/rack
mkpasswd 21746 puppet rtd DIR 254,0 4096 2 /
mkpasswd 21746 puppet txt REG 254,0 18744 15108 /usr/bin/mkpasswd
mkpasswd 21746 puppet mem REG 254,0 1738176 131154 /lib/x86_64-linux-gnu/libc-2.19.so
mkpasswd 21746 puppet mem REG 254,0 35176 131156 /lib/x86_64-linux-gnu/libcrypt-2.19.so
mkpasswd 21746 puppet mem REG 254,0 140928 131151 /lib/x86_64-linux-gnu/ld-2.19.so
mkpasswd 21746 puppet 0u unix 0xffff8800369af800 0t0 553016 /tmp/passenger.vYp3TPz/apps.s/preloader.17awm7e type=STREAM
mkpasswd 21746 puppet 1w FIFO 0,8 0t0 566717 pipe
mkpasswd 21746 puppet 2w FIFO 0,8 0t0 552279 pipe

After ~20 minutes, rake queue became full (100) and foreman-proxy rejects all connection from puppet agents with message "web server under heavy load".

passenger-status output:
Version : 5.1.11
Date : 2017-11-09 15:21:43 +0200
Instance: ODOWGCEJ (Apache/2.4.10 (Debian) OpenSSL/1.0.2k Phusion_Passenger/5.1.11)

----------- General information -----------
Max pool size : 6
App groups : 2
Processes : 6
Requests in top-level queue : 0

----------- Application groups -----------
/usr/share/foreman:
App root: /usr/share/foreman
Requests in queue: 0 * PID: 19725 Sessions: 0 Processed: 594 Uptime: 50m 30s
CPU: 3% Memory : 390M Last used: 10m 19s ago

/etc/puppet/rack:
App root: /etc/puppet/rack
Requests in queue: 100 * PID: 19664 Sessions: 1 Processed: 515 Uptime: 50m 46s
CPU: 3% Memory : 191M Last used: 42m 38s ago * PID: 19671 Sessions: 1 Processed: 212 Uptime: 50m 46s
CPU: 1% Memory : 147M Last used: 41m 45s ago * PID: 20327 Sessions: 1 Processed: 1282 Uptime: 47m 54s
CPU: 9% Memory : 183M Last used: 19m 36s ago * PID: 22044 Sessions: 1 Processed: 133 Uptime: 41m 46s
CPU: 0% Memory : 98M Last used: 30m 46s ago * PID: 26264 Sessions: 1 Processed: 445 Uptime: 22m 31s
CPU: 6% Memory : 134M Last used: 10m 20s ago

If kill these proccesses with "killall mkpasswd" command, the queue is flushed and foreman-proxy accept connections from puppet agents again.

Please advise, how to prevent this behaviour? What is the source and the reason for mkpasswd command?

Actions #1

Updated by Shod Dan over 6 years ago

  • Status changed from New to Resolved

The empty password for root in hiera was cause of the issue.

Actions

Also available in: Atom PDF