Project

General

Profile

Bug #8946

Exception thrown when foreman-proxy restarts with SIGTERM

Added by cristian falcas about 5 years ago. Updated over 1 year ago.

Status:
Closed
Priority:
Normal
Category:
Core
Target version:
Difficulty:
Triaged:
Bugzilla link:
Fixed in Releases:
Found in Releases:

Description

This is what abrt with rubygem-abrt is sending after restarting foreman-proxy:

backtrace:
:/usr/share/foreman-proxy/lib/smart_proxy.rb:143:in `join': SIGTERM (SignalException)
: from /usr/share/foreman-proxy/lib/smart_proxy.rb:143:in `launch'
: from /usr/share/foreman-proxy/bin/smart-proxy:6:in `<main>'

Packages installed:
  1. rpm -qa | grep foreman| sort
    foreman-1.7.1-1.el7.noarch
    foreman-cli-1.7.1-1.el7.noarch
    foreman-compute-1.7.1-1.el7.noarch
    foreman-postgresql-1.7.1-1.el7.noarch
    foreman-proxy-1.7.1-1.el7.noarch
    foreman-selinux-1.7.1-1.el7.noarch
    foreman-vmware-1.7.1-1.el7.noarch
    ruby193-rubygem-foreman_abrt-0.0.5-1.el7.noarch
    ruby193-rubygem-foreman_column_view-0.2.0-1.el7.noarch
    ruby193-rubygem-foreman_docker-0.2.0-2.el7.noarch
    ruby193-rubygem-foreman_hooks-0.3.7-2.el7.noarch
    ruby193-rubygem-foreman_openscap-0.2.0-1.el7.centos.noarch
    ruby193-rubygem-foreman-tasks-0.6.10-3.el7.noarch
    ruby193-rubygem-foreman_templates-1.4.0-2.el7.noarch
    ruby193-rubygem-puppetdb_foreman-0.1.2-1.el7.noarch
    rubygem-hammer_cli_foreman-0.1.3-1.el7.noarch

Associated revisions

Revision 616d41a7 (diff)
Added by Dmitri Dolguikh over 4 years ago

Fixes #8946: Added a handler for SIGTERM

History

#1 Updated by Lukas Zapletal about 5 years ago

  • Category set to 101

Can you provide full steps to reproduce? How you enabled ABRT? What was the procedure, step by step?

I assume RHEL7 with 1.7.1. When you upgrade to 1.7.2 does it disappear?

Thanks.

#2 Updated by cristian falcas about 5 years ago

I can't seem to find 1.7.2 on the official repos.

The installation was like this:
- install abrt, configure it to send emails on
- install rubygem-abrt

That was all. at each smart proxy restart I receive an email with that exception.

#3 Updated by Byron Miller over 4 years ago

  • Tracker changed from Bug to Support
  • Status changed from New to Feedback
  • Assignee set to Byron Miller

Good Morning!

Foreman has been updated to version 1.8.2. I am able to use the ABRT plugin on CentOS 6 & 7.1 without having this issue. Are you still experiencing this issue or have you been able to patch up and see if it works as expected?

Thanks,

#4 Updated by cristian falcas over 4 years ago

I still receive the same crash report:

time: Tue 14 Jul 2015 06:15:43 AM EDT
cmdline: ruby /usr/share/foreman-proxy/bin/smart-proxy
uid: 994 (foreman-proxy)
abrt_version: 2.1.11
executable: /usr/share/foreman-proxy/bin/smart-proxy
hostname: v-so-repo-04.company.net
kernel: 3.10.0-229.7.2.el7.x86_64
last_occurrence: 1436868943
pid: 16243
pkg_arch: noarch
pkg_epoch: 0
pkg_name: foreman-proxy
pkg_release: 2.el7
pkg_version: 1.8.2
runlevel: N 3
username: foreman-proxy

backtrace:
:/usr/share/foreman-proxy/lib/smart_proxy.rb:144:in `join': SIGTERM (SignalException)
: from /usr/share/foreman-proxy/lib/smart_proxy.rb:144:in `launch'
: from /usr/share/foreman-proxy/bin/smart-proxy:6:in `<main>'

Packages installed:
[root@v-so-repo-04 abrt]# rpm -qa | grep foreman | sort
foreman-debug-1.8.2-1.el7.noarch
foreman-proxy-1.8.2-2.el7.noarch

#5 Updated by Dominic Cleal over 4 years ago

  • Tracker changed from Support to Bug
  • Subject changed from foreman smart proxy throws exception on restart to Exception thrown when foreman-proxy restarts with SIGTERM
  • Category changed from 101 to Core
  • Status changed from Feedback to New
  • Assignee deleted (Byron Miller)

The report makes sense to me, as when the proxy receives a SIGTERM (during service foreman-proxy restart, which is triggered by logrotate) then it lets the exception bubble up in order for the process to quit.

Since ABRT hooks into Ruby, it's probably just picking that up and registering it as a crash. An exception handler could be added for SIGTERM instead.

#6 Updated by Dmitri Dolguikh over 4 years ago

change introduced in 335a1610 (https://github.com/theforeman/smart-proxy/commit/335a1610fae8cd74720e2d9841d904deb8ff1881) should fix this. I'm seeing that smart-proxy exits with 143 status after it's been sent SIGTERM.

#7 Updated by Dominic Cleal over 4 years ago

It looks like the handler originally came from #11098, which is also a 1.10.0 change. The thing is that both handlers re-raise the exception, so I'm unsure if it'll help with ABRT picking up the exception - it'll need retesting I think.

#8 Updated by Sher Chowdhury over 4 years ago

Ok I just built a new foreman 1.10.0 on Centos 7.1.

[vagrant@puppetmaster Desktop]$ sudo -i
ABRT has detected 1 problem(s). For more info run: abrt-cli list
abrt-cli list
id 008dd6d67896ba3a3a940e0cf1937db45746c1b3
reason:         /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require': Could not autoload puppet/indirector/report/processor: SIGTERM (Puppet::Error)
time:           Fri 23 Oct 2015 15:34:47 BST
cmdline:        /usr/bin/ruby /usr/bin/puppet agent --no-daemonize
package:        puppet-3.8.3-1.el7
uid:            0 (root)
count:          1
Directory:      /var/spool/abrt/rbhook-2015-10-23-15:34:47-15923

The Autoreporting feature is disabled. Please consider enabling it by issuing
'abrt-auto-reporting enabled' as a user with root privileges
gem list

*** LOCAL GEMS ***

abrt (0.0.6)
ansi (1.4.3)
apipie-bindings (0.0.14)
awesome_print (1.0.2)
bigdecimal (1.2.0)
bundler (1.10.6, 1.3.1)
bundler_ext (0.3.0)
clamp (1.0.0)
colored (1.2)
cri (2.6.1)
faraday (0.9.2)
faraday_middleware (0.9.2)
faraday_middleware-multi_json (0.0.6)
ffi (1.9.3)
gssapi (1.1.2)
hashie (2.0.5)
highline (1.6.21)
io-console (0.4.2)
json (1.7.7)
kafo (0.6.12)
kafo_parsers (0.0.5)
little-plugger (1.1.3)
log4r (1.1.10)
logging (1.8.2)
mime-types (1.19)
minitar (0.5.4)
multi_json (1.10.1)
multipart-post (2.0.0)
net-http-persistent (2.8)
netrc (0.7.7)
oauth (0.4.7)
passenger (4.0.18)
powerbar (1.0.11)
psych (2.0.0)
r10k (2.0.3)
rack (1.5.2)
rack-protection (1.3.2)
rake (10.4.2, 0.9.6)
rdoc (4.0.0)
rest-client (1.6.7)
rkerberos (0.1.3)
rubyipmi (0.10.0)
semantic_puppet (0.1.1)
sinatra (1.3.5)
thor (0.17.0)
tilt (1.3.7)
You have mail in /var/spool/mail/root

[root@puppetmaster ~]# facter -p
architecture => x86_64
augeasversion => 1.1.0
bios_release_date => 12/01/2006
bios_vendor => innotek GmbH
bios_version => VirtualBox
blockdevice_sda_model => VBOX HARDDISK
blockdevice_sda_size => 41943040000
blockdevice_sda_vendor => ATA
blockdevices => sda
boardmanufacturer => Oracle Corporation
boardproductname => VirtualBox
boardserialnumber => 0
dhcp_servers => {"system"=>"10.0.2.2", "enp0s3"=>"10.0.2.2"}
domain => local
facterversion => 2.4.4
filesystems => iso9660,xfs
fqdn => puppetmaster.local
gid => root
hardwareisa => x86_64
hardwaremodel => x86_64
hostname => puppetmaster
id => root
interfaces => enp0s3,enp0s8,lo
ipaddress => 10.0.2.15
ipaddress_enp0s3 => 10.0.2.15
ipaddress_lo => 127.0.0.1
is_virtual => true
kernel => Linux
kernelmajversion => 3.10
kernelrelease => 3.10.0-229.14.1.el7.x86_64
kernelversion => 3.10.0
macaddress => 08:00:27:60:20:81
macaddress_enp0s3 => 08:00:27:60:20:81
macaddress_enp0s8 => 08:00:27:7b:9d:4c
manufacturer => innotek GmbH
memoryfree => 317.06 MB
memoryfree_mb => 317.06
memorysize => 993.52 MB
memorysize_mb => 993.52
mtu_enp0s3 => 1500
mtu_enp0s8 => 1500
mtu_lo => 65536
netmask => 255.255.255.0
netmask_enp0s3 => 255.255.255.0
netmask_lo => 255.0.0.0
network_enp0s3 => 10.0.2.0
network_lo => 127.0.0.0
operatingsystem => CentOS
operatingsystemmajrelease => 7
operatingsystemrelease => 7.1.1503
os => {"name"=>"CentOS", "family"=>"RedHat", "release"=>{"major"=>"7", "minor"=>"1", "full"=>"7.1.1503"}}
osfamily => RedHat
partitions => {"sda1"=>{"uuid"=>"9dac9928-7542-4a76-bae5-d02a5b493f17", "size"=>"1024000", "mount"=>"/boot", "filesystem"=>"xfs"}, "sda2"=>{"size"=>"80893952", "filesystem"=>"LVM2_member"}}
path => /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/bin:/root/bin:/sbin
physicalprocessorcount => 1
processor0 => Intel(R) Core(TM) i7 CPU       M 620  @ 2.67GHz
processor1 => Intel(R) Core(TM) i7 CPU       M 620  @ 2.67GHz
processorcount => 2
processors => {"models"=>["Intel(R) Core(TM) i7 CPU       M 620  @ 2.67GHz", "Intel(R) Core(TM) i7 CPU       M 620  @ 2.67GHz"], "count"=>2, "physicalcount"=>1}
productname => VirtualBox
ps => ps -ef
puppetversion => 3.8.3
rubyplatform => x86_64-linux
rubysitedir => /usr/local/share/ruby/site_ruby/
rubyversion => 2.0.0
selinux => true
selinux_config_mode => permissive
selinux_config_policy => targeted
selinux_current_mode => permissive
selinux_enforced => false
selinux_policyversion => 28
serialnumber => 0
sshecdsakey => AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLvUCR0cHrjKJvegObFEBctUOSWbiOC7kmcIhAZq0bjQVbO25pbbqwA1R73/JGWsTau0VKb0m9yaq4dV5z9MI70=
sshed25519key => AAAAC3NzaC1lZDI1NTE5AAAAIJUWez+MxJyNzJlsHxCHXqlrO1eamzEVslnGyXsBV0XE
sshfp_ecdsa => SSHFP 3 1 49da5c639715666b95cf587c93f6a67754963326
SSHFP 3 2 7bbac5d1d16f00bbbb31f4a7ad7d7034bd162a94125e779fb518ed4e504da8c1
sshfp_ed25519 => SSHFP 4 1 36ddbddeb7d8606df5c46eef747d8978630383b0
SSHFP 4 2 056fb735e786873175cbd6eb1f27aea85f864c32742c3fb440efe4b72711c034
sshfp_rsa => SSHFP 1 1 3931bc434b636c699c0bc79806061502c62f08c8
SSHFP 1 2 8af64d3d355c10aa1a3481f67979e5a2a612915c395baccbf52e063dc4a66fc3
sshrsakey => AAAAB3NzaC1yc2EAAAADAQABAAABAQCXeG4+HexB5kHP3ThDehI44Hn2C3wG6hECAs/q8e5OEdCvGYrOsAiR+0Wf7n5QUYpdijftSGRnI8JAOjNba93lyDvrKb1KnIP8OEPV8Dw1SUJ7pAQ45iShPBcQs4ZCXsO+ixlqaJmpFaf5jDkg2+4r0uVDybOZ6lkCU2gCHGehrEjojwTmqeGJ+QAgZNUkrjjLqgl5k3W+SZAa6G4VFHdRdvIvzXm3s+LnQSDZ4bmpwsZ82oizwGlHnohQ3/wcNMg4PgmXWGBI/91K2aJnFpxxd85wKRIhOPo8emNtE0dvjsb5T+Wm8UJXpvn43t+YER8hcb9bfMlRU9MZ1j33VNwz
swapfree => 860.21 MB
swapfree_mb => 860.21
swapsize => 1024.00 MB
swapsize_mb => 1024.00
system_uptime => {"seconds"=>1582, "hours"=>0, "days"=>0, "uptime"=>"0:26 hours"}
timezone => BST
type => Other
uniqueid => 007f0101
uptime => 0:26 hours
uptime_days => 0
uptime_hours => 0
uptime_seconds => 1582
uuid => 782E535D-4293-4A27-AC32-DE94AE12213C
virtual => virtualbox

[root@puppetmaster ~]# rpm -qa | grep foreman | sort
foreman-1.10.0-0.1.RC1.el7.noarch
foreman-cli-1.10.0-0.1.RC1.el7.noarch
foreman-debug-1.10.0-0.1.RC1.el7.noarch
foreman-installer-1.10.0-0.1.RC1.el7.noarch
foreman-postgresql-1.10.0-0.1.RC1.el7.noarch
foreman-proxy-1.10.0-0.1.RC1.el7.noarch
foreman-release-1.10.0-0.1.RC1.el7.noarch
foreman-release-scl-1-1.el7.x86_64
foreman-selinux-1.10.0-0.1.RC1.el7.noarch
tfm-rubygem-foreman_setup-3.0.2-1.fm1_10.el7.noarch
tfm-rubygem-hammer_cli_foreman-0.4.0-1.el7.noarch

#9 Updated by Dominic Cleal over 4 years ago

I don't think that's related, it's a completely different process reporting an error - it's the Puppet agent, not the smart proxy.

The interesting thing will be to see if your 1.10.0-RC1 system logs a further ABRT notice when logrotation of foreman-proxy occurs.

#10 Updated by Sher Chowdhury over 4 years ago

After rebooting my foreman vm, I then get the following error messages:

login as: root
root@192.168.50.10's password:
ABRT has detected 2 problem(s). For more info run: abrt-cli list --since 1445610273
[root@puppetmaster ~]# abrt-cli list --since 1445610273
id 9af525842c3a277d36c430a4f82b84bbc8ad42ef
reason:         /usr/share/foreman-proxy/lib/launcher.rb:128:in `join': SIGTERM (SignalException)
time:           Fri 23 Oct 2015 15:35:08 BST
cmdline:        ruby /usr/share/foreman-proxy/bin/smart-proxy
package:        foreman-proxy-1.10.0-0.1.RC1.el7
uid:            992 (foreman-proxy)
count:          1
Directory:      /var/spool/abrt/rbhook-2015-10-23-15:35:08-2267

id 008dd6d67896ba3a3a940e0cf1937db45746c1b3
reason:         /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require': Could not autoload puppet/indirector/report/processor: SIGTERM (Puppet::Error)
time:           Fri 23 Oct 2015 15:34:47 BST
cmdline:        /usr/bin/ruby /usr/bin/puppet agent --no-daemonize
package:        puppet-3.8.3-1.el7
uid:            0 (root)
count:          1
Directory:      /var/spool/abrt/rbhook-2015-10-23-15:34:47-15923

The Autoreporting feature is disabled. Please consider enabling it by issuing
'abrt-auto-reporting enabled' as a user with root privileges

#11 Updated by Dominic Cleal over 4 years ago

That looks like it's still there then, thanks for confirming. I'd guess it's also happening during service stop (on shutdown) as well as restart (on logrotate).

#12 Updated by The Foreman Bot over 4 years ago

  • Status changed from New to Ready For Testing
  • Pull request https://github.com/theforeman/smart-proxy/pull/332 added
  • Pull request deleted ()

#13 Updated by The Foreman Bot over 4 years ago

  • Assignee set to Dmitri Dolguikh

#14 Updated by Dominic Cleal over 4 years ago

  • Legacy Backlogs Release (now unused) set to 71

#15 Updated by Anonymous over 4 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100

Also available in: Atom PDF