Bug #20993
closed[Katello 3.4] regular qdrouterd segfaults
Description
Hey,
after the update from katello 3.3 to 3.4.5 in conjunction with moving from rhel 7.3 to rhel 7.4 I've noticed regular segfaults of qdrouterd on my systems.
[Sep11 11:42] qdrouterd[68786]: segfault at 88 ip 00007f2bb65f4770 sp 00007f2ba913a298 error 6 in libqpid-proton.so.10.0.0[7f2bb65d6000+4b000] [Sep11 19:20] qdrouterd[87470]: segfault at 88 ip 00007fdadadc4770 sp 00007fdace10b298 error 6 in libqpid-proton.so.10.0.0[7fdadada6000+4b000] [Sep12 09:56] qdrouterd[48816]: segfault at 88 ip 00007f4c12e7a770 sp 00007ffe3d538198 error 6 in libqpid-proton.so.10.0.0[7f4c12e5c000+4b000] [Sep13 02:34] qdrouterd[111436]: segfault at 88 ip 00007f3b75827770 sp 00007f3b63ffe298 error 6 in libqpid-proton.so.10.0.0[7f3b75809000+4b000] [Sep13 17:11] qdrouterd[69032]: segfault at 88 ip 00007f80c801e770 sp 00007f80ba363298 error 6 in libqpid-proton.so.10.0.0[7f80c8000000+4b000] [Sep14 07:47] qdrouterd[85459]: segfault at 88 ip 00007fd7eb37b770 sp 00007ffe377a37c8 error 6 in libqpid-proton.so.10.0.0[7fd7eb35d000+4b000] [Sep15 01:30] qdrouterd[4718]: segfault at 88 ip 00007f0067168770 sp 00007f005a4af298 error 6 in libqpid-proton.so.10.0.0[7f006714a000+4b000] [Sep15 19:46] qdrouterd[84734]: segfault at 88 ip 00007fc539c36770 sp 00007fff8fe383e8 error 6 in libqpid-proton.so.10.0.0[7fc539c18000+4b000] [Sep16 14:03] qdrouterd[80819]: segfault at 88 ip 00007fdf7b5ed770 sp 00007fff1b886908 error 6 in libqpid-proton.so.10.0.0[7fdf7b5cf000+4b000] [Sep17 08:21] qdrouterd[23812]: segfault at 88 ip 00007fc32eedb770 sp 00007fc322222298 error 6 in libqpid-proton.so.10.0.0[7fc32eebd000+4b000] [Sep18 02:47] qdrouterd[97833]: segfault at 88 ip 00007ff8c6be7770 sp 00007ff8b972d298 error 6 in libqpid-proton.so.10.0.0[7ff8c6bc9000+4b000] [Sep18 21:09] qdrouterd[36139]: segfault at 88 ip 00007f0304d3e770 sp 00007ffda49253c8 error 6 in libqpid-proton.so.10.0.0[7f0304d20000+4b000]
One thing that strikes me as weird is that my content proxies don't crash nearly as often as the main server does although 95% of clients connect to those. 95% of the clients still run RHEL 7.3 thought :)
# rpm -qa|egrep "qpid|proton"|sort [the cert rpms] python-gofer-proton-2.7.6-1.el7.noarch python-gofer-qpid-2.7.6-1.el7.noarch python-qpid-1.36.0-1.el7.noarch python-qpid-proton-0.17.0-4.el7.x86_64 python-qpid-qmf-1.36.0-1.el7.x86_64 qpid-cpp-client-1.36.0-1.el7.x86_64 qpid-cpp-client-devel-1.36.0-1.el7.x86_64 qpid-cpp-server-1.36.0-1.el7.x86_64 qpid-cpp-server-linearstore-1.36.0-1.el7.x86_64 qpid-dispatch-router-0.8.0-1.el7.x86_64 qpid-proton-c-0.17.0-4.el7.x86_64 qpid-qmf-1.36.0-1.el7.x86_64 qpid-tools-1.36.0-1.el7.noarch tfm-rubygem-qpid_messaging-0.34.1-1.el7.x86_64
Foreman packages on my main foreman/katello server:
candlepin-2.0.40-1.el7.noarch candlepin-selinux-2.0.40-1.el7.noarch foremanmain.tld-foreman-client-1.0-1.noarch foremanmain.tld-foreman-proxy-1.0-1.noarch foremanmain.tld-foreman-proxy-client-1.0-1.noarch foremanmain.tld-qpid-broker-1.0-2.noarch foremanmain.tld-qpid-client-cert-1.0-1.noarch foremanmain.tld-qpid-router-client-1.0-1.noarch foremanmain.tld-qpid-router-server-1.0-1.noarch foreman-1.15.3-1.el7.noarch foreman-cli-1.15.3-1.el7.noarch foreman-compute-1.15.3-1.el7.noarch foreman-debug-1.15.3-1.el7.noarch foreman-installer-1.15.3-1.el7.noarch foreman-installer-katello-3.4.5-1.el7.noarch foreman-postgresql-1.15.3-1.el7.noarch foreman-proxy-1.15.3-1.el7.noarch foreman-release-1.15.3-1.el7.noarch foreman-release-scl-3-1.el7.noarch foreman-selinux-1.15.3-1.el7.noarch foreman-vmware-1.15.3-1.el7.noarch katello-3.4.5-1.el7.noarch katello-agent-3.0.2-1.el7.noarch katello-ca-consumer-foremanmain.tld-1.0-1.noarch katello-certs-tools-2.4.0-1.el7.noarch katello-client-bootstrap-1.4.0-2.el7.noarch katello-common-3.4.5-1.el7.noarch katello-debug-3.4.5-1.el7.noarch katello-default-ca-1.0-1.noarch katello-host-tools-3.0.2-1.el7.noarch katello-host-tools-fact-plugin-3.0.2-1.el7.noarch katello-installer-base-3.4.5-1.el7.noarch katello-repos-3.4.0-3.el7.noarch katello-selinux-3.0.2-1.el7.noarch katello-server-ca-1.0-1.noarch katello-service-3.4.5-1.el7.noarch pulp-admin-client-2.13.2-1.el7.noarch pulp-client-1.0-1.noarch pulp-docker-plugins-2.4.0-2.el7.noarch pulp-katello-1.0.2-1.el7.noarch pulp-puppet-plugins-2.13.2-1.el7.noarch pulp-puppet-tools-2.13.2-1.el7.noarch pulp-rpm-admin-extensions-2.13.2-1.el7.noarch pulp-rpm-handlers-2.13.2-1.el7.noarch pulp-rpm-plugins-2.13.2-1.el7.noarch pulp-selinux-2.13.2-1.el7.noarch pulp-server-2.13.2-1.el7.noarch python-gofer-qpid-2.7.6-1.el7.noarch python-isodate-0.5.0-4.pulp.el7.noarch python-kombu-3.0.33-6.pulp.el7.noarch python-pulp-agent-lib-2.13.2-1.el7.noarch python-pulp-bindings-2.13.2-1.el7.noarch python-pulp-client-lib-2.13.2-1.el7.noarch python-pulp-common-2.13.2-1.el7.noarch python-pulp-docker-common-2.4.0-2.el7.noarch python-pulp-oid_validation-2.13.2-1.el7.noarch python-pulp-puppet-common-2.13.2-1.el7.noarch python-pulp-repoauth-2.13.2-1.el7.noarch python-pulp-rpm-common-2.13.2-1.el7.noarch python-pulp-streamer-2.13.2-1.el7.noarch python-qpid-1.36.0-1.el7.noarch python-qpid-proton-0.17.0-4.el7.x86_64 python-qpid-qmf-1.36.0-1.el7.x86_64 qpid-cpp-client-1.36.0-1.el7.x86_64 qpid-cpp-client-devel-1.36.0-1.el7.x86_64 qpid-cpp-server-1.36.0-1.el7.x86_64 qpid-cpp-server-linearstore-1.36.0-1.el7.x86_64 qpid-dispatch-router-0.8.0-1.el7.x86_64 qpid-proton-c-0.17.0-4.el7.x86_64 qpid-qmf-1.36.0-1.el7.x86_64 qpid-tools-1.36.0-1.el7.noarch rubygem-smart_proxy_pulp-1.3.0-1.el7.noarch tfm-rubygem-activerecord-session_store-0.1.2-2.el7.noarch tfm-rubygem-addressable-2.3.6-4.el7.noarch tfm-rubygem-algebrick-0.7.3-4.el7.noarch tfm-rubygem-ancestry-2.2.1-1.el7.noarch tfm-rubygem-anemone-0.7.2-14.el7.noarch tfm-rubygem-angular-rails-templates-1.0.2-1.el7.noarch tfm-rubygem-apipie-bindings-0.2.0-1.el7.noarch tfm-rubygem-apipie-params-0.0.5-1.el7.noarch tfm-rubygem-apipie-rails-0.4.0-2.el7.noarch tfm-rubygem-audited-4.4.1-1.el7.noarch tfm-rubygem-awesome_print-1.7.0-1.el7.noarch tfm-rubygem-bastion-5.0.10-1.fm1_15.el7.noarch tfm-rubygem-bundler_ext-0.4.1-1.el7.noarch tfm-rubygem-clamp-1.0.0-5.el7.noarch tfm-rubygem-concurrent-ruby-1.0.3-1.fm1_15.el7.noarch tfm-rubygem-concurrent-ruby-edge-0.2.3-1.fm1_15.el7.noarch tfm-rubygem-css_parser-1.4.7-1.el7.noarch tfm-rubygem-daemons-1.2.3-3.el7.noarch tfm-rubygem-deacon-1.0.0-1.el7.noarch tfm-rubygem-deep_cloneable-2.2.2-1.el7.noarch tfm-rubygem-deface-1.2.0-1.el7.noarch tfm-rubygem-diffy-3.0.1-3.el7.noarch tfm-rubygem-docker-api-1.28.0-2.el7.noarch tfm-rubygem-domain_name-0.5.20160310-1.el7.noarch tfm-rubygem-dynflow-0.8.24-1.fm1_15.el7.noarch tfm-rubygem-excon-0.51.0-1.el7.noarch tfm-rubygem-facter-2.4.0-3.el7.x86_64 tfm-rubygem-fast_gettext-1.1.0-1.el7.noarch tfm-rubygem-fog-1.38.0-1.el7.noarch tfm-rubygem-fog-aws-1.2.0-1.el7.noarch tfm-rubygem-fog-core-1.42.0-1.el7.noarch tfm-rubygem-fog-google-0.1.0-2.el7.noarch tfm-rubygem-fog-json-1.0.2-4.el7.noarch tfm-rubygem-fog-openstack-0.1.18-1.el7.noarch tfm-rubygem-fog-rackspace-0.1.4-1.el7.noarch tfm-rubygem-fog-vsphere-1.7.0-1.el7.noarch tfm-rubygem-fog-xenserver-0.2.3-1.el7.noarch tfm-rubygem-fog-xml-0.1.2-4.el7.noarch tfm-rubygem-foreman-tasks-0.9.4-1.fm1_15.el7.noarch tfm-rubygem-foreman-tasks-core-0.1.4-1.fm1_15.el7.noarch tfm-rubygem-foreman_docker-3.2.1-1.fm1_15.el7.noarch tfm-rubygem-foreman_hooks-0.3.14-1.fm1_15.el7.noarch tfm-rubygem-foreman_remote_execution-1.3.3-1.fm1_15.el7.noarch tfm-rubygem-foreman_remote_execution_core-1.0.5-1.fm1_15.el7.noarch tfm-rubygem-foreman_scc_manager-1.0.1-1.fm1_15.el7.noarch tfm-rubygem-foreman_snapshot_management-1.0.0-1.fm1_15.el7.noarch tfm-rubygem-foreman_templates-5.0.1-1.fm1_15.el7.noarch tfm-rubygem-formatador-0.2.1-9.el7.noarch tfm-rubygem-friendly_id-5.1.0-3.el7.noarch tfm-rubygem-get_process_mem-0.2.1-1.el7.noarch tfm-rubygem-gettext_i18n_rails-1.2.1-3.el7.noarch tfm-rubygem-git-1.2.5-7.el7.noarch tfm-rubygem-hammer_cli-0.10.2-1.el7.noarch tfm-rubygem-hammer_cli_foreman-0.10.2-1.el7.noarch tfm-rubygem-hammer_cli_foreman_bootdisk-0.1.3-5.el7.noarch tfm-rubygem-hammer_cli_foreman_docker-0.0.4-2.el7.noarch tfm-rubygem-hammer_cli_foreman_tasks-0.0.10-2.fm1_12.el7.noarch tfm-rubygem-hammer_cli_import-0.11.3-1.el7.noarch tfm-rubygem-hammer_cli_katello-0.10.0-1.el7.noarch tfm-rubygem-hashie-2.0.5-4.el7.noarch tfm-rubygem-highline-1.7.8-2.el7.noarch tfm-rubygem-http-cookie-1.0.2-1.el7.noarch tfm-rubygem-ipaddress-0.8.0-9.el7.noarch tfm-rubygem-jquery-ui-rails-4.1.2-5.el7.noarch tfm-rubygem-katello-3.4.5-1.el7.noarch tfm-rubygem-ldap_fluff-0.4.6-1.el7.noarch tfm-rubygem-little-plugger-1.1.3-21.el7.noarch tfm-rubygem-locale-2.0.9-11.el7.noarch tfm-rubygem-logging-1.8.2-4.el7.noarch tfm-rubygem-net-ldap-0.15.0-1.el7.noarch tfm-rubygem-net-ping-2.0.1-1.el7.noarch tfm-rubygem-net-scp-1.1.0-6.el7.noarch tfm-rubygem-net-ssh-4.0.1-2.el7.noarch tfm-rubygem-netrc-0.7.7-7.el7.noarch tfm-rubygem-oauth-0.5.1-1.el7.noarch tfm-rubygem-parse-cron-0.1.4-2.fm1_11.el7.noarch tfm-rubygem-passenger-4.0.18-9.11.el7.x86_64 tfm-rubygem-passenger-native-4.0.18-9.11.el7.x86_64 tfm-rubygem-passenger-native-libs-4.0.18-9.11.el7.x86_64 tfm-rubygem-pg-0.15.1-3.el7.x86_64 tfm-rubygem-powerbar-1.0.17-1.el7.noarch tfm-rubygem-protected_attributes-1.1.3-3.el7.noarch tfm-rubygem-qpid_messaging-0.34.1-1.el7.x86_64 tfm-rubygem-rabl-0.12.0-2.el7.noarch tfm-rubygem-rack-jsonp-1.3.1-5.el7.noarch tfm-rubygem-rails-i18n-4.0.9-1.el7.noarch tfm-rubygem-rails-observers-0.1.2-7.el7.noarch tfm-rubygem-rainbow-2.2.1-1.el7.noarch tfm-rubygem-rbvmomi-1.10.0-1.el7.noarch tfm-rubygem-responders-2.3.0-1.el7.noarch tfm-rubygem-rest-client-1.8.0-1.el7.noarch tfm-rubygem-roadie-3.2.1-1.el7.noarch tfm-rubygem-roadie-rails-1.1.1-1.el7.noarch tfm-rubygem-robotex-1.0.0-19.el7.noarch tfm-rubygem-ruby2ruby-2.1.3-4.el7.noarch tfm-rubygem-ruby_parser-3.6.3-4.el7.noarch tfm-rubygem-runcible-1.12.0-1.el7.noarch tfm-rubygem-safemode-1.3.2-1.el7.noarch tfm-rubygem-scoped_search-4.1.0-1.el7.noarch tfm-rubygem-secure_headers-3.4.1-1.el7.noarch tfm-rubygem-sequel-4.20.0-5.el7.noarch tfm-rubygem-sexp_processor-4.4.4-3.el7.noarch tfm-rubygem-smart_proxy_dynflow_core-0.1.6-2.fm1_15.el7.noarch tfm-rubygem-sshkey-1.9.0-1.el7.noarch tfm-rubygem-trollop-2.1.2-1.el7.noarch tfm-rubygem-unf-0.1.3-5.el7.noarch tfm-rubygem-unf_ext-0.0.6-6.el7.x86_64 tfm-rubygem-unicode-display_width-1.0.5-1.el7.noarch tfm-rubygem-useragent-0.16.8-1.el7.noarch tfm-rubygem-validates_lengths_from_database-0.5.0-3.el7.noarch tfm-rubygem-webpack-rails-0.9.8-1.el7.noarch tfm-rubygem-wicked-1.1.0-4.el7.noarch tfm-rubygem-will_paginate-3.1.5-1.el7.noarch tfm-rubygem-x-editable-rails-1.5.5-1.el7.noarch tfm-runtime-3.2-9.el7.x86_64
Greetings
Klaas
Updated by Klaas D about 7 years ago
I work around the issue by simply restarting qdrouterd on crash; otherwise goferd will go wild on some clients (https://bugzilla.redhat.com/show_bug.cgi?id=1318015).
# cat /etc/systemd/system/qdrouterd.service.d/override.conf [Service] Restart=always RestartSec=3
Updated by Eric Helms almost 7 years ago
- Status changed from New to Needs design
Updated by Eric Helms almost 7 years ago
- Status changed from Needs design to Resolved
- Translation missing: en.field_release set to 166
This issue has been reported to the Qpid team and resolved by them. The fix will go out in the next router release so I am closing this issue out.
Updated by Klaas D almost 7 years ago
Upstream link https://issues.apache.org/jira/browse/DISPATCH-895