Bug #38207
open
Foreman Proxy crashes with "malloc_consolidate(): unaligned fastbin chunk detected" on Ubuntu 22.04
Description
This crash seems to happen most often (but not always) when trying to provision a new host, but it also seems to happen randomly. There is also a warning in the logs about the FreeIPA realm plugin and xmlrpc that looks unrelated to me. This is from the foreman-proxy journal ( journalctl -efu foreman-proxy.service ):
Feb 11 18:05:10 si-foreman02-prod-vrz0506 systemd[1]: Starting Foreman Proxy... Feb 11 18:05:11 si-foreman02-prod-vrz0506 systemd[1]: Started Foreman Proxy. Feb 11 18:17:08 si-foreman02-prod-vrz0506 smart-proxy[2044335]: /usr/share/foreman-proxy/modules/realm_freeipa/provider.rb:46: warning: already initialized constant XMLRPC::Config::ENABLE_NIL_PARSER Feb 11 18:17:08 si-foreman02-prod-vrz0506 smart-proxy[2044335]: /usr/lib/ruby/vendor_ruby/xmlrpc/config.rb:22: warning: previous definition of ENABLE_NIL_PARSER was here Feb 11 18:54:00 si-foreman02-prod-vrz0506 smart-proxy[2044335]: malloc_consolidate(): unaligned fastbin chunk detected Feb 11 18:54:00 si-foreman02-prod-vrz0506 systemd[1]: foreman-proxy.service: Main process exited, code=dumped, status=6/ABRT Feb 11 18:54:00 si-foreman02-prod-vrz0506 systemd[1]: foreman-proxy.service: Failed with result 'core-dump'. Feb 11 18:54:00 si-foreman02-prod-vrz0506 systemd[1]: foreman-proxy.service: Consumed 1.325s CPU time.
The logs in /var/log/foreman-proxy/proxy.log don't show anything about the crash.
This is happening on the Foreman machine itself (i.e. it runs Foreman and Foreman-Proxy). These are the features enabled on that proxy instance: DNS, Realm, Discovery, TFTP, Logs, and HTTPBoot.
The crashes started with version 3.12.1 after we upgraded the machine from Ubuntu 20.04 to 22.04. We then upgraded to Foreman 3.13.0 in the hopes that this will contain a fix for the crash, but unfortunately it keeps happening. The above journal excerpt is from Foreman-Proxy 3.13.0, on Ubuntu 22.04.
As mentioned the crashes happen most often when trying to provision a new host; I am guessing two out of three times.
Please let me know what, if any, information I can provide and how to help debug this.
Updated by Andreas Ntaflos 9 days ago
It seems the cause is the ruby-gssapi Gem/package as provided by Ubuntu 22.04. Version 1.2.0-1.1, available in the Ubuntu repositories, apparently has a bug that causes this crash.
We packaged ruby-gssapi 1.3.1 for Ubuntu 22.04 ourselves and upgraded to that version, and now foreman-proxy no longer crashes when trying to provision hosts.
Updated by Andreas Ntaflos 9 days ago
Relevant issues: https://github.com/zenchild/gssapi/issues/12 (from 2012, but only fixed in 2020) and https://access.redhat.com/solutions/7018038