Bug #12219
closed
Test failures on Ruby 2.2
Description
Ruby 2.2 support in #11016 is incomplete, it still has test failures:
On PostgreSQL it's failing to initialise the pg gem:
08:28:36 + bundle exec rake db:drop db:create db:migrate 08:28:47 Couldn't drop test_develop_pr_core_ruby22-1-dev : #<LoadError: Please install the postgresql adapter: `gem install activerecord-postgresql-adapter` (incompatible library version - /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/pg-0.18.3/lib/pg_ext.so)> 08:28:47 Couldn't drop test_develop_pr_core_ruby22-1-test : #<LoadError: Please install the postgresql adapter: `gem install activerecord-postgresql-adapter` (incompatible library version - /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/pg-0.18.3/lib/pg_ext.so)> 08:28:47 rake aborted! 08:28:47 LoadError: Please install the postgresql adapter: `gem install activerecord-postgresql-adapter` (incompatible library version - /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/pg-0.18.3/lib/pg_ext.so) 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:251:in `require' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:251:in `block in require' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:236:in `load_dependency' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:251:in `require' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/pg-0.18.3/lib/pg.rb:4:in `<top (required)>' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:251:in `require' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:251:in `block in require' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:236:in `load_dependency' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:251:in `require' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activerecord-3.2.22/lib/active_record/connection_adapters/postgresql_adapter.rb:8:in `<top (required)>' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:251:in `require' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:251:in `block in require' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:236:in `load_dependency' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:251:in `require' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activerecord-3.2.22/lib/active_record/connection_adapters/abstract/connection_specification.rb:50:in `resolve_hash_connection' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activerecord-3.2.22/lib/active_record/connection_adapters/abstract/connection_specification.rb:29:in `spec' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activerecord-3.2.22/lib/active_record/connection_adapters/abstract/connection_specification.rb:130:in `establish_connection' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activerecord-3.2.22/lib/active_record/railties/databases.rake:101:in `create_database' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activerecord-3.2.22/lib/active_record/railties/databases.rake:62:in `block (3 levels) in <top (required)>' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activerecord-3.2.22/lib/active_record/railties/databases.rake:62:in `each' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activerecord-3.2.22/lib/active_record/railties/databases.rake:62:in `block (2 levels) in <top (required)>' 08:28:47 LoadError: incompatible library version - /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/pg-0.18.3/lib/pg_ext.so 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:251:in `require' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:251:in `block in require' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:236:in `load_dependency' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:251:in `require' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/pg-0.18.3/lib/pg.rb:4:in `<top (required)>' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:251:in `require' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:251:in `block in require' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:236:in `load_dependency' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:251:in `require' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activerecord-3.2.22/lib/active_record/connection_adapters/postgresql_adapter.rb:8:in `<top (required)>' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:251:in `require' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:251:in `block in require' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:236:in `load_dependency' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activesupport-3.2.22/lib/active_support/dependencies.rb:251:in `require' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activerecord-3.2.22/lib/active_record/connection_adapters/abstract/connection_specification.rb:50:in `resolve_hash_connection' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activerecord-3.2.22/lib/active_record/connection_adapters/abstract/connection_specification.rb:29:in `spec' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activerecord-3.2.22/lib/active_record/connection_adapters/abstract/connection_specification.rb:130:in `establish_connection' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activerecord-3.2.22/lib/active_record/railties/databases.rake:101:in `create_database' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activerecord-3.2.22/lib/active_record/railties/databases.rake:62:in `block (3 levels) in <top (required)>' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activerecord-3.2.22/lib/active_record/railties/databases.rake:62:in `each' 08:28:47 /usr/local/rvm/gems/ruby-2.2.3@test_develop_pr_core_ruby22-1/gems/activerecord-3.2.22/lib/active_record/railties/databases.rake:62:in `block (2 levels) in <top (required)>' 08:28:47 Tasks: TOP => db:create 08:28:47 (See full trace by running task with --trace)
(http://ci.theforeman.org/job/test_develop_pr_core_ruby22/7/database=postgresql,ruby=2.2/console)
On sqlite3 and mysql, it actually runs the test suite and fails with:
08:29:55 =============================================================================== 08:29:55 Failure: 08:29:55 <"localhost"> expected but was 08:29:55 <"foreman.some.host.fqdn">. 08:29:55 test_0001_foreman server header is set(ApplicationMailerTest) 08:29:55 test/unit/application_mailer_test.rb:14:in `block in <class:ApplicationMailerTest>' 08:29:55 =============================================================================== 08:30:23 =============================================================================== 08:30:23 Error: test_0014_add compute resource(ComputeResourceTest): NameError: uninitialized class variable @@current in MiniTest::Unit::TestCase 08:30:23 test/unit/compute_resource_test.rb:109:in `block in <class:ComputeResourceTest>' 08:30:23 =============================================================================== 08:33:33 =============================================================================== 08:33:33 Error: test_0001_be tested(MenuItemTest::MenuItem#caption): NoMethodError: undefined method `skip' for #<#<Class:0x0000000c8fccb0>:0x0000000ad272d8> 08:35:30 =============================================================================== 08:37:17 E 08:37:17 ===============================================================================
(http://ci.theforeman.org/job/test_develop_pr_core_ruby22/7/database=sqlite3,ruby=2.2/console, http://ci.theforeman.org/job/test_develop_pr_core_ruby22/7/database=mysql,ruby=2.2/console)
The mysql test failed with a SIGKILL due to the OOM killer, this may require some consideration given to our Jenkins infrastructure. The RSS of the Ruby process was reported to be 2.54GB.
Updated by Dominic Cleal over 8 years ago
- Related to Bug #11016: missing gem in test.rb under ruby 2.2.0 added
Updated by Dominic Cleal over 8 years ago
- Blocks Feature #7228: Rebuild packages under ror41/ruby22 SCLs added
Updated by Anonymous over 8 years ago
- Status changed from New to Assigned
- Assignee set to Anonymous
Updated by Eric Helms over 8 years ago
During some local testing I noticed that the postgres related failure only occurred with 2.2.3. Using 2.2.1 (the default I got when I ran 'rvm install 2.2') and 2.2.2 I had no issues. Given that the SCL's current versions are 2.2.0 and 2.2.2, would the fact it runs fine on 2.2.1 and 2.2.2 be enough to turn on 2.2 support? (Assuming of course the sqlite and mysql issues were fixed).
Updated by Dominic Cleal over 8 years ago
We should use the latest 2.2.x I think, not just what happens to be in the SCL. Other platforms (such as Debian 9) may use 2.2.3, as might developers.
The test failures seen on sqlite and MySQL seem to be fixed on the Rails 4 branch under ticket #7230.
Updated by Eric Helms over 8 years ago
Some updated findings, locally I have found that the 0.15.1 version of the pg gem does not lead to this error, but every version above that does. Currently, we package and distribute 0.12.2 version of the pg gem via RPM and the bundler asset is set to '~> 0.11'. One solution could be to set ourselves to the 0.15.1 version of the pg gem and package that. I have not been able to determine from diffing why 0.16.0 breaks but 0.15.1 works.
Updated by Dominic Cleal over 8 years ago
Note that the version in the bundler.d file is designed to match the AR adapter, which in 4.1 is ~> 0.11 still. 0.15 is required for Ruby 2.2 compatibility (bug) while Rails master has moved up to ~> 0.18. 4.1 ought to work fine with 0.18 still.
What's the underlying error? Presumably it's just catching a LoadError and reporting a generic message.
Updated by Eric Helms over 8 years ago
Some further information into this, based on Daniel's inability to re-produce on a new setup I spun up new versions of centos6-devel and centos7-devel from katello-deploy. I then installed 2.2.3 and 2.2.4 (released Dec 16th) via rvm install, bundle installed and ran the command. There was no error in any of those cases. The only time I am able to reproduce this is from an existing, older rvm setup on a devel box of mine. The version of RVM is the same so this may be related to how RVM is built/installed and newer libraries? I have zero leads as to the root cause but figured I'd share that bit of information.
Updated by Eric Helms over 8 years ago
Ran the tests on sqlite3 + 2.2.4 with latest (i.e. post Rails 4 merge) and everything passed for me. Which would in theory leave the lingering postgresql issue that seems to be present only on older RVM installations which would largely affect Jenkins. I am still working to see if I can "fix" an existing RVM installation.
Updated by Dominic Cleal over 8 years ago
I managed to reproduce myself on one of the slaves that reported it, looking into it more now. I get the same error that we saw on Rails 3.2 when requiring the gem directly, so the different error on Rails is probably just a rescue.
[dcleal@slave02 foreman]$ rails console
/usr/local/rvm/gems/ruby-2.2.3@dcleal/gems/activesupport-4.1.5/lib/active_support/dependencies.rb:247:in `require': Could not load 'active_record/connection_adapters/postgresql_adapter'. Make sure that the adapter in config/database.yml is valid. If you use an adapter ot
her than 'mysql', 'mysql2', 'postgresql' or 'sqlite3' add the necessary adapter gem to the Gemfile. (LoadError)
from /usr/local/rvm/gems/ruby-2.2.3@dcleal/gems/activesupport-4.1.5/lib/active_support/dependencies.rb:247:in `block in require'
from /usr/local/rvm/gems/ruby-2.2.3@dcleal/gems/activesupport-4.1.5/lib/active_support/dependencies.rb:232:in `load_dependency'
from /usr/local/rvm/gems/ruby-2.2.3@dcleal/gems/activesupport-4.1.5/lib/active_support/dependencies.rb:247:in `require'
from /usr/local/rvm/gems/ruby-2.2.3@dcleal/gems/pg-0.18.4/lib/pg.rb:4:in `<top (required)>'
[dcleal@slave02 foreman]$ bundle exec irb
2.2.3 :001 > require 'pg'
LoadError: incompatible library version - /usr/local/rvm/gems/ruby-2.2.3@dcleal/gems/pg-0.18.4/lib/pg_ext.so
from /usr/local/rvm/gems/ruby-2.2.3@dcleal/gems/pg-0.18.4/lib/pg.rb:4:in `require'
from /usr/local/rvm/gems/ruby-2.2.3@dcleal/gems/pg-0.18.4/lib/pg.rb:4:in `<top (required)>'
from (irb):1:in `require'
from (irb):1
from /usr/local/rvm/rubies/ruby-2.2.3/bin/irb:11:in `<main>'
Updated by Dominic Cleal over 8 years ago
The pg_ext.so library is being linked to the wrong Ruby, even when using rvm. It's being linked to the system Ruby, which isn't going to be ABI compatible with rvm's.
Here's ldd from a git checkout of the pg gem, compiled under rvm's 2.2.3 but is linked to EL6's Ruby 1.8:
[dcleal@slave02 ruby-pg]$ ldd ./tmp/x86_64-linux/pg_ext/2.2.3/pg_ext.so | grep ruby
libruby.so.1.8 => /usr/lib64/libruby.so.1.8 (0x00007fc20c1ab000)
A bisect shows this is ever since https://github.com/ged/ruby-pg/commit/a6ef4bd3a5ba8dce79331f0678dee315813ac754.
rvm on our slaves is currently version 1.26.11. I don't see this on my local Fedora system under rvm 1.25.15.
It appears that the tfm-rubygem-pg RPM I built for EL6 is linked against libruby2.2, so it may be a combination of rvm and pg.
Updated by Dominic Cleal over 8 years ago
Something, probably in mkmf, changed in Ruby 2.2 to cause lib paths (used for linking) to be prepended instead of appended when building native extensions. Comparing 2.0 and 2.2, I see the rpath of the generated pg_ext.so is:
- /usr/local/rvm/rubies/ruby-2.0.0-p643/lib:/usr/lib64
- /usr/lib64:/usr/local/rvm/rubies/ruby-2.2.3/lib
The /usr/lib64 is added because it's the path of the PostgreSQL libraries. When it's prepended though, any libruby in /usr/lib64 is linked instead of the RVM version - this causes the linking against the wrong Ruby version and later, a runtime failure.
This happens on our slaves but not on a clean EL6 installation because we have ruby-devel installed (this package might need to be installed to reproduce the bug).
I'd suggest for now that we pin the pg gem to < 0.16 and then this needs either changing in pg (which moved to using mkmf instead of appending manually to the link path in the breaking commit) or mkmf/Ruby core.
Updated by Eric Helms over 8 years ago
+1 to your assessment - uninstalling ruby-devel on my "broken" box and installing pg afterward provided the correct linking.
Updated by The Foreman Bot over 8 years ago
- Status changed from Assigned to Ready For Testing
- Pull request https://github.com/theforeman/foreman/pull/2990 added
Updated by Dominic Cleal over 8 years ago
- Assignee changed from Anonymous to Eric Helms
Updated by Eric Helms over 8 years ago
- Status changed from Ready For Testing to Closed
- % Done changed from 0 to 100
Applied in changeset 7d54aa3288e394302b8d175ed2c2fbfa5e4a2e67.
Updated by Dominic Cleal over 8 years ago
- Blocked by Feature #7230: Upgrade Ruby on Rails to 4.1 added
Updated by Dominic Cleal over 7 years ago
- Related to Refactor #15753: Unpin pg gem added