Some initial thoughts on this...

It'd be nice to have something like a plugin template available so that all the plugins use the same test framework and have some default tasks like being able to generate the xmlunit reports that Jenkins can consume. For now, all the plugins we want to test will need to have ci_reporter in the Gemfile and and require 'ci/reporter/rake/minitest' in their Rakefile.

Also, as for the job itself - we'll have to clone core into the workspace before running the tests?

I agree on the template - we do have a plugin template, so I'll send an update with the changes I made to foreman_default_hostgroup once #3293 is merged.