Foreman 3.9.1 deployed on EL8.

provisioning virtual machines using the libvirt compute resource to a KVM host running EL9.3 successfully provisions the guest, but fails to support remote virtual console connection.
Launching the virtual console during or post provision generates the on screen error message.

"Failed to set console: Call to virDomainUpdateDeviceFlags failed: internal error: unable to execute QEMU command 'set_password': Could not set password"

the EL9.3 KVM host is running the following component versions

libvirt-libs-9.5.0-7.2.el9_3.x86_64
libvirt-client-9.5.0-7.2.el9_3.x86_64
ibvirt-daemon-9.5.0-7.2.el9_3.x86_64
libvirt-9.5.0-7.2.el9_3.x86_64
Foreman logging supports the error message with the following output on a provisioned guest referenced as "will' for a hostname

2024-04-15T10:53:19 [I|app|c0e262a8] Started GET "/hosts/will.no-dns.co.uk/console" for 10.11.216.109 at 2024-04-15 10:53:19 +0100
2024-04-15T10:53:19 [I|app|c0e262a8] Processing by HostsController#console as HTML
2024-04-15T10:53:19 [I|app|c0e262a8] Parameters: {"id"=>"will.no-dns.co.uk"}
2024-04-15T10:53:20 [W|app|c0e262a8] Failed to set console
2024-04-15T10:53:20 [I|app|c0e262a8] Backtrace for 'Failed to set console' error (Libvirt::Error): Call to virDomainUpdateDeviceFlags failed: internal error: unable to execute QEMU command 'set_password': Could not set password
/usr/share/gems/gems/fog-libvirt-0.12.0/lib/fog/libvirt/requests/compute/update_display.rb:22:in `update_device'
/usr/share/gems/gems/fog-libvirt-0.12.0/lib/fog/libvirt/requests/compute/update_display.rb:22:in `update_display'
/usr/share/gems/gems/fog-libvirt-0.12.0/lib/fog/libvirt/models/compute/server.rb:230:in `update_display'
/usr/share/foreman/app/models/compute_resources/foreman/model/libvirt.rb:184:in `console'
/usr/share/foreman/app/controllers/hosts_controller.rb:357:in `console'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:228:in `process_action'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/rendering.rb:30:in `process_action'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
/usr/share/gems/gems/audited-5.4.2/lib/audited/sweeper.rb:16:in `around'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
/usr/share/gems/gems/audited-5.4.2/lib/audited/sweeper.rb:16:in `around'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:137:in `run_callbacks'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:41:in `process_action'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/rescue.rb:22:in `process_action'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `block in instrument'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `instrument'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:165:in `process'
/usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/rendering.rb:39:in `process'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal.rb:190:in `dispatch'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal.rb:254:in `dispatch'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:33:in `serve'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:50:in `block in serve'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:32:in `each'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:32:in `serve'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:842:in `call'
/usr/share/gems/gems/apipie-dsl-2.6.1/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
/usr/share/gems/gems/apipie-rails-1.2.3/lib/apipie/static_dispatcher.rb:68:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
/usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
/usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
/usr/share/gems/gems/apipie-rails-1.2.3/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/gems/gems/rack-2.2.8/lib/rack/tempfile_reaper.rb:15:in `call'
/usr/share/gems/gems/rack-2.2.8/lib/rack/etag.rb:27:in `call'
/usr/share/gems/gems/rack-2.2.8/lib/rack/conditional_get.rb:27:in `call'
/usr/share/gems/gems/rack-2.2.8/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
/usr/share/gems/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:266:in `context'
/usr/share/gems/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:260:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/cookies.rb:697:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:98:in `run_callbacks'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/usr/share/gems/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:37:in `call_app'
/usr/share/gems/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:28:in `call'
/usr/share/gems/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/usr/share/gems/gems/request_store-1.5.1/lib/request_store/middleware.rb:19:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/request_id.rb:26:in `call'
/usr/share/gems/gems/rack-2.2.8/lib/rack/method_override.rb:24:in `call'
/usr/share/gems/gems/rack-2.2.8/lib/rack/runtime.rb:22:in `call'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/executor.rb:14:in `call'
/usr/share/gems/gems/rack-2.2.8/lib/rack/sendfile.rb:110:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/ssl.rb:77:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/host_authorization.rb:142:in `call'
/usr/share/gems/gems/secure_headers-6.5.0/lib/secure_headers/middleware.rb:11:in `call'
/usr/share/gems/gems/railties-6.1.7.6/lib/rails/engine.rb:539:in `call'
/usr/share/gems/gems/railties-6.1.7.6/lib/rails/railtie.rb:207:in `public_send'
/usr/share/gems/gems/railties-6.1.7.6/lib/rails/railtie.rb:207:in `method_missing'
/usr/share/gems/gems/rack-2.2.8/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.8/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.8/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/puma-6.4.0/lib/puma/configuration.rb:272:in `call'
/usr/share/gems/gems/puma-6.4.0/lib/puma/request.rb:100:in `block in handle_request'
/usr/share/gems/gems/puma-6.4.0/lib/puma/thread_pool.rb:378:in `with_force_shutdown'
/usr/share/gems/gems/puma-6.4.0/lib/puma/request.rb:99:in `handle_request'
/usr/share/gems/gems/puma-6.4.0/lib/puma/server.rb:443:in `process_client'
/usr/share/gems/gems/puma-6.4.0/lib/puma/server.rb:241:in `block in run'
/usr/share/gems/gems/puma-6.4.0/lib/puma/thread_pool.rb:155:in `block in spawn_thread'
/usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2024-04-15T10:53:20 [E|app|c0e262a8] Failed to save:
2024-04-15T10:53:20 [I|app|c0e262a8] Redirected to https://jarvis.no-dns.co.uk/new/hosts/will.no-dns.co.uk
2024-04-15T10:53:20 [I|app|c0e262a8] Completed 302 Found in 843ms (ActiveRecord: 170.9ms | Allocations: 11671)
2024-04-15T10:53:20 [I|app|7ceb1129] Started GET "/new/hosts/will.no-dns.co.uk" for 10.11.216.109 at 2024-04-15 10:53:20 +0100
2024-04-15T10:53:20 [I|app|7ceb1129] Processing by ReactController#index as HTML
2024-04-15T10:53:20 [I|app|7ceb1129] Parameters: {"id"=>"will.no-dns.co.uk"}
2024-04-15T10:53:20 [I|app|47bf21cb] Started GET "/notification_recipients" for 10.11.216.109 at 2024-04-15 10:53:20 +0100
2024-04-15T10:53:20 [I|app|7ceb1129] Rendered react/index.html.erb within layouts/react_application (Duration: 0.9ms | Allocations: 134)
2024-04-15T10:53:20 [I|app|47bf21cb] Processing by NotificationRecipientsController#index as JSON
2024-04-15T10:53:20 [I|app|47bf21cb] Completed 200 OK in 106ms (Views: 0.2ms | ActiveRecord: 88.1ms | Allocations: 4109)
2024-04-15T10:53:20 [I|app|7ceb1129] Rendered layouts/base.html.erb (Duration: 245.3ms | Allocations: 43497)
2024-04-15T10:53:20 [I|app|7ceb1129] Rendered layout layouts/react_application.html.erb (Duration: 248.2ms | Allocations: 44041)
2024-04-15T10:53:20 [I|app|7ceb1129] Completed 200 OK in 315ms (Views: 246.7ms | ActiveRecord: 17.3ms | Allocations: 51221)
2024-04-15T10:53:21 [I|app|2d8c6e13] Started GET "/notification_recipients" for 10.11.216.109 at 2024-04-15 10:53:21 +0100
2024-04-15T10:53:21 [I|app|87bb4e08] Started GET "/api/v2/remote_execution_features" for 10.11.216.109 at 2024-04-15 10:53:21 +0100
2024-04-15T10:53:21 [I|app|475beac8] Started GET "/hosts/will.no-dns.co.uk/statuses" for 10.11.216.109 at 2024-04-15 10:53:21 +0100
2024-04-15T10:53:21 [I|app|92f7ddb6] Started GET "/api/hosts/will.no-dns.co.uk?show_hidden_parameters=true" for 10.11.216.109 at 2024-04-15 10:53:21 +0100
2024-04-15T10:53:21 [I|app|2d8c6e13] Processing by NotificationRecipientsController#index as JSON
2024-04-15T10:53:21 [I|app|d53ad1be] Started GET "/api/audits?search=host%3Dwill.no-dns.co.uk&per_page=3" for 10.11.216.109 at 2024-04-15 10:53:21 +0100
2024-04-15T10:53:21 [I|app|dbbfc1ac] Started GET "/api/hosts/will.no-dns.co.uk/power?timeout=30" for 10.11.216.109 at 2024-04-15 10:53:21 +0100
2024-04-15T10:53:21 [I|app|475beac8] Processing by HostsController#statuses as JSON
2024-04-15T10:53:21 [I|app|475beac8] Parameters: {"id"=>"will.no-dns.co.uk"}
2024-04-15T10:53:21 [I|app|87bb4e08] Processing by Api::V2::RemoteExecutionFeaturesController#index as JSON
2024-04-15T10:53:21 [I|app|87bb4e08] Parameters: {"apiv"=>"v2"}
2024-04-15T10:53:21 [I|app|d53ad1be] Processing by Api::V2::AuditsController#index as JSON
2024-04-15T10:53:21 [I|app|d53ad1be] Parameters: {"search"=>"host=will.no-dns.co.uk", "per_page"=>"3", "apiv"=>"v2"}
2024-04-15T10:53:21 [I|app|92f7ddb6] Processing by Api::V2::HostsController#show as JSON
2024-04-15T10:53:21 [I|app|92f7ddb6] Parameters: {"show_hidden_parameters"=>"true", "apiv"=>"v2", "id"=>"will.no-dns.co.uk"}
2024-04-15T10:53:21 [I|app|dbbfc1ac] Processing by Api::V2::HostsController#power_status as JSON
2024-04-15T10:53:21 [I|app|dbbfc1ac] Parameters: {"timeout"=>"30", "apiv"=>"v2", "id"=>"will.no-dns.co.uk"}
2024-04-15T10:53:21 [I|app|2d8c6e13] Completed 200 OK in 96ms (Views: 0.2ms | ActiveRecord: 76.2ms | Allocations: 4114)
2024-04-15T10:53:21 [I|app|d53ad1be] Rendered api/v2/audits/index.json.rabl within api/v2/layouts/index_layout (Duration: 185.1ms | Allocations: 19039)
2024-04-15T10:53:21 [I|app|d53ad1be] Rendered layout api/v2/layouts/index_layout.json.erb (Duration: 211.5ms | Allocations: 26100)
2024-04-15T10:53:21 [I|app|d53ad1be] Completed 200 OK in 332ms (Views: 156.1ms | ActiveRecord: 143.3ms | Allocations: 33935)
2024-04-15T10:53:21 [I|app|475beac8] Completed 200 OK in 349ms (Views: 0.6ms | ActiveRecord: 202.0ms | Allocations: 31962)
2024-04-15T10:53:21 [I|app|87bb4e08] Rendered /usr/share/gems/gems/foreman_remote_execution-12.0.5/app/views/api/v2/remote_execution_features/index.json.rabl within api/v2/layouts/index_layout (Duration: 261.3ms | Allocations: 23396)
2024-04-15T10:53:21 [I|app|87bb4e08] Rendered layout api/v2/layouts/index_layout.json.erb (Duration: 269.3ms | Allocations: 29380)
2024-04-15T10:53:21 [I|app|87bb4e08] Completed 200 OK in 393ms (Views: 125.6ms | ActiveRecord: 237.4ms | Allocations: 36172)
2024-04-15T10:53:21 [I|app|dbbfc1ac] Completed 200 OK in 822ms (Views: 0.4ms | ActiveRecord: 99.1ms | Allocations: 13833)
2024-04-15T10:53:22 [I|app|92f7ddb6] Rendered api/v2/hosts/show.json.rabl (Duration: 889.5ms | Allocations: 127613)
2024-04-15T10:53:22 [I|app|92f7ddb6] Completed 200 OK in 1087ms (Views: 465.1ms | ActiveRecord: 548.3ms | Allocations: 141855)
2024-04-15T10:53:22 [I|app|d4261536] Started GET "/api/v2/hosts/170/available_remote_execution_features" for 10.11.216.109 at 2024-04-15 10:53:22 +0100
2024-04-15T10:53:22 [I|app|638825d1] Started GET "/job_invocations/preview_job_invocations_per_host?host_id=170&status=failed+or+status%3D+succeeded&limit=3" for 10.11.216.109 at 2024-04-15 10:53:22 +0100
2024-04-15T10:53:22 [I|app|d4261536] Processing by Api::V2::RemoteExecutionFeaturesController#available_remote_execution_features as JSON
2024-04-15T10:53:22 [I|app|d4261536] Parameters: {"apiv"=>"v2", "id"=>"170"}
2024-04-15T10:53:22 [I|app|638825d1] Processing by JobInvocationsController#preview_job_invocations_per_host as JSON
2024-04-15T10:53:22 [I|app|638825d1] Parameters: {"host_id"=>"170", "status"=>"failed or status= succeeded", "limit"=>"3"}
2024-04-15T10:53:22 [I|app|638825d1] Completed 200 OK in 100ms (Views: 0.3ms | ActiveRecord: 63.9ms | Allocations: 8416)
2024-04-15T10:53:22 [I|app|d4261536] Rendered /usr/share/gems/gems/foreman_remote_execution-12.0.5/app/views/api/v2/remote_execution_features/available_remote_execution_features.json.rabl (Duration: 156.3ms | Allocations: 24716)

on the KVM host, the only output referencing the guest is a deprecated video card driver, which is useful to know, but I believe not part of this bug, foreman needs to be changed to use a different default virtual video card when provisioning guests.

2024-04-02T16:15:50.209668Z qemu-kvm: -device {"driver":"cirrus-vga","id":"video0","bus":"pcie.0","addr":"0x1"}: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead