Project

General

Profile

Actions

Bug #9523

closed

Puppet master crashes on AVC when blocking setattr after logrotate

Added by Gerwin Krist about 9 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
High
Assignee:
Category:
General Foreman
Target version:
Difficulty:
easy
Triaged:
Fixed in Releases:
Found in Releases:

Description

Problem description
Once a week our puppet master stops working and the puppet clients are spitting out errors. We are seeing this on a freshly installed 1.7.2 on RHEL 7 (Selinux enabled).

Observations

It seems the problems starts when the weekly logrotate is done:

Feb 23 03:22:19 i-foreman puppet-master[92076]: failed to set mode 644 on /var/log/puppet/http.log: Permission denied - /var/log/puppet/http.log
Feb 23 03:22:19 i-foreman puppet-master[92076]: (/File[/var/log/puppet/http.log]/mode) change from 0644 to 0640 failed: failed to set mode 644 on /var/log/puppet/http.log: Permission denied - /var/log/puppet/http.log
Feb 23 03:22:19 i-foreman puppet-master[92076]: Could not prepare for execution: Got 1 failure(s) while initializing: File[/var/log/puppet/http.log]: change from 0644 to 0640 failed: failed to set mode 644 on /var/log/puppet/http.log: Pe
rmission denied - /var/log/puppet/http.log

I also get an AVC at the same time:

type=AVC msg=audit(1424658139.219:23310): avc:  denied  { setattr } for  pid=92076 comm="ruby" name="http.log" dev="vda2" ino=131193 scontext=system_u:system_r:passenger_t:s0 tcontext=system_u:object_r:puppet_log_t:s0 tclass=file

So my guess it's a bug in the selinux policy.

Agent log output

Feb 23 10:09:04 d-hpwtest start-puppet-agent: /usr/share/ruby/vendor_ruby/puppet/agent.rb:87:in `exit': no implicit conversion from nil to integer (TypeError)                                                                               
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/agent.rb:87:in `block in run_in_fork'                                                                                                                  
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/agent.rb:84:in `fork'                                                                                                                                  
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/agent.rb:84:in `run_in_fork'                                                                                                                           
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/agent.rb:43:in `block in run'                                                                                                                          
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/application.rb:179:in `call'                                                                                                                           
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/application.rb:179:in `controlled_run'                                                                                                                 
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/agent.rb:41:in `run'                                                                                                                                   
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/daemon.rb:163:in `block in run_event_loop'                                                                                                             
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/scheduler/job.rb:49:in `call'                                                                                                                          
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/scheduler/job.rb:49:in `run'                                                                                                                           
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/scheduler/scheduler.rb:39:in `block in run_ready'                                                                                                      
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/scheduler/scheduler.rb:34:in `each'                                                                                                                    
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/scheduler/scheduler.rb:34:in `run_ready'                                                                                                               
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/scheduler/scheduler.rb:11:in `run_loop'                                                                                                                
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/daemon.rb:179:in `run_event_loop'                                                                                                                      
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/daemon.rb:142:in `start'                                                                                                                               
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/application/agent.rb:377:in `main'                                                                                                                     
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/application/agent.rb:323:in `run_command'                                                                                                              
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/application.rb:371:in `block (2 levels) in run'                                                                                                        
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/application.rb:477:in `plugin_hook'                                                                                                                    
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/application.rb:371:in `block in run'                                                                                                                   
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/util.rb:479:in `exit_on_fail'                                                                                                                          
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/application.rb:371:in `run'                                                                                                                            
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'                                                                                                                      
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:91:in `execute'                                                                                                                   
Feb 23 10:09:04 d-hpwtest start-puppet-agent: from /usr/bin/puppet:8:in `<main>'
Actions #1

Updated by Dominic Cleal about 9 years ago

  • Project changed from Foreman to SELinux
  • Subject changed from Puppet master stops working every week to Puppet master crashes on AVC when blocking setattr after logrotate
  • Category changed from 56 to General Foreman

Seems there's probably a bug in the Puppet logrotate configuration too if it's creating files with mode 0644 instead of 0640.

Actions #2

Updated by Gerwin Krist about 9 years ago

FYI:

/var/log/puppet/*log {
  missingok
  notifempty
  create 0644 puppet puppet
  sharedscripts
  postrotate
    pkill -USR2 -u puppet -f 'puppet master' || true
    [ -e /etc/init.d/puppet ] && /etc/init.d/puppet reload > /dev/null 2>&1 || true
  endscript
}

Actions #3

Updated by Dominic Cleal about 9 years ago

Ah yes, it'd be worth raising that small discrepancy over at https://tickets.puppetlabs.com/browse/PUP

https://github.com/puppetlabs/puppet/blob/3.7.4/lib/puppet/defaults.rb#L405 shows the internal configuration that Puppet's trying to assert.

Actions #4

Updated by Gerwin Krist about 9 years ago

@Dominic Cleal
Was checking if I can write a patch for the policy. But I don't know if there is a policy for selinux policies :-) I see 2 options:
  1. allow passenger_t access to puppet_log_t
  2. change /var/log/puppet/(http.log*) to passenger_log_t

Please let me know if you other ideas and I will check if I can fix it :-)

Actions #5

Updated by Dominic Cleal about 9 years ago

I think option (1), allowing access to puppet_log_t would be right. You'd need to add setattr here:

https://github.com/theforeman/foreman-selinux/blob/develop/foreman.te#L264

Actions #6

Updated by Gerwin Krist about 9 years ago

Made a patch and tested (forced logrotate) it. But I want to wait for the regular logrotation (this monday) and see if that works too. Then will do a pull request.

Actions #7

Updated by The Foreman Bot about 9 years ago

  • Status changed from New to Ready For Testing
  • Pull request https://github.com/theforeman/foreman-selinux/pull/46 added
  • Pull request deleted ()
Actions #8

Updated by Gerwin Krist about 9 years ago

No problems either with the normale logrotate. Did a pull request

Actions #9

Updated by Anonymous about 9 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100
Actions #10

Updated by Dominic Cleal about 9 years ago

  • Assignee set to Gerwin Krist
  • translation missing: en.field_release set to 28
Actions

Also available in: Atom PDF