Project

General

Profile

Bug #16021

inotify queue overflow halts ISC lease file monitoring

Added by Dominic Cleal about 3 years ago. Updated about 1 year ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
DHCP
Target version:
Team Backlog:
Fixed in Releases:
Found in Releases:

Description

When using ISC lease file monitoring (nightly), the inotify event queue can overflow with many events if they're not processed quickly enough:

E, [2016-08-09T09:39:18.774323 #18572] ERROR -- : Error occured when monitoring /tmp/dhcp.leases
D, [2016-08-09T09:39:18.774363 #18572] DEBUG -- : inotify event queue has overflowed. (INotify::QueueOverflowError)
/home/dcleal/.rvm/gems/ruby-2.3.0@proxy/gems/rb-inotify-0.9.7/lib/rb-inotify/event.rb:120:in `initialize'
/home/dcleal/.rvm/gems/ruby-2.3.0@proxy/gems/rb-inotify-0.9.7/lib/rb-inotify/event.rb:100:in `new'
/home/dcleal/.rvm/gems/ruby-2.3.0@proxy/gems/rb-inotify-0.9.7/lib/rb-inotify/event.rb:100:in `consume'
/home/dcleal/.rvm/gems/ruby-2.3.0@proxy/gems/rb-inotify-0.9.7/lib/rb-inotify/notifier.rb:284:in `read_events'
/home/dcleal/.rvm/gems/ruby-2.3.0@proxy/gems/rb-inotify-0.9.7/lib/rb-inotify/notifier.rb:238:in `process'
/home/dcleal/.rvm/gems/ruby-2.3.0@proxy/gems/rb-inotify-0.9.7/lib/rb-inotify/notifier.rb:221:in `run'
/home/dcleal/code/foreman/smart-proxy/modules/dhcp_isc/inotify_leases_file_observer.rb:30:in `monitor_leases'
/home/dcleal/code/foreman/smart-proxy/modules/dhcp_isc/inotify_leases_file_observer.rb:37:in `block in start'

Mine's in a directory with lots of other noise (/tmp), but it seems possible to me that if re-parsing the file is slow then a limit could be hit.

The observer service should catch the exception and reinitialise the inotify tracking and re-parse the file.


Related issues

Related to Smart Proxy - Bug #2687: Performance issues with large ISC dataset (DHCP smart proxy)Closed2013-06-20
Related to Smart Proxy - Refactor #16140: Register ISC inotify leases watcher on leases file only, not directoryNew2016-08-17

Associated revisions

Revision 2e0765fe (diff)
Added by Dominic Cleal about 3 years ago

fixes #16021 - restart inotify monitoring on queue overflow

inotify watches are now more specific to only monitor for modifications
to the lease file and incoming file moves in the directory, reducing
the spurious wakeups from accessing other files in the same dir.

Revision c4314a8b (diff)
Added by Dominic Cleal about 3 years ago

refs #16021 - monitor modifications on entire directory

inotify file watches are based on the inode, so after the leases file
is recreated the watch no longer fires. Later versions of rb-inotify
will properly allow for the watch to be re-registered after the file is
recreated (issue #16140).

History

#1 Updated by Dominic Cleal about 3 years ago

  • Related to Bug #2687: Performance issues with large ISC dataset (DHCP smart proxy) added

#2 Updated by The Foreman Bot about 3 years ago

  • Status changed from New to Ready For Testing
  • Assignee set to Dominic Cleal
  • Pull request https://github.com/theforeman/smart-proxy/pull/455 added

#3 Updated by Dominic Cleal about 3 years ago

  • Related to Refactor #16140: Register ISC inotify leases watcher on leases file only, not directory added

#4 Updated by The Foreman Bot about 3 years ago

  • Pull request https://github.com/theforeman/smart-proxy/pull/456 added

#5 Updated by Dominic Cleal about 3 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100

#6 Updated by Dominic Cleal about 3 years ago

  • Legacy Backlogs Release (now unused) set to 160

Also available in: Atom PDF