candlepin event listener does not release messages after error
Description of problem:
The candlepin event listener will call 'acknowledge' on messages that it processes, but does not call 'release' or 'reject' on messages that it is unable to process.
This can cause some messages to become stuck in the katello_event queue, since they are being held but will never be released.
The best behavior may be to just log the message and then reject it, so potentially bad messages do not get reprocessed over and over.
Version-Release number of selected component (if applicable): 6.2.8
How reproducible: not sure how to repro yet
note: I put this under the 'hosts' component but I'm not sure if that is the best place for this to live. It is related to the ListenOnCandlepinEvents task.
Fixes #20532 - acknowledge all candlepin messages
This solves two different issues, after every 100 messages
recieved that match the subjects we look for, we would
sleep for a second instead of
a) acknowledging the message
b) performing some action on that messsage
This makes two changes to ensure that all messages are
acknowledged and the sleep isn't performed until after
messages are acted upon.
#2 Updated by Pavel Moravec almost 5 years ago
Nondeterministc reproducer (not real user story, but worth for developers/testers):
1) stop foreman-tasks service (to populate katello_event_queue a bit)
2) generate several hundreds candlepin events (i.e. (un)register a Content Host with an activation key, or remove all and attach back a subscription pool to another Host) - do that in a loop until katello_event_queue has few hundreds of messages
3) start foreman-tasks service - leave step 2) running (at least I did so, it might but not need to be important)
4) Once ListenOnCandlepinEvents task consumes the backlog, check if katello_event_queue has zero queue depth (see #c7)
5) if there some messages constantly acquired but not acknowledged, you got it. Otherwise, goto step 1).