Log when enumerator has nothing to iterate #49
Conversation
| assert_logged(%([JobIteration::Iteration] Enumerator found nothing to iterate!)) do | ||
| work_one_job | ||
| end | ||
|
|
|
Thoughts on moving this one level up the stack and adding this logging to Iteration API so that we log when any enumerator is empty? |
@moechaieb this as low as you can get, right? What do you mean with the Iteration API? |
djmortonShopify
force-pushed
the
add_logging_when_nothing_to_iterate
branch
from
f4867e2
to
e2ceb42
Compare
|
The ActiveRecord case is the one that seems to come up most often, but I'll look at moving it up and see if it makes sense. |
|
@moechaieb Sorry... my brain blanked there... My test is for activerecord enumerators, but the actual code is in |
|
@moechaieb @djmortonShopify this change deal with all enumerators. We first |
djmortonShopify
force-pushed
the
add_logging_when_nothing_to_iterate
branch
from
e2ceb42
to
d2766e6
Compare
| @@ -128,6 +130,11 @@ def iterate_with_enumerator(enumerator, arguments) | |||
| return false | |||
| end | |||
|
|
|||
| logger.info( | |||
| "[JobIteration::Iteration] Enumerator found nothing to iterate! " \ | |||
| "(times_interrupted: #{times_interrupted}, cursor_position: #{cursor_position})" | |||
There was a problem hiding this comment.
Why not use tagged logging like we do everywhere? E.g. times_interrupted=#{times_interrupted} cursor_position=#{cursor_position}?
They're easier to parse and Splunk matches them by default.
|
The version you guys pushed should solve the immediate problem. An alternative way that I thought of would be to count number of iterations that has been done in a run and always log that number, which would make the log output consistent (you'd see it with all jobs) and maybe help to debug some other problems in the future. |
If an Enumerator is built and has nothing to iterate, it can be confusing for developers. This is particularly true with ActiveRecord enumerators, where developers can be tripped up by default scopes excluding records they don't realize will be excluded. This PR adds a log message if we attempt to iterate an enumerator with no records to hopefully hint to developers that their enumerator did not actually return anything to help them pinpoint the source of their problem.