delayed_job process on OSX consistently consuming massive amounts of system resources

[constantm]

I've had this issue for a while on Yosemite, and it still persists after a reinstall. When starting delayed_job, it runs fine for a few minutes, after which the ruby process starts eating more and more CPU resources until it caps out at 100% and brings the Mac to a halt. The job queue is empty while this happens. Looking at CPU usage, it seems to start with the usage of the Ruby process shooting up every few seconds, and then gets worse and worse. Not sure where to even start looking for the issue?

I'm on OSX 10.10.3 (although it happened on earlier versions as well), Ruby 2.2.2 (it also happened on previous versions) and delayed_job 4.0.6.

[albus522]

Start with updating DJ. Then look for anything you have that might be triggering a background thread. Also look for anything that you have running in the worker loop hook.

[constantm]

When I posted the issue four months ago I had updated everything from DJ to Ruby. I was using DJ purely for sending mails and analytics tracking - no heavy lifting.

[QuentinFchx]

Might be due to delayed_job reloading the app too often.
https://github.com/collectiveidea/delayed_job/blob/master/lib/delayed/worker.rb#L317
Monkey patching the reload_app? in dev env fixed the issue for me.

module Delayed
    class Worker
        # Prevent high CPU usage in dev. But we lose automatic code reloading in workers.
        def self.reload_app?
            false
        end
    end
end

@albus522 do you think we could be more clever on when to reload the app ?

[jturkel]

We're also seeing Delayed Job consume a lot of CPU when the job queue is empty. Setting Rails.configuration.cache_classes = false (which has the same effect as @QuentinFchx's monkey patch) makes the problem go away.

@albus522 - Could we instead trigger code reloading before running each job which is the moral equivalent of what Rails is doing with web requests? I'm happy to put together a PR if you'd be open to folding in the change.

[jturkel]

For anyone interested here's a monkey patch that causes workers to only reload code before running a job rather than during each iteration of the job polling loop. On my machine it cuts CPU usage of an idle worker from 30% to 0.2%.

module Delayed::WorkerClassReloadingPatch
  # Override Delayed::Worker#reserve_job to optionally reload classes before running a job
  def reserve_job(*)
    job = super

    if job && self.class.reload_app?
      ActionDispatch::Reloader.cleanup!
      ActionDispatch::Reloader.prepare!
    end

    job
  end

  # Override Delayed::Worker#reload! which is called from the job polling loop to not reload classes
  def reload!
    # no-op
  end
end
Delayed::Worker.send(:prepend, Delayed::WorkerClassReloadingPatch)

EDIT: Updated this to override Delayed::Worker#reserve_job rather than Delayed::Worker#run since before perform hooks can access the job's payload object.

[ypresto]

If you are using ActiveJob, you can simply disable delayed_job's reload feature.
See: #776 (comment)

[nicbor]

+1. We started seeing this problem in Sierra dev environment when we upgraded to Rails 5. We've tried @jturkel's solution and others, nothing has worked so far.

[albus522]

Closing stale