Slow ProjectCacheWorker
Created by: bagedevimo
Since updating from 8.2.x to 8.5.3, we've experienced very slow hook delivery, among other things.
The background workers admin page suggests that it's due to ProjectCacheWorker running very slowly (upwards of 40 minutes). Some quick investigation on the rails console suggests that it might be due to the calls to pre-cache the number of diverging commits for every branch (https://github.com/gitlabhq/gitlabhq/blob/master/app/models/repository.rb#L219-L224).
We have a large number of branches (4,000+), so even running that block of code via the console takes upwards of 10 minutes.
It appears that the cache is cleared out pretty often, and so the interval between the cache being flushed and regen starting is smaller than the interval between us pushing code.