GitLab

Created by: oscx

For some months, though upgrading monthly, my gitlab instance can no longer search well.

For mid and large sized repos, we can not search at all. Always, this fails with 502 after about 60sec.

Checking the logs in gitlab admin UI didn't provide details.

However, for unicorn I found...

tail log/unicorn.stderr.log -n 100
I, [2014-12-23T16:42:23.257239 #4737]  INFO -- : reaped #<Process::Status: pid 23823 exit 0> worker=0
I, [2014-12-23T16:42:23.279542 #26264]  INFO -- : worker=0 ready
W, [2014-12-23T16:47:19.635130 #26264]  WARN -- : #<Unicorn::HttpServer:0x007f759a04a990>: worker (pid: 26264) exceeds memory limit (240123904 bytes > 234709541 bytes)
W, [2014-12-23T16:47:19.635360 #26264]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 26264) alive: 285 sec (trial 1)
I, [2014-12-23T16:47:20.101343 #4737]  INFO -- : reaped #<Process::Status: pid 26264 exit 0> worker=0
I, [2014-12-23T16:47:20.123077 #28638]  INFO -- : worker=0 ready
W, [2014-12-23T16:52:19.230075 #28638]  WARN -- : #<Unicorn::HttpServer:0x007f759a04a990>: worker (pid: 28638) exceeds memory limit (241348608 bytes > 214569582 bytes)
W, [2014-12-23T16:52:19.230302 #28638]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 28638) alive: 281 sec (trial 1)
I, [2014-12-23T16:52:19.761094 #4737]  INFO -- : reaped #<Process::Status: pid 28638 exit 0> worker=0
I, [2014-12-23T16:52:19.781429 #32106]  INFO -- : worker=0 ready
W, [2014-12-23T16:58:49.227168 #32106]  WARN -- : #<Unicorn::HttpServer:0x007f759a04a990>: worker (pid: 32106) exceeds memory limit (235241472 bytes > 211732555 bytes)
W, [2014-12-23T16:58:49.227379 #32106]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 32106) alive: 386 sec (trial 1)
I, [2014-12-23T16:58:49.713186 #4737]  INFO -- : reaped #<Process::Status: pid 32106 exit 0> worker=0
I, [2014-12-23T16:58:49.737693 #3004]  INFO -- : worker=0 ready
W, [2014-12-23T17:25:34.343507 #3004]  WARN -- : #<Unicorn::HttpServer:0x007f759a04a990>: worker (pid: 3004) exceeds memory limit (259817472 bytes > 259027618 bytes)
W, [2014-12-23T17:25:34.343806 #3004]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 3004) alive: 1601 sec (trial 1)
I, [2014-12-23T17:25:34.768918 #4737]  INFO -- : reaped #<Process::Status: pid 3004 exit 0> worker=0
I, [2014-12-23T17:25:34.791034 #17463]  INFO -- : worker=0 ready
W, [2014-12-23T17:27:34.353177 #21968]  WARN -- : #<Unicorn::HttpServer:0x007f759a04a990>: worker (pid: 21968) exceeds memory limit (271720448 bytes > 257261236 bytes)
W, [2014-12-23T17:27:34.353435 #21968]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 21968) alive: 3240 sec (trial 1)
I, [2014-12-23T17:27:34.816100 #4737]  INFO -- : reaped #<Process::Status: pid 21968 exit 0> worker=1
I, [2014-12-23T17:27:34.840156 #18367]  INFO -- : worker=1 ready
W, [2014-12-23T17:37:56.682224 #17463]  WARN -- : #<Unicorn::HttpServer:0x007f759a04a990>: worker (pid: 17463) exceeds memory limit (254042112 bytes > 241448214 bytes)
W, [2014-12-23T17:37:56.682437 #17463]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 17463) alive: 727 sec (trial 1)
I, [2014-12-23T17:37:57.299455 #4737]  INFO -- : reaped #<Process::Status: pid 17463 exit 0> worker=0
I, [2014-12-23T17:37:57.320615 #23823]  INFO -- : worker=0 ready
W, [2014-12-23T17:48:04.466998 #23823]  WARN -- : #<Unicorn::HttpServer:0x007f759a04a990>: worker (pid: 23823) exceeds memory limit (236318720 bytes > 231153030 bytes)
W, [2014-12-23T17:48:04.467232 #23823]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 23823) alive: 561 sec (trial 1)
I, [2014-12-23T17:48:04.946567 #4737]  INFO -- : reaped #<Process::Status: pid 23823 exit 0> worker=0
I, [2014-12-23T17:48:04.968795 #28665]  INFO -- : worker=0 ready
W, [2014-12-23T18:34:34.458624 #18367]  WARN -- : #<Unicorn::HttpServer:0x007f759a04a990>: worker (pid: 18367) exceeds memory limit (275238912 bytes > 260396637 bytes)
W, [2014-12-23T18:34:34.459168 #18367]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 18367) alive: 3975 sec (trial 1)
I, [2014-12-23T18:34:34.937111 #4737]  INFO -- : reaped #<Process::Status: pid 18367 exit 0> worker=1
I, [2014-12-23T18:34:34.959498 #21134]  INFO -- : worker=1 ready
E, [2014-12-23T18:53:10.785761 #4737] ERROR -- : worker=1 PID:21134 timeout (61s > 60s), killing
E, [2014-12-23T18:53:10.855872 #4737] ERROR -- : reaped #<Process::Status: pid 21134 SIGKILL (signal 9)> worker=1
I, [2014-12-23T18:53:10.877158 #31183]  INFO -- : worker=1 ready

Gitlab project has 921 commits 111 branches 0 tags 56.65 MB, so I wouldn't classify this as large project..

The VM has 8GB RAM and 8vCPU. Shouldn't this server be able to deal with the load?

We don't worry about slow search, but error 502 doesn't return any search results, even after waiting a minute...

Have I missed any sphinx or alternative search module for gitlab or is search "limited" and time 300 is the only chance for fixing?