Skip to content

GitLab

Closed
Created by Administrator@rootOwner

(yet another) 502 GitLab is not responding.

Created by: JonathonReinhart

I currently cannot access GitLab at all. I am forgoing restarting the server to see if anyone can help me determine a root cause to keep this from happening.

Related:

  • #2057 (closed) - Looks very similar, however his/her solution was to change:
proxy_read_timeout 300 -> 3000
proxy_connect_timeout 300 -> 3000

which doesn't seem like an acceptable solution.

Logs:

[root@gitlab ~]# gitlab-ctl tail
==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <==
(contents removed)

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/" for 127.0.0.1 at 2015-01-10 21:06:48 -0500
Started GET "/" for 127.0.0.1 at 2015-01-10 21:08:48 -0500
Started GET "/" for 127.0.0.1 at 2015-01-10 22:52:04 -0500
Started GET "/" for 127.0.0.1 at 2015-01-11 11:57:36 -0500
Started GET "/" for 127.0.0.1 at 2015-01-11 11:57:40 -0500
Started GET "/" for 127.0.0.1 at 2015-01-11 12:02:53 -0500
Processing by DashboardController#show as HTML
Started GET "/" for 127.0.0.1 at 2015-01-11 12:04:48 -0500
Processing by DashboardController#show as HTML

==> /var/log/gitlab/gitlab-rails/application.log <==
(contents removed)

==> /var/log/gitlab/gitlab-rails/satellites.log <==
(contents removed)


==> /var/log/gitlab/gitlab-rails/githost.log <==
# Logfile created on 2014-11-22 19:33:41 -0500 by logger.rb/44203

==> /var/log/gitlab/redis/config <==
s209715200
n30
t86400
!gzip



==> /var/log/gitlab/redis/state <==

==> /var/log/gitlab/redis/current <==

==> /var/log/gitlab/postgresql/config <==
s209715200
n30
t86400
!gzip



==> /var/log/gitlab/postgresql/current <==

==> /var/log/gitlab/postgresql/state <==

==> /var/log/gitlab/unicorn/config <==
s209715200
n30
t86400
!gzip



==> /var/log/gitlab/unicorn/unicorn_stderr.log <==
I, [2015-01-11T11:58:02.858427 #28801]  INFO -- : worker=1 spawned pid=28801
I, [2015-01-11T11:58:02.860367 #28801]  INFO -- : worker=1 ready
E, [2015-01-11T12:03:24.803695 #892] ERROR -- : worker=1 PID:28801 timeout (31s > 30s), killing
E, [2015-01-11T12:03:24.829101 #892] ERROR -- : reaped #<Process::Status: pid 28801 SIGKILL (signal 9)> worker=1
I, [2015-01-11T12:03:24.901809 #29273]  INFO -- : worker=1 spawned pid=29273
I, [2015-01-11T12:03:24.905204 #29273]  INFO -- : worker=1 ready
E, [2015-01-11T12:05:19.262994 #892] ERROR -- : worker=1 PID:29273 timeout (31s > 30s), killing
E, [2015-01-11T12:05:19.589239 #892] ERROR -- : reaped #<Process::Status: pid 29273 SIGKILL (signal 9)> worker=1
I, [2015-01-11T12:05:19.658644 #29420]  INFO -- : worker=1 spawned pid=29420
I, [2015-01-11T12:05:19.660911 #29420]  INFO -- : worker=1 ready

==> /var/log/gitlab/unicorn/unicorn_stdout.log <==
2014-11-23T00:34:25Z 9744 TID-4s6ck INFO: Sidekiq client with redis options {:url=>"redis://127.0.0.1:6379", :namespace=>"resque:gitlab"}
2014-11-23T00:34:30Z 8700 TID-4s6ck INFO: Sidekiq client with redis options {:url=>"redis://127.0.0.1:6379", :namespace=>"resque:gitlab"}
Initialized empty Git repository in /var/opt/gitlab/git-data/repositories/....git/
Initialized empty Git repository in /var/opt/gitlab/git-data/repositories/....git/
Initialized empty Git repository in /var/opt/gitlab/git-data/repositories/....git/

==> /var/log/gitlab/unicorn/current <==

==> /var/log/gitlab/unicorn/state <==

==> /var/log/gitlab/sidekiq/config <==
s209715200
n30
t86400
!gzip



==> /var/log/gitlab/sidekiq/state <==

==> /var/log/gitlab/sidekiq/current <==

==> /var/log/gitlab/nginx/config <==
s209715200
n30
t86400
!gzip



==> /var/log/gitlab/nginx/error.log <==

==> /var/log/gitlab/nginx/gitlab_access.log <==
10.0.1.128 - - [10/Jan/2015:21:06:56 -0500] "GET / HTTP/1.1" 499 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36"
10.0.1.128 - - [10/Jan/2015:21:07:28 -0500] "GET / HTTP/1.1" 502 323 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36"
10.0.1.128 - - [10/Jan/2015:21:07:28 -0500] "GET /static.css HTTP/1.1" 200 290 "http://[url]/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36"
10.0.1.128 - - [10/Jan/2015:21:08:52 -0500] "GET / HTTP/1.1" 502 323 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36"
10.0.1.128 - - [10/Jan/2015:22:52:31 -0500] "GET / HTTP/1.1" 502 323 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36"
10.0.1.128 - - [11/Jan/2015:11:57:31 -0500] "GET / HTTP/1.1" 499 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36"
10.0.1.128 - - [11/Jan/2015:11:58:02 -0500] "GET / HTTP/1.1" 502 323 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36"
10.0.1.128 - - [11/Jan/2015:12:03:24 -0500] "GET / HTTP/1.1" 502 323 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36"
10.0.1.128 - - [11/Jan/2015:12:05:19 -0500] "GET / HTTP/1.1" 502 323 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36"

==> /var/log/gitlab/nginx/gitlab_error.log <==
2014/10/05 22:59:32 [error] 608#0: *621 upstream prematurely closed connection while reading response header from upstream, client: 10.0.1.101, server: [hostname], request: "POST /users/sign_in HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-rails/tmp/sockets/gitlab.socket:/users/sign_in", host: "[hostname]", referrer: "http://[hostname]/users/sign_in"
2015/01/10 06:44:24 [error] 608#0: *1310 upstream prematurely closed connection while reading response header from upstream, client: 10.0.1.128, server: [hostname], request: "GET /[user]/[repo]?limit=20&offset=0 HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-rails/tmp/sockets/gitlab.socket:/[user]/[repo]?limit=20&offset=0", host: "[hostname]"
2015/01/10 21:07:28 [error] 608#0: *1316 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.0.1.128, server: [hostname], request: "GET / HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-rails/tmp/sockets/gitlab.socket:/", host: "[hostname]"
2015/01/10 21:08:52 [error] 608#0: *1316 upstream prematurely closed connection while reading response header from upstream, client: 10.0.1.128, server: [hostname], request: "GET / HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-rails/tmp/sockets/gitlab.socket:/", host: "[hostname]"
2015/01/10 22:52:31 [error] 608#0: *1321 upstream prematurely closed connection while reading response header from upstream, client: 10.0.1.128, server: [hostname], request: "GET / HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-rails/tmp/sockets/gitlab.socket:/", host: "[hostname]"
2015/01/11 11:58:02 [error] 608#0: *1325 upstream prematurely closed connection while reading response header from upstream, client: 10.0.1.128, server: [hostname], request: "GET / HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-rails/tmp/sockets/gitlab.socket:/", host: "[hostname]"
2015/01/11 12:03:24 [error] 608#0: *1328 upstream prematurely closed connection while reading response header from upstream, client: 10.0.1.128, server: [hostname], request: "GET / HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-rails/tmp/sockets/gitlab.socket:/", host: "[hostname]"
2015/01/11 12:05:19 [error] 608#0: *1331 upstream prematurely closed connection while reading response header from upstream, client: 10.0.1.128, server: [hostname], request: "GET / HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-rails/tmp/sockets/gitlab.socket:/", host: "[hostname]"

==> /var/log/gitlab/nginx/current <==

==> /var/log/gitlab/nginx/state <==