workhorse net/http: timeout awaiting response headers
Created by: ochorocho
Hey,
i have i issue with a huge history, gitlab fails with: 502 Whoops, GitLab is taking too much time to respond.
the timout happens after UI seconds i already increased timeout for git in gitlab.yaml and unicorn as well as nginx config
Setup: GitLab 8.3.1 GitLab Shell 2.6.9 GitLab API v3 Ruby 2.1.7p400 Rails 4.2.4
==> gitlab-workhorse.log <==
2016/01/14 21:50:56 error: proxyRoundTripper: GET "/linuxmce/linuxmce/issues/1170" failed with: "net/http: timeout awaiting response headers"
2016/01/14 21:50:56 ErrorPage: serving predefined error page: 502
git.linuxmce.org @ - - [2016-01-14 21:49:56.458083176 +0100 CET] "GET /linuxmce/linuxmce/issues/1170 HTTP/1.1" 502 429 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:43.0) Gecko/20100101 Firefox/43.0" 60.001366
2016/01/14 21:50:56 Send static file "/home/git/gitlab/public/static.css" ("") for GET "/static.css"
git.linuxmce.org @ - - [2016-01-14 21:50:56.823645601 +0100 CET] "GET /static.css HTTP/1.1" 304 0 "https://git.linuxmce.org/linuxmce/linuxmce/issues/1170" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:43.0) Gecko/20100101 Firefox/43.0" 0.000190
2016/01/14 21:54:01 error: proxyRoundTripper: GET "/linuxmce/linuxmce/issues/1170" failed with: "net/http: timeout awaiting response headers"
2016/01/14 21:54:01 ErrorPage: serving predefined error page: 502
git.linuxmce.org @ - - [2016-01-14 21:53:01.901751763 +0100 CET] "GET /linuxmce/linuxmce/issues/1170 HTTP/1.1" 502 429 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:43.0) Gecko/20100101 Firefox/43.0" 60.000863
2016/01/14 21:54:02 Send static file "/home/git/gitlab/public/static.css" ("") for GET "/static.css"
git.linuxmce.org @ - - [2016-01-14 21:54:02.104927668 +0100 CET] "GET /static.css HTTP/1.1" 304 0 "https://git.linuxmce.org/linuxmce/linuxmce/issues/1170" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:43.0) Gecko/20100101 Firefox/43.0" 0.000170
==> production.log <==
Parameters: {"namespace_id"=>"linuxmce", "project_id"=>"linuxmce", "id"=>"1170"}
Completed 200 OK in 71813ms (Views: 50793.6ms | ActiveRecord: 76.3ms)
Started GET "/linuxmce/linuxmce/issues/1170" for 84.179.85.209 at 2016-01-14 21:49:56 +0100
Processing by Projects::IssuesController#show as HTML
Parameters: {"namespace_id"=>"linuxmce", "project_id"=>"linuxmce", "id"=>"1170"}
Completed 200 OK in 73778ms (Views: 52133.4ms | ActiveRecord: 59.6ms)
Started GET "/linuxmce/linuxmce/issues/1170" for 84.179.85.209 at 2016-01-14 21:53:01 +0100
Processing by Projects::IssuesController#show as HTML
Parameters: {"namespace_id"=>"linuxmce", "project_id"=>"linuxmce", "id"=>"1170"}
Completed 200 OK in 71506ms (Views: 50199.0ms | ActiveRecord: 59.2ms)
==> sidekiq.log <==
2016-01-14T19:10:39.735Z 27725 TID-ovrf6ibzs INFO: Scheduler exiting...
2016-01-14T19:10:39.735Z 27725 TID-ovrf6im98 INFO: Terminating quiet workers
2016-01-14T19:10:39.735Z 27725 TID-ovrfw1pak INFO: Scheduler exiting...
2016-01-14T19:10:40.237Z 27725 TID-ovrf6im98 INFO: Bye!
Sidekiq shut down gracefully.
2016-01-14T19:11:12.166Z 28301 TID-oxo1cmo0k INFO: Booting Sidekiq 4.0.1 with redis options {:url=>"unix:/var/run/redis/redis.sock", :namespace=>"resque:gitlab"}
2016-01-14T19:11:12.173Z 28301 TID-oxo1cmo0k INFO: Cron Jobs - add job with name: stuck_ci_builds_worker
2016-01-14T19:11:14.810Z 28301 TID-oxo1cmo0k INFO: Running in ruby 2.1.7p400 (2015-08-18 revision 51632) [x86_64-linux]
2016-01-14T19:11:14.810Z 28301 TID-oxo1cmo0k INFO: See LICENSE and the LGPL-3.0 for licensing details.
2016-01-14T19:11:14.810Z 28301 TID-oxo1cmo0k INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org
==> unicorn.stderr.log <==
I, [2016-01-14T20:10:37.981069 #27664] INFO -- : reaped #<Process::Status: pid 27680 exit 0> worker=2
I, [2016-01-14T20:10:38.081681 #27664] INFO -- : reaped #<Process::Status: pid 27677 exit 0> worker=1
I, [2016-01-14T20:10:38.082039 #27664] INFO -- : master complete
I, [2016-01-14T20:10:44.078336 #28241] INFO -- : Refreshing Gem list
I, [2016-01-14T20:11:02.138664 #28241] INFO -- : listening on addr=/home/git/gitlab/tmp/sockets/gitlab.socket fd=13
I, [2016-01-14T20:11:02.139178 #28241] INFO -- : listening on addr=127.0.0.1:8080 fd=14
I, [2016-01-14T20:11:02.147306 #28251] INFO -- : worker=0 ready
I, [2016-01-14T20:11:02.152865 #28254] INFO -- : worker=1 ready
I, [2016-01-14T20:11:02.156961 #28241] INFO -- : master process ready
I, [2016-01-14T20:11:02.161766 #28257] INFO -- : worker=2 ready
Thank you.