Gitlab 5.2 Attempt to unlock a mutex which is locked by another thread / Debian 7.0
Created by: yurisum
Gitlab service hangs up randomly and does not respond to Nginx (Nginx returns 504 HTTP response code) and puma reports "ThreadError"
Steps to reproduce
Unknown, issue happens randomly once an hour or so.
Relevant Logs
2013-05-29 16:03:27 -0400: Rack app error: #<ThreadError: Attempt to unlock a mutex which is locked by another thread>/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/lock.rb:20:in unlock' /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/lock.rb:20:inensure in call'
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/lock.rb:21:in call' /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:inforward'
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:245:in fetch' /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:185:inlookup'
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:66:in call!' /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:incall'
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/engine.rb:479:in call' /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/application.rb:223:incall'
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/railtie/configurable.rb:30:in method_missing' /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/builder.rb:134:incall'
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/urlmap.rb:64:in block in call' /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/urlmap.rb:49:ineach'
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/urlmap.rb:49:in call' /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/puma-2.0.1/lib/puma/configuration.rb:66:incall'
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/puma-2.0.1/lib/puma/server.rb:364:in handle_request' /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/puma-2.0.1/lib/puma/server.rb:243:inprocess_client'
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/puma-2.0.1/lib/puma/server.rb:142:in block in run' /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/puma-2.0.1/lib/puma/thread_pool.rb:92:incall'
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/puma-2.0.1/lib/puma/thread_pool.rb:92:in block in spawn_thread' /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/lock.rb:20:inunlock'
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/lock.rb:20:in ensure in call' /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/lock.rb:21:incall'
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in forward' /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:245:infetch'
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:185:in lookup' /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:66:incall!'
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in call' /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/engine.rb:479:incall'
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/application.rb:223:in call' /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/railtie/configurable.rb:30:inmethod_missing'
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/builder.rb:134:in call' /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/urlmap.rb:64:inblock in call'
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/urlmap.rb:49:in each' /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/urlmap.rb:49:incall'
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/puma-2.0.1/lib/puma/configuration.rb:66:in call' /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/puma-2.0.1/lib/puma/server.rb:364:inhandle_request'
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/puma-2.0.1/lib/puma/server.rb:243:in process_client' /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/puma-2.0.1/lib/puma/server.rb:142:inblock in run'
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/puma-2.0.1/lib/puma/thread_pool.rb:92:in call' /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/puma-2.0.1/lib/puma/thread_pool.rb:92:inblock in spawn_thread'
Tried to workaround this by commenting out "config.threadsafe!" in config/environments/production.rb but this resulted in returning 500 response code for any request.
Output of checks
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production
Checking Environment ...
Git configured for git user? ... yes Has python2? ... yes python2 is supported version? ... yes
Checking Environment ... Finished
Checking GitLab Shell ...
GitLab Shell version >= 1.4.0 ? ... OK (1.4.0) Repo base directory exists? ... yes Repo base directory is a symlink? ... no Repo base owned by git:git? ... yes Repo base access is drwxrws---? ... yes post-receive hook up-to-date? ... yes post-receive hooks in repos are links: ... crm / standard ... ok platform / standard ... ok
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... no Try fixing it: sudo -u git -H bundle exec rake sidekiq:start RAILS_ENV=production For more information see: doc/install/installation.md in section "Install Init Script" see log/sidekiq.log for possible errors Please fix the error above and rerun the checks.
Checking Sidekiq ... Finished
Checking GitLab ...
Database config exists? ... yes Database is SQLite ... no All migrations up? ... yes GitLab config exists? ... yes GitLab config outdated? ... no Log directory writable? ... yes Tmp directory writable? ... yes Init script exists? ... yes Init script up-to-date? ... no Try fixing it: Redownload the init script For more information see: doc/install/installation.md in section "Install Init Script" Please fix the error above and rerun the checks. Projects have satellites? ... vendors / SonataAdminBundle ... yes crm / crm ... yes platform-application ... yes crm-application ... yes platform ... yes crm ... yes platform / platform ... yes crm / dev ... yes platform / dev ... yes crm / standard ... yes platform / standard ... yes Redis version >= 2.0.0? ... yes Your git bin path is "/usr/bin/git" Git version >= 1.7.10 ? ... yes (1.7.10)
Checking GitLab ... Finished
This output is confusing: Sidekiq seems to be rinning fine, PID file exist, "sudo -u git -H bundle exec rake sidekiq:start RAILS_ENV=production" produces no output and there is no traces of problems in log/sidekiq.log as far as I can tell. Also "Init script up-to-date? ... no" is confusing: I tool the one from here https://raw.github.com/gitlabhq/gitlabhq/5-2-stable/lib/support/init.d/gitlab - is it really outdated?
Please advice how I can investigate this, what info would help to resolve it.
Thanks!