Skip to content

GitLab

Closed
Created by Administrator@rootOwner

Sidekiq Timeout Connection Pool only 5

Closed

Sidekiq Timeout Connection Pool only 5

Created by: thomasconner

Sidekiq is having a hard time getting a connection to the database. In the sidekiq.log I can see the following:

... "error_message"=>"could not obtain a database connection within 5 seconds (waited 5.000163677 seconds). The max pool size is currently 5; consider increasing it."...

I have changed the connection pool from 5 to 25 in the database.yml file but it seems to be getting overridden. Am I missing something?

  • Administrator
    Administrator @root ·
    Owner

    Created by: senny

    Is this still a problem with GitLab 5?

    By Administrator on 2013-05-05T08:45:23 (imported from GitLab project)

  • Administrator
    Administrator @root ·
    Owner

    Created by: Razer6

    Hey @ThomasConner

    It's been a month since we heared from you. I'm closing this issue but if you still experience this problem, please report back. Make sure to also include the necessary debug information and we can consider to reopen.

    Thanks for your understanding.

    By Administrator on 2013-06-08T16:59:57 (imported from GitLab project)

  • Administrator
    Administrator @root ·
    Owner

    Created by: thomasconner

    Ok. Yea it happens intermittently. Sometimes it is able to connect and sometimes it is not. I will let you know if it continues to become more problematic. Currently it is working fine.

    By Administrator on 2013-06-08T17:06:59 (imported from GitLab project)

  • Administrator
    Administrator @root ·
    Owner

    Created by: bbodenmiller

    @Razer6 @dosire I'm seeing a similar issue on 7.1.1. Worst one appears to be:

    could not obtain a database connection within 5.000 seconds (waited 27.388 seconds)

    Here is part of the sidekiq log:

    2014-09-08T16:37:59Z 21526 TID-oxcif7hig WARN: could not obtain a database connection within 5.000 seconds (waited 25.924 seconds)
2014-09-08T16:37:59Z 21526 TID-oxciw70hw WARN: {"class"=>"PostReceive", "args"=>["/home/git/repositories/externalopensource/gitlabhq.git", "0000000000000000000000000000000000000000", "f9017afe68c03148b4b917c3ff953584a6d4e3d5", "refs/tags/v6.9.2", "key-861"]}
2014-09-08T16:37:59Z 21526 TID-oxciw5icg WARN: could not obtain a database connection within 5.000 seconds (waited 26.301 seconds)
2014-09-08T16:37:59Z 21526 TID-oxciw4dp4 WARN: /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:190:in `block in wait_poll'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:181:in `loop'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:181:in `wait_poll'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:136:in `block in poll'
/usr/local/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:146:in `synchronize'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:134:in `poll'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:418:in `acquire_connection'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:351:in `block in checkout'
/usr/local/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:350:in `checkout'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:265:in `block in connection'
/usr/local/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:264:in `connection'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:541:in `retrieve_connection'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_handling.rb:113:in `retrieve_connection'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_handling.rb:87:in `connection'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/relation/delegation.rb:48:in `connection'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/relation/query_methods.rb:844:in `build_arel'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/relation/query_methods.rb:830:in `arel'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/relation.rb:603:in `exec_queries'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/relation.rb:487:in `load'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/relation.rb:231:in `to_a'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/relation/finder_methods.rb:451:in `find_take'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/relation/finder_methods.rb:98:in `take'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/relation/finder_methods.rb:81:in `find_by'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/querying.rb:7:in `find_by'
/home/git/gitlab/app/models/project.rb:187:in `find_with_namespace'
/home/git/gitlab/app/workers/post_receive.rb:18:in `perform'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/processor.rb:49:in `block (3 levels) in process'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/chain.rb:122:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/chain.rb:122:in `block in invoke'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/server/active_record.rb:6:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/chain.rb:124:in `block in invoke'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/server/retry_jobs.rb:62:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/chain.rb:124:in `block in invoke'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/logging.rb:22:in `with_context'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/server/logging.rb:7:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/chain.rb:124:in `block in invoke'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/chain.rb:127:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/chain.rb:127:in `invoke'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/processor.rb:48:in `block (2 levels) in process'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/processor.rb:105:in `stats'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/processor.rb:47:in `block in process'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/processor.rb:86:in `do_defer'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/processor.rb:37:in `process'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `public_send'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `dispatch'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:122:in `dispatch'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/celluloid-0.15.2/lib/celluloid/actor.rb:322:in `block in handle_message'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/celluloid-0.15.2/lib/celluloid/actor.rb:416:in `block in task'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/celluloid-0.15.2/lib/celluloid/tasks.rb:55:in `block in initialize'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/celluloid-0.15.2/lib/celluloid/tasks/task_fiber.rb:13:in `block in create'
2014-09-08T16:37:59Z 21526 TID-oxciw52zo WARN: {"class"=>"PostReceive", "args"=>["/home/git/repositories/externalopensource/gitlabhq.git", "0000000000000000000000000000000000000000", "79b16ce93ceb0814f147ecb552d8d30c836484de", "refs/tags/v6.9.1", "key-861"]}
2014-09-08T16:37:59Z 21526 TID-oxciw3o9k WARN: {"class"=>"PostReceive", "args"=>["/home/git/repositories/externalopensource/gitlabhq.git", "0000000000000000000000000000000000000000", "e4af150da6e2876bbd6f88ecce17ada6a00f0e73", "refs/tags/v7.2.0.rc1", "key-861"]}
2014-09-08T16:37:59Z 21526 TID-oxcien254 WARN: /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:190:in `block in wait_poll'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:181:in `loop'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:181:in `wait_poll'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:136:in `block in poll'
/usr/local/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:146:in `synchronize'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:134:in `poll'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:418:in `acquire_connection'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:351:in `block in checkout'
/usr/local/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:350:in `checkout'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:265:in `block in connection'
/usr/local/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:264:in `connection'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:541:in `retrieve_connection'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_handling.rb:113:in `retrieve_connection'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/connection_handling.rb:87:in `connection'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/relation/delegation.rb:48:in `connection'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/relation/query_methods.rb:844:in `build_arel'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/relation/query_methods.rb:830:in `arel'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/relation.rb:603:in `exec_queries'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/relation.rb:487:in `load'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/relation.rb:231:in `to_a'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/relation/finder_methods.rb:451:in `find_take'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/relation/finder_methods.rb:98:in `take'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/relation/finder_methods.rb:81:in `find_by'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/querying.rb:7:in `find_by'
/home/git/gitlab/app/models/project.rb:187:in `find_with_namespace'
/home/git/gitlab/app/workers/post_receive.rb:18:in `perform'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/processor.rb:49:in `block (3 levels) in process'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/chain.rb:122:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/chain.rb:122:in `block in invoke'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/server/active_record.rb:6:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/chain.rb:124:in `block in invoke'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/server/retry_jobs.rb:62:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/chain.rb:124:in `block in invoke'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/logging.rb:22:in `with_context'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/server/logging.rb:7:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/chain.rb:124:in `block in invoke'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/chain.rb:127:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/middleware/chain.rb:127:in `invoke'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/processor.rb:48:in `block (2 levels) in process'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/processor.rb:105:in `stats'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/processor.rb:47:in `block in process'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/processor.rb:86:in `do_defer'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-2.17.0/lib/sidekiq/processor.rb:37:in `process'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `public_send'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `dispatch'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:122:in `dispatch'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/celluloid-0.15.2/lib/celluloid/actor.rb:322:in `block in handle_message'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/celluloid-0.15.2/lib/celluloid/actor.rb:416:in `block in task'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/celluloid-0.15.2/lib/celluloid/tasks.rb:55:in `block in initialize'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/celluloid-0.15.2/lib/celluloid/tasks/task_fiber.rb:13:in `block in create'
2014-09-08T16:37:59Z 21526 TID-oxciemmf0 WARN: {"class"=>"PostReceive", "args"=>["/home/git/repositories/externalopensource/gitlabhq.git", "0000000000000000000000000000000000000000", "80236d6c6c763651f398ec4f6b43591b745e385b", "refs/tags/v6.7.0", "key-861"]}
2014-09-08T16:37:59Z 21526 TID-oxciemmf0 WARN: could not obtain a database connection within 5.000 seconds (waited 27.455 seconds)
2014-09-08T16:37:59Z 21526 TID-oxcife7zc WARN: could not obtain a database connection within 5.000 seconds (waited 26.964 seconds)
2014-09-08T16:37:59Z 21526 TID-oxciw7kzo WARN: {"class"=>"PostReceive", "args"=>["/home/git/repositories/externalopensource/gitlabhq.git", "0000000000000000000000000000000000000000", "4ec19d22719ff334bd02c99dc2ba6155eef84fc8", "refs/tags/v7.2.0", "key-861"]}
2014-09-08T16:37:59Z 21526 TID-oxcifeauo WARN: could not obtain a database connection within 5.000 seconds (waited 23.262 seconds)
2014-09-08T16:37:59Z 21526 TID-oxcifdpdc PostReceive JID- INFO: start

    By Administrator on 2014-09-08T17:19:34 (imported from GitLab project)

  • Administrator
    Administrator @root ·
    Owner

    Created by: dosire

    Do you have enough MySQL connections configured for all unicorn workers + all sidekiq workers?

    By Administrator on 2014-09-09T08:39:21 (imported from GitLab project)

  • Administrator
    Administrator @root ·
    Owner

    Created by: bbodenmiller

    Good question. Where might one find how many MySQL connections are configured and how many sidekiq workers I have? Unicorns is https://github.com/gitlabhq/gitlabhq/blob/master/config/unicorn.rb.example#L18, correct?

    By Administrator on 2014-09-09T15:43:12 (imported from GitLab project)

  • Administrator
    Administrator @root ·
    Owner

    Created by: dosire

    Sidekiq workers are 25 by default, you are correct about the unicorn worker location. MySQL docs for this are in http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html#sysvar_max_user_connections

    By Administrator on 2014-09-10T11:25:28 (imported from GitLab project)

  • Administrator
    Administrator @root ·
    Owner

    Created by: bbodenmiller

    Appears to be enough connections:

    image

    Running 7.1.1 with Unicorn timeout at 30 with 4 workers on 4 core machine. nginx-ssl proxy_connect_timeout is 300 and proxy_read_timeout is 300. No idea how to set number of sidekiq workers so haven't set that.

    By Administrator on 2014-09-11T00:56:07 (imported from GitLab project)

  • Administrator
    Administrator @root ·
    Owner

    Created by: bbodenmiller

    @dosire could this have to do with pool setting in database.yml? Mine is only set to 5 rather than current default of 10. Proposed change at https://github.com/gitlabhq/gitlabhq/pull/7742 to add this to upgrade guide.

    By Administrator on 2014-09-12T08:01:31 (imported from GitLab project)

  • Administrator
    Administrator @root ·
    Owner

    Created by: dosire

    @bbodenmiller Yes that could have to do with that. Sorry for pointing you in the wrong direction.

    By Administrator on 2014-09-12T09:41:05 (imported from GitLab project)

  • Administrator
    Administrator @root ·
    Owner

    Created by: bbodenmiller

    No worries - it was a learning experience. I'll give the connections pool change a try.

    By Administrator on 2014-09-12T19:48:41 (imported from GitLab project)

Please register or sign in to reply