Init.d script reports Sidekiq Job Dispatcher as Not Running in v6.1
Closed
Init.d script reports Sidekiq Job Dispatcher as Not Running in v6.1
Created by: demisx
After upgrading to 6.1-stable, the gitlab init.d script, besides taking much time to start, reports sidekiq not running:
# service gitlab start
Starting the GitLab Unicorn web server...
Starting the GitLab Sidekiq event dispatcher...
The GitLab Unicorn webserver with pid 27448 is running.
The GitLab Sidekiq job dispatcher is not running.
However, it appears to be running and GitLab functions properly. Here is my output of ps
after the start:
# ps -ef | grep sidekiq | grep -v grep
git 27484 1 7 11:20 pts/1 00:00:44 sidekiq 2.14.0 gitlab [0 of 25 busy]
And the environment:
$ bundle exec rake gitlab:env:info
System information
System: Ubuntu 12.04
Current User: git
Using RVM: yes
RVM Version: 1.22.3
Ruby Version: 1.9.3p429
Gem Version: 1.8.25
Bundler Version:1.3.5
Rake Version: 10.1.0
GitLab information
Version: 6.1.0.rc1
Revision: 16b6040
Directory: /home/git/gitlab
DB Adapter: mysql2
URL: http://git.colddata.com
HTTP Clone URL: http://git.colddata.com/some-project.git
SSH Clone URL: git@git.colddata.com:some-project.git
Using LDAP: no
Using Omniauth: no
GitLab Shell
Version: 1.7.0
Repositories: /home/git/repositories/
Hooks: /home/git/gitlab-shell/hooks/
Git: /usr/bin/git
Created by: dyadyaMisha
Help!
Last login: Mon Sep 23 09:30:47 2013 misha@GitLab:~$ cd .. misha@GitLab:/home$ cd git/gitlab misha@GitLab:/home/git/gitlab$ sudo service gitlab stop [sudo] password for misha: Stopping the GitLab Unicorn web server... Stopping Sidekiq job dispatcher............................ Sidekiq shut down gracefully. GitLab is not running. misha@GitLab:/home/git/gitlab$ sudo service gitlab start Starting the GitLab Unicorn web server... Starting the GitLab Sidekiq event dispatcher... The GitLab Unicorn webserver with pid 2019 is running. The GitLab Sidekiq job dispatcher is not running.
misha@GitLab:/home/git/gitlab$ sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production Checking Environment ... Git configured for git user? ... yes Has python2? ... yes python2 is supported version? ... yes Checking Environment ... Finished Checking GitLab Shell ... GitLab Shell version >= 1.7.0 ? ... OK (1.7.1) Repo base directory exists? ... yes Repo base directory is a symlink? ... no Repo base owned by git:git? ... yes Repo base access is drwxrws---? ... yes update hook up-to-date? ... yes update hooks in repos are links: ... ANHK / ServerExtension ... repository is empty DocsVision / soglasovanie ... repository is empty Roma / test1 ... repository is empty Checking GitLab Shell ... Finished Checking Sidekiq ... Running? ... yes Checking Sidekiq ... Finished Checking GitLab ... Database config exists? ... yes Database is SQLite ... no All migrations up? ... yes GitLab config exists? ... yes GitLab config outdated? ... no Log directory writable? ... yes Tmp directory writable? ... yes Init script exists? ... yes Init script up-to-date? ... yes Projects have satellites? ... A / ServerExtension ... can't create, repository is empty D / soglasovanie ... can't create, repository is empty Roma / test1 ... can't create, repository is empty Redis version >= 2.0.0? ... yes Your git bin path is "/usr/bin/git" Git version >= 1.7.10 ? ... yes (1.8.4) Checking GitLab ... Finished misha@GitLab:/home/git/gitlab$ sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production System information System: Ubuntu 12.04 Current User: git Using RVM: no Ruby Version: 2.0.0p247 Gem Version: 2.0.3 Bundler Version:1.3.5 Rake Version: 10.1.0 GitLab information Version: 6.1.0.rc1 Revision: 16b6040 Directory: /home/git/gitlab DB Adapter: mysql2 URL: http://10.0.0.153 HTTP Clone URL: http://10.0.0.153/some-project.git SSH Clone URL: git@10.0.0.153:some-project.git Using LDAP: no Using Omniauth: no GitLab Shell Version: 1.7.1 Repositories: /home/git/repositories/ Hooks: /home/git/gitlab-shell/hooks/ Git: /usr/bin/git misha@GitLab:/home/git/gitlab$
By Administrator on 2013-09-23T01:07:42 (imported from GitLab project)
Created by: julien-hedoux
I have exactly the same problem : root@website:/# /etc/init.d/gitlab restart Stopping the GitLab Unicorn web server... Stopping Sidekiq job dispatcher..........................Sidekiq shut down gracefully.
GitLab is not running. Starting the GitLab Unicorn web server... Starting the GitLab Sidekiq event dispatcher... The GitLab Unicorn webserver with pid 11878 is running. The GitLab Sidekiq job dispatcher is not running.
By Administrator on 2013-09-25T17:12:45 (imported from GitLab project)
Created by: oscx
rdanbrook: Thanks for this comment. However, I can not confirm the issue to be resolved after process stop/restart. It's still pending. 6.0 started without troubles. Next sub version notifies about sidekiq issue on start.
By Administrator on 2013-09-30T13:47:10 (imported from GitLab project)
Created by: cedricfung
The same issue here.
sudo service gitlab start Starting the GitLab Unicorn web server... Starting the GitLab Sidekiq event dispatcher... The GitLab Unicorn webserver with pid 4255 is running. The GitLab Sidekiq job dispatcher is not running.
The check
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production Checking Sidekiq ... Running? ... yes Number of Sidekiq processes ... 2 Try fixing it: sudo service gitlab stop sudo pkill -u git -f sidekiq sleep 10 && sudo pkill -9 -u git -f sidekiq sudo service gitlab start Please fix the error above and rerun the checks. Checking Sidekiq ... Finished
The sidekiq is actually running, 1 process
ps -ef | grep sidekiq | grep -v grep git 4273 1 7 23:43 pts/0 00:00:18 sidekiq 2.14.0 gitlab [0 of 25 busy]
By Administrator on 2013-10-05T03:49:52 (imported from GitLab project)
Created by: ladyka
I have this problem too. URI http://dev.quadrate64.ru/ Screen from terminal http://img585.imageshack.us/img585/3213/sbc9.png ( You can see the different colors )
By Administrator on 2013-10-08T11:49:28 (imported from GitLab project)
Created by: dosire
Thanks for reporting people. This is fixed with https://github.com/gitlabhq/gitlabhq/commit/f3d77a7661a9933bd194ed6634f8383ac9fd0448
By Administrator on 2013-10-16T12:32:36 (imported from GitLab project)
Created by: jacobvosmaer
There are two separate issues here. @vecio mentions that
rake gitlab:check
erroneously reports that there are two sidekiq processes while there is only one. This issue with GitLab 6.2 was fixed by f3d77a76 and 73d19bb3.The original issue reported by @colddata is about the init script in GitLab 6.1 saying Sidekiq is not running while in fact it is. I wonder if this has to do with this timeout: https://github.com/gitlabhq/gitlabhq/blob/master/lib/support/init.d/gitlab#L140 . What do you think @rovanion?
By Administrator on 2013-10-18T07:37:55 (imported from GitLab project)
Created by: Rovanion
Yes that timing was set on my machine which although it is very slow, it only has an AMD Sempron 145, may not be the slowest. The best thing would be if we could have Sidekiq tell us when it's started, but that's not likely to happen. We could make a loop which checks if there exists a pidfile there. I'll thinker and make a pull request.
By Administrator on 2013-10-18T08:18:19 (imported from GitLab project)
Created by: jacobvosmaer
@Rovanion I suspect that 97f124b3 may be the fix for this.
In 6-1-stable, the
rake sidekiq:start
task exits while Sidekiq itself is still booting in the background: note thenohup ... &
in https://github.com/gitlabhq/gitlabhq/blob/6-1-stable/lib/tasks/sidekiq.rake#L9.I will make a Merge Request to get 97f124b3 backported into 6-1-stable.
By Administrator on 2013-10-18T09:07:59 (imported from GitLab project)
Created by: jacobvosmaer
@Rovanion I think we should keep the
sleep 2
because the PID only gets written by Sidekiq after daemonization: https://github.com/mperham/sidekiq/blob/v2.14.0/lib/sidekiq/cli.rb#L39The fix I proposed just pushes the exit time of
rake sidekiq:start
closer to Sidekiq'swrite_pid
.By Administrator on 2013-10-18T09:37:09 (imported from GitLab project)
Created by: Rovanion
I began typing a solution which would wait for the pid to be written, it's basically:
while [ ! -f $pid_path ] do sleep 100ms;
Perhaps a break after a set amount of time to declare failure would be good, but other than that I can't think of any. A patch doing this should probably be done either way because there will still be some machine where the 2 second timeout isn't enough. It also speeds up the script where the whole 2 seconds aren't needed.
By Administrator on 2013-10-19T09:37:20 (imported from GitLab project)
Created by: Rovanion
@jacobvosmaer These changes are now included in the pull request I got here: https://github.com/gitlabhq/gitlabhq/pull/5326
By Administrator on 2013-10-21T21:24:55 (imported from GitLab project)