GitLab 5.3 deadlocks on Debian 7.1 Wheezy when visiting project pages
Created by: JoeKun
1. Summary
I have a GitLab instance installed on Debian GNU/kFreeBSD 7.1 Wheezy; it works pretty well for the most part, except that it randomly deadlocks when I visit URLs of projects in the web app, like https://my.gitlab.instance/user/project
.
2. Steps to reproduce
- Install Debian GNU/kFreeBSD 7.1 Wheezy
- Install GitLab 5.3
- Create a project and push some commit to it
- Visit
https://my.gitlab.instance/user/project
3. Expected behavior
I expect my web browser to get a response eventually.
4. Observed behavior
My web browser never gets any response.
5. Relevant logs
I just tried to grab as much information as possible after one of those deadlocks happened, but I'm not sure this information is enough to understand the problem. Let me know if you need me to do anything more to gather relevant debugging information.
Anyway, here's the list of processes that I have related to GitLab:
11:06 root@orange ~git/gitlab/log# ps aux | grep ruby
git 30100 0.0 0.7 388520 125952 ? S 11:07 0:00 ruby1.9.1 /home/git/gitlab/vendor/bundle/ruby/1.9.1/bin/puma -C /home/git/gitlab/config/puma.rb -e production
git 30091 0.1 0.0 40048 8940 ? S 11:06 0:00 python2 /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/gitlab-pygments.rb-0.3.2/lib/pygments/mentos.py
git 30089 17.8 0.7 418800 117548 ? S 11:06 0:11 ruby1.9.1 /home/git/gitlab/vendor/bundle/ruby/1.9.1/bin/sidekiq -q post_receive,mailer,system_hook,project_web_hook,gitlab_shell,common,default -e production -P /home/git/gitlab/tmp/pids/sidekiq.pid
git 30084 0.1 0.0 40048 8940 ? S 11:06 0:00 python2 /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/gitlab-pygments.rb-0.3.2/lib/pygments/mentos.py
git 30075 16.9 0.7 388540 126112 ? Ss 11:06 0:12 ruby1.9.1 /home/git/gitlab/vendor/bundle/ruby/1.9.1/bin/puma -C /home/git/gitlab/config/puma.rb -e production
I attached gdb to each of the ruby processes, and printed backtraces; I also tried to invoke rb_backtrace()
while tailing the logs.
For process 30100:
(gdb) thread apply all backtrace
Thread 1 (process 30100):
#0 0x0000000800c3e407 in __pthread_sigsuspend () from /lib/x86_64-kfreebsd-gnu/libpthread.so.0
#1 0x0000000800c3d3b8 in __pthread_wait_for_restart_signal () from /lib/x86_64-kfreebsd-gnu/libpthread.so.0
#2 0x0000000800c3f7b4 in __pthread_alt_lock () from /lib/x86_64-kfreebsd-gnu/libpthread.so.0
#3 0x0000000800c3d023 in pthread_mutex_lock () from /lib/x86_64-kfreebsd-gnu/libpthread.so.0
#4 0x000000080179df2c in free () from /lib/x86_64-kfreebsd-gnu/libc.so.0.1
#5 0x000000080088f6ca in vm_xfree (ptr=<optimized out>, objspace=<optimized out>) at gc.c:830
#6 ruby_xfree (x=<optimized out>) at gc.c:894
#7 0x0000000800920599 in st_clear (table=0x71cd90) at st.c:256
#8 0x000000080097fd2a in rb_thread_atfork_internal (atfork=atfork@entry=0x800979290 <terminate_atfork_before_exec_i>) at thread.c:3085
#9 0x000000080097fdbc in rb_thread_atfork_before_exec () at thread.c:3134
#10 0x00000008008e42e1 in rb_exec_atfork (arg=0x80d3dc3e0, errmsg=0x80d3dc410 "", errmsg_buflen=80) at process.c:2513
#11 0x00000008008decb5 in chfunc_protect (arg=arg@entry=34581889888) at process.c:2586
#12 0x000000080087fb54 in rb_protect (proc=proc@entry=0x8008deca0 <chfunc_protect>, data=data@entry=34581889888, state=state@entry=0x80d3dc34c) at eval.c:719
#13 0x00000008008e493f in rb_fork_err (status=status@entry=0x80d3dc3cc, chfunc=chfunc@entry=0x8008e42c0 <rb_exec_atfork>, charg=charg@entry=0x80d3dc3e0, fds=<optimized out>, errmsg=errmsg@entry=0x80d3dc410 "",
errmsg_buflen=errmsg_buflen@entry=80) at process.c:2679
#14 0x00000008008e4c52 in rb_spawn_process (earg=earg@entry=0x80d3dc3e0, errmsg=errmsg@entry=0x80d3dc410 "", errmsg_buflen=errmsg_buflen@entry=80, prog=<error reading variable: Unhandled dwarf expression opcode 0xfa>) at process.c:3059
#15 0x00000008008e4cac in rb_f_spawn (argc=<optimized out>, argv=<optimized out>) at process.c:3430
#16 0x00000008009729f3 in vm_call_cfunc (me=0x7016e0, blockptr=0x0, recv=<optimized out>, num=3, reg_cfp=0x80d4db360, th=0x6d6e030) at vm_insnhelper.c:404
#17 vm_call_method (th=th@entry=0x6d6e030, cfp=cfp@entry=0x80d4db360, num=3, blockptr=0x0, flag=<optimized out>, id=<optimized out>, me=0x7016e0, recv=32500840) at vm_insnhelper.c:534
#18 0x00000008009688a9 in vm_exec_core (th=th@entry=0x6d6e030, initial=initial@entry=0) at insns.def:1015
#19 0x000000080096df24 in vm_exec (th=0x6d6e030, th@entry=0x0) at vm.c:1220
#20 0x000000080096eb30 in invoke_block_from_c (cref=0x0, blockptr=<error reading variable: Cannot access memory at address 0x40>, argv=0x80d3deec0, argc=0, self=87411360, block=0x6f72240, th=0x0) at vm.c:624
#21 rb_vm_invoke_proc (th=0x0, proc=0x6f72240, self=87411360, argc=0, argv=0x80d3deec0, blockptr=<error reading variable: Cannot access memory at address 0x40>) at vm.c:670
#22 0x00000008009729f3 in vm_call_cfunc (me=0x70de40, blockptr=0x0, recv=<optimized out>, num=0, reg_cfp=0x80d4dc228, th=0x6d6e030) at vm_insnhelper.c:404
#23 vm_call_method (th=th@entry=0x6d6e030, cfp=cfp@entry=0x80d4dc228, num=0, blockptr=0x0, flag=<optimized out>, id=<optimized out>, me=0x70de40, recv=100706040) at vm_insnhelper.c:534
#24 0x00000008009688a9 in vm_exec_core (th=th@entry=0x6d6e030, initial=initial@entry=0) at insns.def:1015
#25 0x000000080096df24 in vm_exec (th=0x6d6e030, th@entry=0x0) at vm.c:1220
#26 0x000000080096eb30 in invoke_block_from_c (cref=0x0, blockptr=<error reading variable: Cannot access memory at address 0x40>, argv=0x80d3deab8, argc=1, self=96349480, block=0x60d6dc0, th=0x0) at vm.c:624
#27 rb_vm_invoke_proc (th=0x0, proc=0x60d6dc0, self=96349480, argc=1, argv=0x80d3deab8, blockptr=<error reading variable: Cannot access memory at address 0x40>) at vm.c:670
#28 0x00000008009729f3 in vm_call_cfunc (me=0x70de40, blockptr=0x0, recv=<optimized out>, num=1, reg_cfp=0x80d4dcb18, th=0x6d6e030) at vm_insnhelper.c:404
#29 vm_call_method (th=th@entry=0x6d6e030, cfp=cfp@entry=0x80d4dcb18, num=1, blockptr=0x0, flag=<optimized out>, id=<optimized out>, me=0x70de40, recv=87411640) at vm_insnhelper.c:534
#30 0x00000008009688a9 in vm_exec_core (th=th@entry=0x6d6e030, initial=initial@entry=0) at insns.def:1015
#31 0x000000080096df24 in vm_exec (th=th@entry=0x6d6e030) at vm.c:1220
#32 0x000000080097463c in invoke_block_from_c (cref=0x0, blockptr=0x0, argv=0x80d3dcdb8, argc=1, self=45134480, block=0x80d4dccf8, th=0x6d6e030) at vm.c:624
#33 vm_yield (argv=0x80d3dcdb8, argc=1, th=0x6d6e030) at vm.c:654
#34 rb_yield_0 (argv=0x80d3dcdb8, argc=1) at vm_eval.c:760
#35 rb_yield (val=83137360) at vm_eval.c:770
#36 0x000000080084e8c2 in rb_ary_each (array=83143160) at array.c:1478
#37 0x00000008009729f3 in vm_call_cfunc (me=0x6cd010, blockptr=0x80d4dccf8, recv=<optimized out>, num=0, reg_cfp=0x80d4dccd0, th=0x6d6e030) at vm_insnhelper.c:404
#38 vm_call_method (th=th@entry=0x6d6e030, cfp=cfp@entry=0x80d4dccd0, num=0, blockptr=0x80d4dccf8, flag=<optimized out>, id=<optimized out>, me=0x6cd010, recv=83143160) at vm_insnhelper.c:534
#39 0x00000008009688a9 in vm_exec_core (th=th@entry=0x6d6e030, initial=initial@entry=0) at insns.def:1015
#40 0x000000080096df24 in vm_exec (th=th@entry=0x6d6e030) at vm.c:1220
#41 0x0000000800973dd5 in invoke_block_from_c (cref=0x0, blockptr=0x0, argv=0x80d3dd088, argc=1, self=40084600, block=0x80d4dceb0, th=0x6d6e030) at vm.c:624
#42 vm_yield (argv=0x80d3dd088, argc=1, th=0x6d6e030) at vm.c:654
#43 rb_yield_0 (argv=0x80d3dd088, argc=1) at vm_eval.c:760
#44 catch_i (tag=34581889216, tag@entry=16793614, data=data@entry=0) at vm_eval.c:1479
#45 0x00000008009666fa in rb_catch_obj (tag=16793614, func=func@entry=0x800973bf0 <catch_i>, data=data@entry=0) at vm_eval.c:1554
#46 0x000000080096679e in rb_f_catch (argc=<optimized out>, argv=<optimized out>) at vm_eval.c:1530
#47 0x00000008009729f3 in vm_call_cfunc (me=0x6a0f40, blockptr=0x80d4dceb0, recv=<optimized out>, num=1, reg_cfp=0x80d4dce88, th=0x6d6e030) at vm_insnhelper.c:404
#48 vm_call_method (th=th@entry=0x6d6e030, cfp=cfp@entry=0x80d4dce88, num=1, blockptr=0x80d4dceb0, flag=<optimized out>, id=<optimized out>, me=0x6a0f40, recv=40084600) at vm_insnhelper.c:534
#49 0x00000008009688a9 in vm_exec_core (th=th@entry=0x6d6e030, initial=initial@entry=0) at insns.def:1015
#50 0x000000080096df24 in vm_exec (th=th@entry=0x6d6e030) at vm.c:1220
#51 0x000000080096efbd in vm_call0 (th=0x6d6e030, recv=44437720, id=<optimized out>, argc=2, argv=0x80d3dd490, me=0x19f0bf0) at vm_eval.c:66
#52 0x000000080096f96a in vm_method_missing (th=th@entry=0x6d6e030, id=6008, recv=recv@entry=44437720, num=num@entry=1, blockptr=blockptr@entry=0x0, opt=<optimized out>) at vm_insnhelper.c:458
#53 0x0000000800972b3f in vm_call_method (th=th@entry=0x6d6e030, cfp=cfp@entry=0x80d4ddb40, num=1, blockptr=0x0, flag=<optimized out>, id=<optimized out>, me=0x0, recv=44437720) at vm_insnhelper.c:677
#54 0x00000008009688a9 in vm_exec_core (th=th@entry=0x6d6e030, initial=initial@entry=0) at insns.def:1015
#55 0x000000080096df24 in vm_exec (th=th@entry=0x6d6e030) at vm.c:1220
#56 0x000000080097463c in invoke_block_from_c (cref=0x0, blockptr=0x0, argv=0x80d3dd798, argc=1, self=101535560, block=0x80d4ddcc8, th=0x6d6e030) at vm.c:624
#57 vm_yield (argv=0x80d3dd798, argc=1, th=0x6d6e030) at vm.c:654
#58 rb_yield_0 (argv=0x80d3dd798, argc=1) at vm_eval.c:760
#59 rb_yield (val=101533800) at vm_eval.c:770
#60 0x000000080084e8c2 in rb_ary_each (array=101533640) at array.c:1478
#61 0x00000008009729f3 in vm_call_cfunc (me=0x6cd010, blockptr=0x80d4ddcc8, recv=<optimized out>, num=0, reg_cfp=0x80d4ddca0, th=0x6d6e030) at vm_insnhelper.c:404
#62 vm_call_method (th=th@entry=0x6d6e030, cfp=cfp@entry=0x80d4ddca0, num=0, blockptr=0x80d4ddcc8, flag=<optimized out>, id=<optimized out>, me=0x6cd010, recv=101533640) at vm_insnhelper.c:534
#63 0x00000008009688a9 in vm_exec_core (th=th@entry=0x6d6e030, initial=initial@entry=0) at insns.def:1015
#64 0x000000080096df24 in vm_exec (th=0x6d6e030, th@entry=0x0) at vm.c:1220
#65 0x000000080096eb30 in invoke_block_from_c (cref=0x0, blockptr=<error reading variable: Cannot access memory at address 0x40>, argv=0x80d3de080, argc=2, self=101533280, block=0x6480550, th=0x0) at vm.c:624
#66 rb_vm_invoke_proc (th=0x0, proc=0x6480550, self=101533280, argc=2, argv=0x80d3de080, blockptr=<error reading variable: Cannot access memory at address 0x40>) at vm.c:670
#67 0x00000008009729f3 in vm_call_cfunc (me=0x70de40, blockptr=0x0, recv=<optimized out>, num=2, reg_cfp=0x80d4ddf08, th=0x6d6e030) at vm_insnhelper.c:404
#68 vm_call_method (th=th@entry=0x6d6e030, cfp=cfp@entry=0x80d4ddf08, num=2, blockptr=0x0, flag=<optimized out>, id=<optimized out>, me=0x70de40, recv=101529000) at vm_insnhelper.c:534
#69 0x00000008009688a9 in vm_exec_core (th=th@entry=0x6d6e030, initial=initial@entry=0) at insns.def:1015
#70 0x000000080096df24 in vm_exec (th=0x6d6e030, th@entry=0x0) at vm.c:1220
#71 0x000000080096eb30 in invoke_block_from_c (cref=0x0, blockptr=<error reading variable: Cannot access memory at address 0x40>, argv=0x4fc7c60, argc=0, self=101529120, block=0x6268810, th=0x0) at vm.c:624
#72 rb_vm_invoke_proc (th=0x0, th@entry=0x80d3dbe98, proc=0x6268810, self=101529120, argc=0, argv=0x4fc7c60, ) at vm.c:670
#73 0x000000080097e4bf in thread_start_func_2 (th=th@entry=0x6d6e030, stack_start=0x80d3de000) at thread.c:453
#74 0x000000080097e59e in thread_start_func_1 (th_ptr=0x6d6e030) at thread_pthread.c:653
#75 0x0000000800c3bc4a in pthread_start_thread () from /lib/x86_64-kfreebsd-gnu/libpthread.so.0
#76 0x0000000000000000 in ?? ()
(gdb) call rb_backtrace()
The rb_backtrace()
call hung forever on that process, and I saw nothing in the logs.
For process 30089:
(gdb) thread apply all backtrace
Thread 1 (process 30089):
#0 0x0000000800c3e407 in __pthread_sigsuspend () from /lib/x86_64-kfreebsd-gnu/libpthread.so.0
#1 0x0000000800c3d3b8 in __pthread_wait_for_restart_signal () from /lib/x86_64-kfreebsd-gnu/libpthread.so.0
#2 0x0000000800c3b42a in pthread_cond_wait@GLIBC_2.3 () from /lib/x86_64-kfreebsd-gnu/libpthread.so.0
#3 0x0000000800978d89 in native_cond_wait (cond=cond@entry=0x8081ae960, mutex=mutex@entry=0x8081ae9c8) at thread_pthread.c:304
#4 0x000000080097cd3d in native_sleep (th=th@entry=0x8081ae8d0, timeout_tv=timeout_tv@entry=0x0) at thread_pthread.c:905
#5 0x000000080097e7dd in sleep_forever (th=0x8081ae8d0, deadlockable=deadlockable@entry=1) at thread.c:855
#6 0x000000080097eef8 in rb_thread_sleep_deadly () at thread.c:923
#7 rb_mutex_sleep_forever (time=time@entry=4) at thread.c:3680
#8 0x000000080087fd06 in rb_ensure (b_proc=b_proc@entry=0x80097eee0 <rb_mutex_sleep_forever>, data1=data1@entry=4, e_proc=e_proc@entry=0x80097ef10 <rb_mutex_lock>, data2=data2@entry=47945320) at eval.c:744
#9 0x000000080097d23a in rb_mutex_sleep (self=self@entry=47945320, timeout=<optimized out>) at thread.c:3704
#10 0x000000080097d268 in mutex_sleep (argc=<optimized out>, argv=<optimized out>, self=47945320) at thread.c:3727
#11 0x00000008009729f3 in vm_call_cfunc (me=0x721bd0, blockptr=0x0, recv=<optimized out>, num=1, reg_cfp=0x8137d5ca0, th=0x8081ae8d0) at vm_insnhelper.c:404
#12 vm_call_method (th=th@entry=0x8081ae8d0, cfp=cfp@entry=0x8137d5ca0, num=1, blockptr=0x0, flag=<optimized out>, id=<optimized out>, me=0x721bd0, recv=47945320) at vm_insnhelper.c:534
#13 0x00000008009688a9 in vm_exec_core (th=th@entry=0x8081ae8d0, initial=initial@entry=0) at insns.def:1015
#14 0x000000080096df24 in vm_exec (th=0x8081ae8d0, th@entry=0x0) at vm.c:1220
#15 0x000000080096eb30 in invoke_block_from_c (cref=0x0, blockptr=<error reading variable: Cannot access memory at address 0x40>, argv=0x8136d6058, argc=0, self=47990080, block=0x8081ae800, th=0x0) at vm.c:624
#16 rb_vm_invoke_proc (th=0x0, proc=0x8081ae800, self=47990080, argc=0, argv=0x8136d6058, blockptr=<error reading variable: Cannot access memory at address 0x40>) at vm.c:670
#17 0x00000008009729f3 in vm_call_cfunc (me=0x70de40, blockptr=0x0, recv=<optimized out>, num=0, reg_cfp=0x8137d5f08, th=0x8081ae8d0) at vm_insnhelper.c:404
#18 vm_call_method (th=th@entry=0x8081ae8d0, cfp=cfp@entry=0x8137d5f08, num=0, blockptr=0x0, flag=<optimized out>, id=<optimized out>, me=0x70de40, recv=47986400) at vm_insnhelper.c:534
#19 0x00000008009688a9 in vm_exec_core (th=th@entry=0x8081ae8d0, initial=initial@entry=0) at insns.def:1015
#20 0x000000080096df24 in vm_exec (th=0x8081ae8d0, th@entry=0x0) at vm.c:1220
#21 0x000000080096eb30 in invoke_block_from_c (cref=0x0, blockptr=<error reading variable: Cannot access memory at address 0x40>, argv=0x2dc2d18, argc=0, self=6550840, block=0x8081c2bc0, th=0x0) at vm.c:624
#22 rb_vm_invoke_proc (th=0x0, th@entry=0x8138573a8, proc=0x8081c2bc0, self=6550840, argc=0, argv=0x2dc2d18, ) at vm.c:670
#23 0x000000080097e4bf in thread_start_func_2 (th=th@entry=0x8081ae8d0, stack_start=0x813858000) at thread.c:453
#24 0x000000080097e59e in thread_start_func_1 (th_ptr=0x8081ae8d0) at thread_pthread.c:653
#25 0x0000000800c3bc4a in pthread_start_thread () from /lib/x86_64-kfreebsd-gnu/libpthread.so.0
#26 0x0000000000000000 in ?? ()
(gdb) call rb_backtrace()
Program received signal 32, LinuxThreads restart signal.
0x0000000800c3e407 in __pthread_sigsuspend () from /lib/x86_64-kfreebsd-gnu/libpthread.so.0
The program being debugged was signaled while in a function called from GDB.
GDB remains in the frame where the signal was received.
To change this behavior use "set unwindonsignal on".
Evaluation of the expression containing the function
(rb_backtrace) will be abandoned.
When the function is done executing, GDB will silently stop.
(gdb) set unwindonsignal on
(gdb) call rb_backtrace()
warning: Unable to restore previously selected frame.
$1 = 1
and here's what I found in the logs:
from /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/celluloid-0.13.0/lib/celluloid/internal_pool.rb:55:in `block in create'
from /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/celluloid-0.13.0/lib/celluloid/internal_pool.rb:55:in `call'
from /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/celluloid-0.13.0/lib/celluloid/thread_handle.rb:12:in `block in initialize'
from /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/celluloid-0.13.0/lib/celluloid/actor.rb:171:in `block in initialize'
from /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/celluloid-0.13.0/lib/celluloid/actor.rb:182:in `run'
from /home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/celluloid-0.13.0/lib/celluloid/mailbox.rb:67:in `receive'
from /usr/lib/ruby/1.9.1/thread.rb:71:in `wait'
from /usr/lib/ruby/1.9.1/thread.rb:71:in `sleep'
For process 30075:
(gdb) thread apply all backtrace
Thread 1 (process 30075):
#0 0x00000008017f5530 in select () from /lib/x86_64-kfreebsd-gnu/libc.so.0.1
#1 0x000000080097d01b in thread_timer (p=0x602008) at thread_pthread.c:1151
#2 0x0000000800c3bc4a in pthread_start_thread () from /lib/x86_64-kfreebsd-gnu/libpthread.so.0
#3 0x0000000000000000 in ?? ()
(gdb) call rb_backtrace()
$1 = 1
Nothing appeared in the logs for this one.
6. Output of checks
18:21 root@orange ~git/gitlab# sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production
Checking Environment ...
Git configured for git user? ... yes
Has python2? ... yes
python2 is supported version? ... yes
Checking Environment ... Finished
Checking GitLab Shell ...
GitLab Shell version >= 1.4.0 ? ... OK (1.5.0)
Repo base directory exists? ... yes
Repo base directory is a symlink? ... no
Repo base owned by git:git? ... yes
Repo base access is drwxrws---? ... yes
post-receive hook up-to-date? ... yes
post-receive hooks in repos are links: ...
Joel Lopes Da Silva / Joel Web ... ok
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Checking Sidekiq ... Finished
Checking GitLab ...
Database config exists? ... yes
Database is SQLite ... no
All migrations up? ... yes
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Init script exists? ... yes
Init script up-to-date? ... yes
Projects have satellites? ...
Joel Lopes Da Silva / Joel Web ... yes
Redis version >= 2.0.0? ... yes
Your git bin path is "/usr/bin/git"
Git version >= 1.7.10 ? ... yes (1.7.10)
Checking GitLab … Finished
Version of GitLab: 5.3, i.e. latest stable.
Corresponding commit hash: 148eade9.
18:30 root@orange ~git/gitlab# sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production
System information
System: Debian 7.1
Current User: git
Using RVM: no
Ruby Version: 1.9.3p194
Gem Version: 1.8.23
Bundler Version:1.3.5
Rake Version: 10.0.4
GitLab information
Version: 5.3.0
Revision: 148eade
Directory: /home/git/gitlab
DB Adapter: postgresql
URL: https://gitlab.foo.com
HTTP Clone URL: https://gitlab.foo.com/some-project.git
SSH Clone URL: [email protected]:some-project.git
Using LDAP: no
Using Omniauth: no
GitLab Shell
Version: 1.5.0
Repositories: /home/git/repositories/
Hooks: /home/git/gitlab-shell/hooks/
Git: /usr/bin/git