Skip to content

GitLab

Closed
Created by Administrator@rootOwner

Performance degradation over time

Created by: caspervg

My gitlab instance's web interfaces becomes slower as time goes on, until I manually reboot it.

Expected behavior: Gitlab should maintain the same performance over time Actual behavior: The Gitlab Web Interface responds very quickly at first, but becomes slower and slower over time. This problem is 'resolved' by restarting Gitlab, but then the it just starts over again until the next reboot. The git server itself does not suffer from any slowdown whatsoever.

Relevant logs/screenshots: System specs: 4 GB RAM, 150 GB SSD storage, Intel(R) Xeon(R) CPU E3-1240 V2 @ 3.40GHz, CentOS release 6.5 (Final)

Gitlab specs: 18 users, 8 projects As you can see, the system is quite overpowered for the very small amount of users we have (and no other taxing applications are running on the server at all).

Production.log

Started GET "/" for 127.0.0.1 at 2014-07-17 16:27:49 +0400
Processing by DashboardController#show as HTML
Completed 200 OK in 11267ms (Views: 2528.6ms | ActiveRecord: 904.9ms)

Started GET "/?limit=20&offset=0" for 127.0.0.1 at 2014-07-17 16:28:03 +0400
Processing by DashboardController#show as JSON
Parameters: {"limit"=>"20", "offset"=>"0"}

Read fragment views/events/110-20140716155511000000000/6addc44650dc1f7c66594c3371b56320 (4.0ms)
Read fragment views/events/109-20140716154201000000000/6addc44650dc1f7c66594c3371b56320 (62.0ms)
Read fragment views/events/108-20140716144953000000000/6addc44650dc1f7c66594c3371b56320 (8.5ms)
Read fragment views/events/107-20140716144750000000000/6addc44650dc1f7c66594c3371b56320 (101.6ms)
Read fragment views/events/106-20140716134512000000000/6addc44650dc1f7c66594c3371b56320 (22.4ms)
Read fragment views/events/105-20140716122657000000000/6addc44650dc1f7c66594c3371b56320 (0.3ms)
Read fragment views/events/104-20140716112032000000000/6addc44650dc1f7c66594c3371b56320 (0.3ms)
Read fragment views/events/103-20140716111138000000000/6addc44650dc1f7c66594c3371b56320 (60.0ms)

Started GET "/projects/new" for 127.0.0.1 at 2014-07-17 16:28:10 +0400
Processing by ProjectsController#new as HTML

Read fragment views/events/102-20140716094633000000000/6addc44650dc1f7c66594c3371b56320 (128.8ms)
Read fragment views/events/101-20140716094424000000000/6addc44650dc1f7c66594c3371b56320 (121.1ms)
Read fragment views/events/100-20140716093834000000000/6addc44650dc1f7c66594c3371b56320 (152.9ms)
Read fragment views/events/99-20140713223612000000000/6addc44650dc1f7c66594c3371b56320 (85.1ms)
Read fragment views/events/98-20140713184225000000000/6addc44650dc1f7c66594c3371b56320 (196.0ms)
Read fragment views/events/97-20140713183953000000000/6addc44650dc1f7c66594c3371b56320 (82.6ms)
Read fragment views/events/96-20140713154724000000000/6addc44650dc1f7c66594c3371b56320 (0.2ms)
Read fragment views/events/94-20140713161105000000000/6addc44650dc1f7c66594c3371b56320 (20.2ms)
Read fragment views/events/93-20140713161105000000000/6addc44650dc1f7c66594c3371b56320 (94.7ms)
Read fragment views/events/92-20140713161105000000000/6addc44650dc1f7c66594c3371b56320 (19.4ms)
Read fragment views/events/91-20140713161105000000000/6addc44650dc1f7c66594c3371b56320 (51.5ms)
Read fragment views/events/90-20140713154136000000000/6addc44650dc1f7c66594c3371b56320 (103.5ms)

Completed 200 OK in 21186ms (Views: 226.3ms | ActiveRecord: 1207.3ms)

Started GET "/" for 127.0.0.1 at 2014-07-17 16:29:07 +0400

Processing by DashboardController#show as HTML
Started GET "/" for 127.0.0.1 at 2014-07-17 16:29:10 +0400
Processing by DashboardController#show as HTML
Completed 200 OK in 98ms (Views: 19.0ms | ActiveRecord: 43.1ms)

Started GET "/?limit=20&offset=0" for 127.0.0.1 at 2014-07-17 16:29:11 +0400
Processing by DashboardController#show as JSON
Parameters: {"limit"=>"20", "offset"=>"0"}
Completed 200 OK in 62723ms (Views: 62001.2ms | ActiveRecord: 2.7ms)

Read fragment views/events/110-20140716155511000000000/6addc44650dc1f7c66594c3371b56320 (0.3ms)
Read fragment views/events/109-20140716154201000000000/6addc44650dc1f7c66594c3371b56320 (0.2ms)
Read fragment views/events/107-20140716144750000000000/6addc44650dc1f7c66594c3371b56320 (0.2ms)
Read fragment views/events/106-20140716134512000000000/6addc44650dc1f7c66594c3371b56320 (0.2ms)
Read fragment views/events/104-20140716112032000000000/6addc44650dc1f7c66594c3371b56320 (0.2ms)
Read fragment views/events/103-20140716111138000000000/6addc44650dc1f7c66594c3371b56320 (0.1ms)
Read fragment views/events/102-20140716094633000000000/6addc44650dc1f7c66594c3371b56320 (0.2ms)
Read fragment views/events/101-20140716094424000000000/6addc44650dc1f7c66594c3371b56320 (0.2ms)
Read fragment views/events/100-20140716093834000000000/6addc44650dc1f7c66594c3371b56320 (0.3ms)
Read fragment views/events/96-20140713154724000000000/6addc44650dc1f7c66594c3371b56320 (0.3ms)
Read fragment views/events/95-20140713154719000000000/6addc44650dc1f7c66594c3371b56320 (0.2ms)
Read fragment views/events/91-20140713161105000000000/6addc44650dc1f7c66594c3371b56320 (0.2ms)
Read fragment views/events/88-20140713161105000000000/6addc44650dc1f7c66594c3371b56320 (0.3ms)
Read fragment views/events/86-20140713161105000000000/addc44650dc1f7c66594c3371b56320 (0.3ms)
Read fragment views/events/83-20140713161105000000000/6addc44650dc1f7c66594c3371b56320 (0.2ms)
Read fragment views/events/79-20140713161105000000000/6addc44650dc1f7c66594c3371b56320 (0.2ms)
Read fragment views/events/78-20140713161105000000000/6addc44650dc1f7c66594c3371b56320 (0.2ms)
Read fragment views/events/77-20140713161105000000000/6addc44650dc1f7c66594c3371b56320 (0.1ms)
Read fragment views/events/72-20140713161105000000000/6addc44650dc1f7c66594c3371b56320 (0.2ms)
Read fragment views/events/71-20140713161105000000000/6addc44650dc1f7c66594c3371b56320 (0.2ms)

Completed 200 OK in 3982ms (Views: 3.1ms | ActiveRecord: 70.1ms)
Completed 200 OK in 22716ms (Views: 18689.3ms | ActiveRecord: 424.9ms)

As you can see from this small portion of production.log performance can be really erratic and certain requests can take up to 60s to complete.

Output of checks:

  • Output of sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production 
Instance method "lock!" is already defined in ActiveRecord::Base, use generic helper instead or set StateMachine::Machine.ignore_method_conflicts = true.
Checking Environment ...

Git configured for git user? ... yes

Checking Environment ... Finished

Checking GitLab Shell ...

GitLab Shell version >= 1.9.6 ? ... OK (1.9.6)
Repo base directory exists? ... yes
Repo base directory is a symlink? ... no
Repo base owned by git:git? ... yes
Repo base access is drwxrws---? ... yes
Satellites access is drwxr-x---? ... yes
update hook up-to-date? ... yes
update hooks in repos are links: ... [all ok]
Running /home/git/gitlab-shell/bin/check
Check GitLab API access: OK
Check directories and files:
        /home/git/repositories: OK
        /home/git/.ssh/authorized_keys: OK
Test redis-cli executable: redis-cli 2.4.10
Send ping to redis server: PONG
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Database config exists? ... yes
Database is SQLite ... no
All migrations up? ... yes
Database contains orphaned UsersGroups? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Init script exists? ... yes
Init script up-to-date? ... yes

projects have namespace: ... [all yes]
Projects have satellites? ... [all yes]

Redis version >= 2.0.0? ... yes
Your git bin path is "/usr/bin/git"
Git version >= 1.7.10 ? ... yes (1.8.3)

Checking GitLab ... Finished
  • Version of GitLab: currently running Gitlab 7.0-stable, but the problem also occurred on 6.2-stable
  • Relevant output of sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production 
System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.1.2p95
Gem Version:    2.2.1
Bundler Version:1.6.3
Rake Version:   10.3.2
Sidekiq Version:2.17.0

GitLab information
Version:        7.0.0
Revision:       de9bf7d
Directory:      /home/git/gitlab
DB Adapter:     mysql2
<snip>

GitLab Shell
Version:        1.9.6
Repositories:   /home/git/repositories/
Hooks:          /home/git/gitlab-shell/hooks/
Git:            /usr/bin/git

All the assets have been properly compiled too.

Note: This issue seems very similar to #2883 (closed). However, since that issue has been closed a long time ago and happened back on v4.2/v5.0, I decided it would be better to open a new issue instead of commenting there.

Can someone please help me out?