Skip to content

GitLab

Closed
Created by Administrator@rootOwner

Blame crashes with "Rugged::TreeError" on specific repository

Created by: Alexey-I

Summary: Blame crashes on some files of one specific repository. Normal operation is expected.

Steps to reproduce: not clear yet, advice needed (see below).

Expected behavior: normal operation.

Observed behavior

There is a repository with several brances, let's call it a-repository. Branch dev is being actively updated. File of interest is gradle.properties. It is located in root folder of dev branch.

When we go to Files —> (switch to dev branch) —> "gradle.properties" and click "blame", it goes "Blam!" and says "500 We're sorry, but something went wrong."

This is what is seen in production.log (log level "debug" is in effect):

Started GET "/a-group/a-repository/blame/dev/gradle.properties" for 127.0.0.1 at 2014-12-05 13:26:09 +0300
Processing by Projects::BlameController#show as HTML
  Parameters: {"project_id"=>"a-group/a-repository", "id"=>"dev/gradle.properties"}
  User Load (0.7ms)  SELECT  `users`.* FROM `users`  WHERE `users`.`id` = 3  ORDER BY `users`.`id` ASC LIMIT 1
  Namespace Load (0.3ms)  SELECT  `namespaces`.* FROM `namespaces`  WHERE `namespaces`.`path` = 'a-group' LIMIT 1
  Project Load (0.2ms)  SELECT  `projects`.* FROM `projects`  WHERE `projects`.`namespace_id` = 246 AND `projects`.`path` = 'a-repository' LIMIT 1
  ProjectMember Load (0.2ms)  SELECT  `members`.* FROM `members`  WHERE `members`.`source_type` = 'Project' AND `members`.`type` IN ('ProjectMember') AND `members`.`source_id` = 864 AND `members`.`source_type` = 'Project' AND `members`.`user_id` = 3 LIMIT 1
  Group Load (0.2ms)  SELECT  `namespaces`.* FROM `namespaces`  WHERE `namespaces`.`type` IN ('Group') AND `namespaces`.`id` = 246 AND `namespaces`.`type` = 'Group' LIMIT 1
  GroupMember Load (0.2ms)  SELECT  `members`.* FROM `members`  WHERE `members`.`source_type` = 'Namespace' AND `members`.`type` IN ('GroupMember') AND `members`.`source_id` = 246 AND `members`.`source_type` = 'Namespace' AND `members`.`user_id` = 3 LIMIT 1
  CACHE (0.0ms)  SELECT  `members`.* FROM `members`  WHERE `members`.`source_type` = 'Project' AND `members`.`type` IN ('ProjectMember') AND `members`.`source_id` = 864 AND `members`.`source_type` = 'Project' AND `members`.`user_id` = 3 LIMIT 1  [["source_id", 864], ["source_type", "Project"]]
  CACHE (0.0ms)  SELECT  `members`.* FROM `members`  WHERE `members`.`source_type` = 'Namespace' AND `members`.`type` IN ('GroupMember') AND `members`.`source_id` = 246 AND `members`.`source_type` = 'Namespace' AND `members`.`user_id` = 3 LIMIT 1  [["source_id", 246], ["source_type", "Namespace"]]
  CACHE (0.0ms)  SELECT  `members`.* FROM `members`  WHERE `members`.`source_type` = 'Project' AND `members`.`type` IN ('ProjectMember') AND `members`.`source_id` = 864 AND `members`.`source_type` = 'Project' AND `members`.`user_id` = 3 LIMIT 1  [["source_id", 864], ["source_type", "Project"]]
  CACHE (0.0ms)  SELECT  `members`.* FROM `members`  WHERE `members`.`source_type` = 'Namespace' AND `members`.`type` IN ('GroupMember') AND `members`.`source_id` = 246 AND `members`.`source_type` = 'Namespace' AND `members`.`user_id` = 3 LIMIT 1  [["source_id", 246], ["source_type", "Namespace"]]
  CACHE (0.0ms)  SELECT  `members`.* FROM `members`  WHERE `members`.`source_type` = 'Project' AND `members`.`type` IN ('ProjectMember') AND `members`.`source_id` = 864 AND `members`.`source_type` = 'Project' AND `members`.`user_id` = 3 LIMIT 1  [["source_id", 864], ["source_type", "Project"]]
  CACHE (0.0ms)  SELECT  `members`.* FROM `members`  WHERE `members`.`source_type` = 'Namespace' AND `members`.`type` IN ('GroupMember') AND `members`.`source_id` = 246 AND `members`.`source_type` = 'Namespace' AND `members`.`user_id` = 3 LIMIT 1  [["source_id", 246], ["source_type", "Namespace"]]
  GroupMember Load (0.2ms)  SELECT `members`.* FROM `members`  WHERE `members`.`source_type` = 'Namespace' AND `members`.`type` IN ('GroupMember') AND `members`.`source_id` = 246 AND `members`.`source_type` = 'Namespace' AND `members`.`access_level` = 50
  User Load (0.7ms)  SELECT  `users`.* FROM `users`  WHERE `users`.`id` = 1 LIMIT 1
  User Load (0.4ms)  SELECT  `users`.* FROM `users`  WHERE `users`.`id` = 195 LIMIT 1
  User Load (0.3ms)  SELECT  `users`.* FROM `users`  WHERE `users`.`id` = 24 LIMIT 1
  User Load (0.3ms)  SELECT  `users`.* FROM `users`  WHERE `users`.`id` = 249 LIMIT 1
  User Load (0.4ms)  SELECT  `users`.* FROM `users`  WHERE `users`.`id` = 138 LIMIT 1
  User Load (0.4ms)  SELECT  `users`.* FROM `users`  WHERE `users`.`id` = 39 LIMIT 1
  Namespace Load (0.2ms)  SELECT  `namespaces`.* FROM `namespaces`  WHERE `namespaces`.`id` = 246 LIMIT 1
Completed 500 Internal Server Error in 55ms

Rugged::TreeError (The path 'gradle.properties' does not exist in the given tree):
  app/controllers/projects/blame_controller.rb:11:in `new'
  app/controllers/projects/blame_controller.rb:11:in `show'

Error message made me guess if the file cannot be extracted from it's branch somehow. Is something not right with git implementation? Not sure.

I tried to push this branch into an empty repository: created new personal project, changed remote and pushed. Blame fails the same way. So it's reproducible (somehow).

Then I tried to make this branch master like this:

alexey@host:~/tmp$ git clone --branch dev --single-branch [email protected]:a-group/a-repository.git
Cloning into 'a-repository'...
remote: Counting objects: 82742, done.
remote: Compressing objects: 100% (18633/18633), done.
remote: Total 82742 (delta 51603), reused 82429 (delta 51402)
Receiving objects: 100% (82742/82742), 70.15 MiB | 1.84 MiB/s, done.
Resolving deltas: 100% (51603/51603), done.
Checking connectivity... done.
alexey@host:~/tmp$ mv a-repository pesky-blame
alexey@host:~/tmp$ cd pesky-blame/
alexey@host:~/tmp/pesky-blame$ git branch -m dev master
alexey@host:~/tmp/pesky-blame$ git remote rm origin                                                            
alexey@host:~/tmp/pesky-blame$ git remote add origin [email protected]:alexey/pesky-blame.git
alexey@host:~/tmp/pesky-blame$ git push -u origin master
Counting objects: 82742, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (18432/18432), done.
Writing objects: 100% (82742/82742), 70.15 MiB | 1.96 MiB/s, done.
Total 82742 (delta 51603), reused 82742 (delta 51603)
To [email protected]:alexey/pesky-blame.git
 * [new branch]      master -> master
Branch master set up to track remote branch master from origin.

This time I've got another error:

git@git:~/gitlab/log$ tail -n 0 -f production.log
Started GET "/alexey/pesky-blame/blame/master/gradle.properties" for 127.0.0.1 at 2014-12-05 15:10:28 +0300
Processing by Projects::BlameController#show as HTML
  Parameters: {"project_id"=>"alexey/pesky-blame", "id"=>"master/gradle.properties"}
  User Load (0.3ms)  SELECT  `users`.* FROM `users`  WHERE `users`.`id` = 3  ORDER BY `users`.`id` ASC LIMIT 1
  Namespace Load (0.2ms)  SELECT  `namespaces`.* FROM `namespaces`  WHERE `namespaces`.`path` = 'alexey' LIMIT 1
  Project Load (0.2ms)  SELECT  `projects`.* FROM `projects`  WHERE `projects`.`namespace_id` = 3 AND `projects`.`path` = 'pesky-blame' LIMIT 1
  ProjectMember Load (0.2ms)  SELECT  `members`.* FROM `members`  WHERE `members`.`source_type` = 'Project' AND `members`.`type` IN ('ProjectMember') AND `members`.`source_id` = 959 AND `members`.`source_type` = 'Project' AND `members`.`user_id` = 3 LIMIT 1
  Group Load (0.3ms)  SELECT  `namespaces`.* FROM `namespaces`  WHERE `namespaces`.`type` IN ('Group') AND `namespaces`.`id` = 3 AND `namespaces`.`type` = 'Group' LIMIT 1
  Namespace Load (0.2ms)  SELECT  `namespaces`.* FROM `namespaces`  WHERE `namespaces`.`id` = 3 LIMIT 1
  User Load (0.2ms)  SELECT  `users`.* FROM `users`  WHERE `users`.`id` = 3 LIMIT 1
^C

It's shorter this time as unicorn process crashed and logged this:

git@git:~/gitlab/log$ tail -n 0 -f unicorn.stderr.log
unicorn_rails worker[3] -D -c /home/git/gitlab/config/unicorn.rb -E production: /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rugged-0.21.0/vendor/libgit2/src/commit.c:478: git_commit_tree: Assertion `commit' failed.
E, [2014-12-05T15:11:05.874306 #20484] ERROR -- : reaped #<Process::Status: pid 19959 SIGABRT (signal 6)> worker=3
I, [2014-12-05T15:11:05.961757 #21789]  INFO -- : worker=3 ready

It kinda makes me believe that I did have a glimpse of some git implementation problem. But this report is still not detailed enough. Is there a way to make unicorn log even more verbose? I tried to squeeze more info but did not succeed. Not experienced enough in debugging ruby :(

I'd like to provide a test repo. But I cannot provide access to copy of this particular one. It potentially contains sensitive data. And I was not able to reproduce this error blindly. Please advice on further debugging steps.

Setup/installation info

GitLab 7.5.1 36679b57

git@git:~/gitlab$ bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
Checking Environment ...

Git configured for git user? ... yes

Checking Environment ... Finished

Checking GitLab Shell ...

GitLab Shell version >= 2.2.0 ? ... OK (2.2.0)
Repo base directory exists? ... yes
Repo base directory is a symlink? ... no
Repo base owned by git:git? ... yes
Repo base access is drwxrws---? ... yes
Satellites access is drwxr-x---? ... yes
hooks directories in repos are links: ... 
(all OK, some empty)

Running /home/git/gitlab-shell/bin/check
Check GitLab API access: OK
Check directories and files: 
        /home/git/repositories: OK
        /home/git/.ssh/authorized_keys: OK
Test redis-cli executable: redis-cli 2.4.14
Send ping to redis server: PONG
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking LDAP ...

LDAP users with access to your GitLab server (only showing the first 100 results)
Server: ldap
(listing goes)

Checking LDAP ... Finished

Checking GitLab ...

Database config exists? ... yes
Database is SQLite ... no
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Init script exists? ... yes
Init script up-to-date? ... yes
projects have namespace: ... 
(all yes)

Projects have satellites? ...
(all OK, some empty)

Redis version >= 2.0.0? ... yes
Ruby version >= 2.0.0 ? ... yes (2.1.2)
Your git bin path is "/usr/bin/git"
Git version >= 1.7.10 ? ... yes (1.7.10)

Checking GitLab ... Finished
System information
System:         Debian 7.7
Current User:   git
Using RVM:      no
Ruby Version:   2.1.2p95
Gem Version:    2.2.2
Bundler Version:1.6.3
Rake Version:   10.3.2
Sidekiq Version:2.17.0

GitLab information
Version:        7.5.1
Revision:       36679b5
Directory:      /home/git/gitlab
DB Adapter:     mysql2
URL:            https://git.fqdn.co
HTTP Clone URL: https://git.fqdn.co/some-project.git
SSH Clone URL:  [email protected]:some-project.git
Using LDAP:     yes
Using Omniauth: no

GitLab Shell
Version:        2.2.0
Repositories:   /home/git/repositories/
Hooks:          /home/git/gitlab-shell/hooks/
Git:            /usr/bin/git