GitLab

Created by: stanhu

The master branch has been failing on Semaphore over the last week due to two failures (see: https://semaphoreci.com/gitlabhq/gitlabhq/branches/master/builds/1552):

rspec ./spec/requests/api/groups_spec.rb:37 # API::API GET /groups when authenticated as  admin admin: should return an array of all groups
rspec ./spec/requests/api/namespaces_spec.rb:19 # API::API GET /namespaces when authenticated as admin admin: should return an array of all namespaces

It looks like this is an issue with the database not being cleaned out properly. When I modified one of the specs to clean the database with a full truncation, the test passed fine.

I was able to reproduce this same failure on PostgreSQL 9.4.4, but not on 9.3.5, which GitLab normally uses. Semaphore CI uses 9.4.1. These failures only seem to happen when you run the entire rspec test suite (e.g. rspec spec). If you just run a subset of tests, then the tests pass fine. Note that the tests pass fine on GitLab CI and locally with 9.3.5.

I added debugging to watch the state of the number of groups and namespaces before and after in db_cleaner.rb. It looks like the ROLLBACK commands after some test cases don't seem to be wiping the database clean. For example, in the namespace spec that failed, you can see the state of the namespace and group counts never really get reset after DatabaseCleaner.clean:

=== Cleaning end: namespace count = 77, groups count = 15
=== Cleaning end: namespace count = 81, groups count = 15
=== Cleaning end: namespace count = 81, groups count = 15
=== Cleaning end: namespace count = 77, groups count = 17
!!! Getting namespaces
=== Cleaning end: namespace count = 78, groups count = 17
=== Cleaning end: namespace count = 78, groups count = 17
=== Cleaning end: namespace count = 78, groups count = 17
=== Cleaning end: namespace count = 78, groups count = 17
=== Cleaning end: namespace count = 77, groups count = 15

More specifically, you can see the count doesn't get reset in this case:

=== Cleaning end: namespace count = 5, groups count = 1
  ^[[1m^[[35m (0.2ms)^[[0m  ROLLBACK
  ^[[1m^[[36m (0.3ms)^[[0m  ^[[1mSELECT COUNT(*) FROM "namespaces"^[[0m
  ^[[1m^[[35m (0.3ms)^[[0m  SELECT COUNT(*) FROM "namespaces"  WHERE "namespaces"."type" IN ('Group')
=== Done cleaning end: namespace count = 4, groups count = 1                              

One odd thing is that I do see the results from mail being sent right before the test case that fials:

Notify#group_access_granted_email: processed outbound mail in 14.1ms` spec 


Sent mail to [email protected] (2.9ms)
Date: Mon, 27 Jul 2015 23:26:17 -0700
From: GitLab <gitlab@localhost>
Reply-To: noreply@localhost
<snip>
----==_mimepart_55b720895f012_16fa23fc778c65bb84141a9--^M

  ^[[1m^[[36mSystemHook Load (0.5ms)^[[0m  ^[[1mSELECT "web_hooks".* FROM "web_hooks"  WHERE "web_hooks"."type" IN ('SystemHook')  ORDER BY "web_hooks"."created_at" DESC, "web_hooks"."id" DESC^[[0m
  ^[[1m^[[35m (0.2ms)^[[0m  RELEASE SAVEPOINT active_record_1
!!! Getting /groups

I wonder if this mail is throwing the DB checkpoints off. That may just be a red herring, though.