flaky-test.md

Flaky Test

Cause of the problem

Date Timezone

Timezone

When your system is support timezone, usually it will become complicated.

• Timezone in DB
• Timezone in App
• Timezone on machine

midnight/BST

When BST start or end you will lose one hour and when exactly should be the beginning_of_day will be very different

Timecop.freeze('2024-03-11 00:50 +1') do Time.zone.now.beginning_of_day end #=> Sun, 10 Mar 2024 00:00:00.000000000 UTC +00:00

• BST start
• BST end

Leak

when wrap the test cases, if not careful some setup might not reset and will affect the following tests

Leak of global setup: Rails.cache, ENV, and so on

ENV[my_var]=value this will set globally Rails.cache=MemoryStore

Leak of mock: Timecop not reset

• Timecop forget return after freeze
• Allow s3 access forget resetting

before do
  Time.freeze(1.day.ago)
end

# after is missing
after do
...

Leak of function defined in rake tasks

namespace('my') do
  private
  def myfunc
    puts 'A'
  end
end

namespace('my2') do
  private
  def myfunc
    puts 'B'
  end
end

The function myfunc will be overwritten by the second one. When testing rake tasks in spec, you need to be very careful about this.

Rake task duplicate invokeing

Rails.application.load_tasks

When called agian, task.invoke will trigger the task twice.

DB

The data in daily life might be very different from the fixtures we used for testing

• id starts from 1
• date time would be the same
• table joins
• result ordering

Check different items, actually concrete and abstract. When comparing created_at one is from abstract the other is from concrete class tables.

DB order

usually date will be different not in test it’s very common to created at the same time and date filled in at the same time. So there should be a second column to rely on to sort.

The following is not alwasy true

Book.first == Book.order(:id).first

which one created first

who submitted first

select id, submitted_at from tests order by submitted_at

select id, submitted_at from tests order by submitted_at, id

It's very common to have same submitted_at in test cases.

7. Wrong instance passed in Book.find(comic) Book.find(book) because usually record start from 1, so both commit or book might have same id 1

8. Polymorphic leaking Author.book Author.comic books and comics table could have same id but for author it’s two different items if type in Author not specified it will give the wrong one.

Article -> Playable -> Audio/Video playable_type: 'Audio' playable_id: 1 playable_type: 'Video' playable_id: 1

select articles.id from articles join audios ON audios.id=articles.playable_id

It will return articles with video as well.

Feature Test

Test timeout

when waiting for element to be rendered.

Capybara.default_max_wait_time

You can specifiy custom time for each test case if you are dealing with really slow ones. But there is a tradeoff between duration of the CI running and the flaky test which blocking the deployment.

Retry to the rescue?

retry will help us deal with onetime problems like db order, connection failure or third party failure and so on. But it might hide some potential bugs so please be careful about that.

Retry failed ones only

Retry immediatly for 3 times

How to Debug

circle CI debugging

CircleCI support ssh directly to the instance to debug This is very helpful escpecially when ENV for CI and local are very different.

combine test cases

sequence of the runned test cases might create flakiness

combine test files

sequence of test file might create flakiness especially when something is leaked.

artifacts from CI

• screenshot
• html
• record console output

Rails log

Runtime

For feature test most of flaky are caused by slowness.