ATF scrict mode

ATF_Stirinct_Mode.md

Currently there is issue on ATF with Times calls.

Problem illustration

Times(0) does not prolongues test execution for waiting not expected event. So not expected RPC comes in scope of next test case and ATF ignores it.

Note Currently ATF ignores RPC that cames from mobile and do not expected

Same situation is for Times(N). SDL can loose some RPC's that will exceed expectations, because will start execute next test case.

Proposed solution

Proposition is to add optional strict mode to ATF.

In that mode ATF will fail test case in case if any not expected data received from SDL.

Strict mode should be enabled in script

Test:TestCasePrecindition() 
  Test:StrictMode(true)
end

Test:TestCasePostContifion() 
  Test:StopSDL(common_steps::stop_sdl_flow)
  Test:StrictMode(false)
end

Disadvanteges

Test script will need to expect a lot of data that is not related to tested workflow.

Solution :

For spliting tested workflow and stubs in strict mode I propose to add ATF API IGNORE(RPC) It will add exceptions to strict mode. But any Other expectation should override IGNORE notation

Test:TestCasePrecindition() 
  Test:StrictMode(true)
  IGNORE("OnHMIStatus")
  IGNORE("OnHashChange")
  ...
end

Additional rescrictions

Notion of Times(0) should means that SDL does not wait for sime RPC until <Event> This event should be specified.

Possible events:

• Timeout (not clear what timeout should be and it will make test scripts too slow)
• SDL shutdown (by requirenments SDL should process add data before shutdown, so in case SDL was succesfully stopped and didn't send any not expected RPC, it won't send them after shutdown also 😄)
• Full Mobile-SDL-HMI RPC paththrow (requires not BlackBox testing, works only in singlethread SDL request processing mode)

My choice is SDL shutdown

@LuxoftAKutsan
I propose create IGNORE notation as a part of test module, so its call should be looks like Test:IGNORE("RPC")

@LuxoftAKutsan

1. I agree with VVeremjova, and propose next name (Test).IgnoreExpectation()
2. I think ignore notation should "work" only in one test step, not during all test script (Test case)
3. Please replace all "TestCase" with "Test step", Test case is another entity. (One atomic test script represents one test case)
4. It is few misprints in text:

Notion of Times(0) should means that SDL does not wait for sime RPC
Test:TestCasePrecindition
Disadvanteges