Skip to content

Instantly share code, notes, and snippets.

@pgreze

pgreze/config.yml

Last active February 7, 2023 23:36
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save pgreze/dc5391050ed740e618b2a7fe27c56aee to your computer and use it in GitHub Desktop.
Save pgreze/dc5391050ed740e618b2a7fe27c56aee to your computer and use it in GitHub Desktop.
Download ZIP
Flank presentation
Raw
config.yml
gcloud:
results-history-name: Flank for app_jp
timeout: 45m
record-video: true
use-orchestrator: true
test-targets:
- size large
environment-variables:
clearPackageData: true
device:
- model: flame
version: 29
orientation: portrait
locale: ja_JP
- model: beyond1
version: 28
orientation: portrait
locale: ja_JP
- model: G8142
version: 26
orientation: portrait
locale: ja_JP
- model: hwALE-H
version: 21
orientation: portrait
locale: ja_JP
flank:
## test shards - the amount of groups to split the test suite into
max-test-shards: 8
## shard time - the amount of time tests within a shard should take
## when set to > 0, the shard count is dynamically set based on time up to the maximum limit defined by max-test-shards
## 2 minutes (120) is recommended.
## default: -1 (unlimited)
shard-time: 240
## test runs - the amount of times to run the tests.
num-test-runs: 1
## The billing enabled Google Cloud Platform project name to use
project: my-firebase-project
## The max time this test run can execute before it is cancelled (default: unlimited).
run-timeout: 180m
## Terminate with exit code 0 when there are failed tests. (default: false)
# ignore-failed-tests: true
## Enable create additional local junit result on local storage with failure nodes on passed flaky tests.
# full-junit-result: false
## Disables flank results upload on gcloud storage.
# disable-results-upload: false
Raw
gsutil.sh
$ gsutil ls gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/android_shards.json
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/session_id.txt
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/
$ gsutil cat gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/android_shards.json | head -n 28
{
"matrix-0": {
"app": "/tmp/tmp4891597321175276529.tmp",
"test": "/tmp/tmp5741042665540177315.tmp",
"shards": {
"shard-0": [
"class com.kouzoh.mercari.uitest.itemdetail.frictionless.FrictionlessActionButtonTest#onSaleItem_bystander_purchaseButton_notTapable",
"class com.kouzoh.mercari.uitest.itemdetail.frictionless.FrictionlessActionButtonTest#onSaleItem_seller_clickEditing_goToEditItemPage",
"class com.kouzoh.mercari.uitest.itemdetail.optimizesp.OptimizeSPTest#optimizeSP_smartPayButton_not_displayed_bystander_soldOut_item",
"class com.kouzoh.mercari.uitest.itemdetail.optimizesp.OptimizeSPTest#optimizeSP_smartPayButton_not_displayed_seller_soldOut_item",
"class com.kouzoh.mercari.uitest.personal_inventory.InventoryListTest#testEmptyList",
"class com.kouzoh.mercari.uitest.personal_inventory.InventoryListTest#testErrorDialog",
"class com.kouzoh.mercari.uitest.personal_inventory.MercariDetailsTest#viewCountExperimentIsNotDisplayedForControlGroup",
"class com.kouzoh.mercari.uitest.personal_inventory.MerpayDetailsTest#linkedProductOnMerpayDetails",
"class com.kouzoh.mercari.uitest.purchase.famipay.FamiPayTest#selectFamiPayInPaymentMethodFromMyPage"
],
"shard-1": [
"class com.kouzoh.mercari.uitest.itemdetail.frictionless.FrictionlessActionButtonTest#onSaleItem_bystander_clickPurchase_goToPurchasePage",
"class com.kouzoh.mercari.uitest.itemdetail.frictionless.FrictionlessActionButtonTest#soldOutItem_buyer_clickTrading_goToTradingPage",
"class com.kouzoh.mercari.uitest.itemdetail.optimizesp.OptimizeSPTest#optimizeSP_click_SP_button_should_change_payment_method_to_SmartPay",
"class com.kouzoh.mercari.uitest.itemdetail.relateditem.RelatedItemTest#checkRelatedItem",
"class com.kouzoh.mercari.uitest.location.LocationSelectTest#testDropOffGuide",
"class com.kouzoh.mercari.uitest.personal_inventory.InventoryListTest#testListingButtonABTest",
"class com.kouzoh.mercari.uitest.personal_inventory.MercariDetailsTest#viewCountExperimentIsDisplayedForNumberOfSimilarSearches",
"class com.kouzoh.mercari.uitest.personal_inventory.MerpayDetailsTest#testMerpayDetails",
"class com.kouzoh.mercari.uitest.purchase.famipay.FamiPayTest#selectFamiPayInPaymentMethodFromCheckout"
],
"shard-2": [
$ gsutil ls gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/G8142-26-ja_JP-portrait-test_results_merged.xml
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/crownlteks-28-ja_JP-portrait-test_results_merged.xml
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/flame-30-ja_JP-portrait-test_results_merged.xml
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/G8142-26-ja_JP-portrait-shard_0/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/G8142-26-ja_JP-portrait-shard_1/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/G8142-26-ja_JP-portrait-shard_2/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/G8142-26-ja_JP-portrait-shard_3/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/G8142-26-ja_JP-portrait-shard_4/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/G8142-26-ja_JP-portrait-shard_5/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/G8142-26-ja_JP-portrait-shard_6/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/G8142-26-ja_JP-portrait-shard_7/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/crownlteks-28-ja_JP-portrait-shard_0/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/crownlteks-28-ja_JP-portrait-shard_1/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/crownlteks-28-ja_JP-portrait-shard_2/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/crownlteks-28-ja_JP-portrait-shard_3/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/crownlteks-28-ja_JP-portrait-shard_4/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/crownlteks-28-ja_JP-portrait-shard_5/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/crownlteks-28-ja_JP-portrait-shard_6/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/crownlteks-28-ja_JP-portrait-shard_7/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/flame-30-ja_JP-portrait-shard_0/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/flame-30-ja_JP-portrait-shard_1/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/flame-30-ja_JP-portrait-shard_2/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/flame-30-ja_JP-portrait-shard_3/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/flame-30-ja_JP-portrait-shard_4/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/flame-30-ja_JP-portrait-shard_5/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/flame-30-ja_JP-portrait-shard_6/
gs://test-lab-xxx/2021-03-31_14-27-07.810000_tGMU/matrix_0/flame-30-ja_JP-portrait-shard_7/
Raw
logs.txt
WARNING: environment-variables set but directories-to-pull is empty, this will result in the coverage file not downloading to the bucket.
AndroidArgs
gcloud:
results-bucket: test-lab-xxx
results-dir: 2021-02-14_14-24-32.171000_MzWr
record-video: true
timeout: 45m
async: false
client-details:
network-profile: null
results-history-name: Flank for app_jp
# Android gcloud
app: gs://xxx/jp/dev/master/jp-dev-743086.apk
test: gs://xxx/jp/dev-test/master/jp-dev-test-743086.apk
additional-apks:
auto-google-login: false
use-orchestrator: true
directories-to-pull:
grant-permissions: all
type: null
other-files:
scenario-numbers:
scenario-labels:
performance-metrics: false
num-uniform-shards: null
test-runner-class: null
test-targets:
- size large
robo-directives:
robo-script: null
device:
- model: flame
version: 29
locale: ja_JP
orientation: portrait
- model: beyond1
version: 28
locale: ja_JP
orientation: portrait
- model: G8142
version: 26
locale: ja_JP
orientation: portrait
- model: hwALE-H
version: 21
locale: ja_JP
orientation: portrait
num-flaky-test-attempts: 0
flank:
max-test-shards: 8
shard-time: 240
num-test-runs: 1
smart-flank-gcs-path:
smart-flank-disable-upload: false
default-test-time: 120.0
use-average-test-time-for-new-tests: false
files-to-download:
test-targets-always-run:
disable-sharding: false
project: xxx
local-result-dir: results
full-junit-result: false
# Android Flank Yml
keep-file-path: false
additional-app-test-apks:
run-timeout: 180m
legacy-junit-result: false
ignore-failed-tests: false
output-style: multi
disable-results-upload: false
default-class-test-time: 240.0
RunTests
Smart Flank cache hit: 0% (0 / 298)
Shard times: 4440s, 4440s, 4440s, 4440s, 4440s, 4440s, 4560s, 4560s
Saved 1 shards to /root/code/android_shards.json
Uploading android_shards.json .
298 tests / 8 shards
1 matrix ids created in 0m 12s
https://console.developers.google.com/storage/browser/test-lab-xxx/xxx
Matrices webLink
matrix-2vthedobj4itm https://console.firebase.google.com/project/fury-panda/testlab/histories/bh.xxx/matrices/xxx
0m 0s matrix-2vthedobj4itm beyond1-28 shard-3 PENDING
0m 0s matrix-2vthedobj4itm hwALE-H-21 shard-4 PENDING
0m 0s matrix-2vthedobj4itm beyond1-28 shard-1 PENDING
0m 0s matrix-2vthedobj4itm G8142-26 shard-0 PENDING
0m 0s matrix-2vthedobj4itm G8142-26 shard-4 PENDING
0m 0s matrix-2vthedobj4itm G8142-26 shard-2 PENDING
0m 0s matrix-2vthedobj4itm flame-29 shard-3 PENDING
0m 0s matrix-2vthedobj4itm flame-29 shard-0 PENDING
0m 0s matrix-2vthedobj4itm flame-29 shard-1 PENDING
0m 0s matrix-2vthedobj4itm hwALE-H-21 shard-0 PENDING
0m 0s matrix-2vthedobj4itm G8142-26 shard-5 PENDING
...
0m 53s matrix-2vthedobj4itm G8142-26 shard-2 Preparing device.
0m 53s matrix-2vthedobj4itm flame-29 shard-3 RUNNING
0m 53s matrix-2vthedobj4itm flame-29 shard-0 RUNNING
0m 53s matrix-2vthedobj4itm flame-29 shard-1 RUNNING
0m 53s matrix-2vthedobj4itm hwALE-H-21 shard-0 Preparing device.
0m 53s matrix-2vthedobj4itm G8142-26 shard-5 RUNNING
0m 42s matrix-2vthedobj4itm hwALE-H-21 shard-7 Preparing device.
0m 53s matrix-2vthedobj4itm beyond1-28 shard-6 Preparing device.
0m 47s matrix-2vthedobj4itm beyond1-28 shard-4 RUNNING
0m 53s matrix-2vthedobj4itm beyond1-28 shard-5 RUNNING
0m 0s matrix-2vthedobj4itm G8142-26 shard-7 PENDING
0m 53s matrix-2vthedobj4itm flame-29 shard-4 Installing apps.
...
29m 59s matrix-2vthedobj4itm beyond1-28 shard-0 Completed results processing. Time taken = 113 (secs)
4m 8s matrix-2vthedobj4itm hwALE-H-21 shard-2 Starting instrumentation test.
28m 6s matrix-2vthedobj4itm flame-29 shard-6 FINISHED
5m 0s matrix-2vthedobj4itm G8142-26 shard-1 Starting instrumentation test.
28m 38s matrix-2vthedobj4itm flame-29 shard-7 FINISHED
4m 24s matrix-2vthedobj4itm G8142-26 shard-6 Starting instrumentation test.
28m 44s matrix-2vthedobj4itm beyond1-28 shard-7 Starting results processing. Attempt: 1
...
31m 29s matrix-2vthedobj4itm G8142-26 shard-7 FINISHED
30m 47s matrix-2vthedobj4itm flame-29 shard-4 FINISHED
29m 59s matrix-2vthedobj4itm beyond1-28 shard-0 Completed results processing. Time taken = 113 (secs)
46m 21s matrix-2vthedobj4itm hwALE-H-21 shard-2 FINISHED
28m 6s matrix-2vthedobj4itm flame-29 shard-6 FINISHED
33m 50s matrix-2vthedobj4itm G8142-26 shard-1 FINISHED
33m 1s matrix-2vthedobj4itm beyond1-28 shard-2 FINISHED
49m 44s matrix-2vthedobj4itm FINISHED
FetchArtifacts
................................
Updating matrix file
CostReport
Physical devices
$79.08 for 15h 49m
Uploading CostReport.txt .
MatrixResultsReport
0 / 1 (0.00%)
1 matrices failed
┌─────────┬──────────────────────┬───────────────────────────┬──────────────────────────────────┐
│ OUTCOME │ MATRIX ID │ TEST AXIS VALUE │ TEST DETAILS │
├─────────┼──────────────────────┼───────────────────────────┼──────────────────────────────────┤
│ failure │ matrix-xxxxxxxxxxxxx │ flame-29-ja_JP-portrait │ 21 test cases failed, 277 passed │
│ failure │ matrix-xxxxxxxxxxxxx │ hwALE-H-21-ja_JP-portrait │ Application crashed │
│ failure │ matrix-xxxxxxxxxxxxx │ beyond1-28-ja_JP-portrait │ Application crashed │
│ failure │ matrix-xxxxxxxxxxxxx │ G8142-26-ja_JP-portrait │ 55 test cases failed, 243 passed │
└─────────┴──────────────────────┴───────────────────────────┴──────────────────────────────────┘
More details are available at:
https://console.firebase.google.com/project/fury-panda/testlab/histories/bh.547d9d6541000300/matrices/7452810494679329774
Uploading MatrixResultsReport.txt .
Uploading HtmlErrorReport.html .
Uploading JUnitReport.xml .
Matrices webLink
matrix-2vthedobj4itm https://console.firebase.google.com/project/fury-panda/testlab/histories/bh.xxx/matrices/xxx
Raw
run.sh
$ curl -L https://github.com/Flank/flank/releases/download/v21.03.1/flank.jar -o flank.jar
$ java -jar flank.jar
flank.jar
[-v] [--debug] [COMMAND]
--debug Enables debug logging
-v, --version Prints the version
Commands:
firebase
ios
android
refresh Downloads results for the last Firebase Test Lab run
cancel Cancels the last Firebase Test Lab run
auth Manage oauth2 credentials for Google Cloud
provided-software
network-profiles Explore network profiles available for testing.
ip-blocks Explore IP blocks used by Firebase Test Lab devices.
Raw
smart-logs.txt
...
flank:
max-test-shards: 100
shard-time: 120
num-test-runs: 1
smart-flank-gcs-path: gs://test-lab-xxx/xxx/JUnitReport.xml
...
RunTests
Smart Flank cache hit: 100% (72 / 72)
Shard times: 110s, 110s, 110s, 111s, 111s, 112s, 112s, 112s, 112s, 112s, 112s, 121s, 121s, 121s
Saved 1 shards to .../results/2021-04-05_08-43-13.701850_OYSM/android_shards.json
Uploading [android_shards.json] to https://console.developers.google.com/storage/browser/test-lab-xxx/2021-04-05_08-43-13.701850_OYSM/...
72 tests / 14 shards
...
Actual shard times:
Shard 0: Expected: 110s, Actual: 127s, Diff: 16s
Shard 1: Expected: 110s, Actual: 127s, Diff: 17s
Shard 2: Expected: 110s, Actual: 130s, Diff: 19s
Shard 3: Expected: 111s, Actual: 125s, Diff: 14s
Shard 4: Expected: 111s, Actual: 131s, Diff: 20s
Shard 5: Expected: 112s, Actual: 135s, Diff: 23s
Shard 6: Expected: 112s, Actual: 133s, Diff: 21s
Shard 7: Expected: 112s, Actual: 134s, Diff: 22s
Shard 8: Expected: 112s, Actual: 143s, Diff: 31s
Shard 9: Expected: 112s, Actual: 131s, Diff: 19s
Shard 10: Expected: 112s, Actual: 136s, Diff: 24s
Shard 11: Expected: 121s, Actual: 139s, Diff: 18s
Shard 12: Expected: 121s, Actual: 141s, Diff: 20s
Shard 13: Expected: 121s, Actual: 139s, Diff: 18s
Uploading [smart flank XML] to https://console.developers.google.com/storage/browser/test-lab-xxx/2021-04-05_07-22-33.519000_htQO/JUnitReport.xml...
Uploading [matrix_ids.json] to https://console.developers.google.com/storage/browser/test-lab-xxx/2021-04-05_08-43-13.701850_OYSM/...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment