Skip to content

Instantly share code, notes, and snippets.

@dgaubert

dgaubert/e2e_tests_enriched_vs_develop.md

Created February 21, 2020 18:25
Show Gist options
  • Save dgaubert/811d3732139a4a3d4835248663510c0d to your computer and use it in GitHub Desktop.
Save dgaubert/811d3732139a4a3d4835248663510c0d to your computer and use it in GitHub Desktop.
Download ZIP
Raw
e2e_tests_enriched_vs_develop.md

Test: tests/e2e/data/observatory/enrichment/test_enrichment.py::TestEnrichment::test_points_and_private_data

Test is failing in both branches because of the same reason: do_area area doesn't match between enriched vs expected. The easiest solution is to update fixture to match with current enriched reault.

  • Branch enrichment

    • Enriched DataFrame

          BLOCKGROUP  MLTCY7224  RSGCY7224       do_area                    geometry
0  170312604001      20020      20020  3.276807e+05  POINT (-87.73692 41.88582)
1  170312604001      20020      20020  3.276807e+05  POINT (-87.73682 41.88592)
2  551010021003      39680     -99700  2.896988e+07  POINT (-88.13782 42.67032)
3  172010043003     132348     132348  1.663832e+08  POINT (-89.35730 42.21631)
4  170630006003      47627      47627  2.808913e+08  POINT (-88.52783 41.14143)

    • Expected DataFrame

          BLOCKGROUP  MLTCY7224  RSGCY7224       do_area                    geometry
0  170312604001      20020      20020  3.276807e+05  POINT (-87.73692 41.88582)
1  170312604001      20020      20020  3.276807e+05  POINT (-87.73682 41.88592)
2  551010021003      39680     -99700  2.896988e+07  POINT (-88.13782 42.67032)
3  172010043003     132348     132348  1.657655e+08  POINT (-89.35730 42.21631)
4  170630006003      47627      47627  2.807869e+08  POINT (-88.52783 41.14143)

  • Branch develop

    • Enriched DataFrame

          BLOCKGROUP  MLTCY7224  RSGCY7224       do_area                    geometry
0  170312604001      20020      20020  3.276807e+05  POINT (-87.73692 41.88582)
1  170312604001      20020      20020  3.276807e+05  POINT (-87.73682 41.88592)
2  551010021003      39680     -99700  2.896988e+07  POINT (-88.13782 42.67032)
3  172010043003     132348     132348  1.663832e+08  POINT (-89.35730 42.21631)
4  170630006003      47627      47627  2.808913e+08  POINT (-88.52783 41.14143)

    • Expected DataFrame

          BLOCKGROUP  MLTCY7224  RSGCY7224       do_area                    geometry
0  170312604001      20020      20020  3.276807e+05  POINT (-87.73692 41.88582)
1  170312604001      20020      20020  3.276807e+05  POINT (-87.73682 41.88592)
2  551010021003      39680     -99700  2.896988e+07  POINT (-88.13782 42.67032)
3  172010043003     132348     132348  1.657655e+08  POINT (-89.35730 42.21631)
4  170630006003      47627      47627  2.807869e+08  POINT (-88.52783 41.14143)

Test: tests/e2e/data/observatory/enrichment/test_enrichment.py::TestEnrichment::test_points_public_data_and_filters

Test is failing in both branches because of the same reason: do_area area doesn't match between enriched vs expected. The easiest solution is to update fixture to match with current enriched reault.

  • Branch enrichment

    • Enriched DataFrame

              do_area         geoid                    geometry  one_car  poverty
0           NaN           NaN  POINT (-87.73692 41.88582)      NaN      NaN
1           NaN           NaN  POINT (-87.73682 41.88592)      NaN      NaN
2  8.894818e+07  5.510100e+10  POINT (-88.13782 42.67032)    301.0    167.0
3  1.843876e+08  1.720100e+10  POINT (-89.35730 42.21631)    547.0    175.0
4           NaN           NaN  POINT (-88.52783 41.14143)      NaN      NaN

    • Expected DataFrame

              do_area        geoid                    geometry  one_car  poverty
0           NaN         None  POINT (-87.73692 41.88582)      NaN      NaN
1           NaN         None  POINT (-87.73682 41.88592)      NaN      NaN
2  8.894818e+07  55101002100  POINT (-88.13782 42.67032)    301.0    167.0
3  1.837390e+08  17201004300  POINT (-89.35730 42.21631)    547.0    175.0
4           NaN         None  POINT (-88.52783 41.14143)      NaN      NaN

  • Branch develop

    • Enriched DataFrame

              do_area        geoid                    geometry  one_car  poverty
0           NaN          NaN  POINT (-87.73692 41.88582)      NaN      NaN
1           NaN          NaN  POINT (-87.73682 41.88592)      NaN      NaN
2  8.894818e+07  55101002100  POINT (-88.13782 42.67032)    301.0    167.0
3  1.843876e+08  17201004300  POINT (-89.35730 42.21631)    547.0    175.0
4           NaN          NaN  POINT (-88.52783 41.14143)      NaN      NaN

    • Expected DataFrame

              do_area        geoid                    geometry  one_car  poverty
0           NaN         None  POINT (-87.73692 41.88582)      NaN      NaN
1           NaN         None  POINT (-87.73682 41.88592)      NaN      NaN
2  8.894818e+07  55101002100  POINT (-88.13782 42.67032)    301.0    167.0
3  1.837390e+08  17201004300  POINT (-89.35730 42.21631)    547.0    175.0
4           NaN         None  POINT (-88.52783 41.14143)      NaN      NaN

Test: tests/e2e/data/observatory/enrichment/test_enrichment.py::TestEnrichment::test_polygons_private_data_and_agg_none

Test is failing in both branches because of different reason: in enrichemnt are we are aggregating the dataframe despite of we sent aggregation NULL. We need to fix that first, then figure out why this test is failing in develop as the ouput for both enriched and expected are the same.

  • Branch enrichment

    • Enriched DataFrame

                                                  geometry   one_car  poverty
0  POLYGON ((-87.83844 41.83299, -87.75021 41.807...  1058.649  708.714

    • Expected DataFrame

              do_area     do_geoid        geoid                                           geometry  intersected_area  one_car  poverty     user_area
0   1883075.522  17031815701  17031815701  POLYGON ((-87.83844 41.83299, -87.75021 41.807...        301279.057    397.0    101.0  1.102272e+07
1   1982064.816  17031815800  17031815800  POLYGON ((-87.83844 41.83299, -87.75021 41.807...         55552.649    220.0    107.0  1.102272e+07
2    653408.168  17031560300  17031560300  POLYGON ((-87.83844 41.83299, -87.75021 41.807...          2165.794    314.0    279.0  1.102272e+07
3    778455.974  17031815300  17031815300  POLYGON ((-87.83844 41.83299, -87.75021 41.807...         30040.920    374.0    297.0  1.102272e+07
4   1156221.896  17031814500  17031814500  POLYGON ((-87.83844 41.83299, -87.75021 41.807...        835810.219    410.0    298.0  1.102272e+07
5    661587.632  17031560200  17031560200  POLYGON ((-87.83844 41.83299, -87.75021 41.807...        160095.515    213.0    351.0  1.102272e+07
6   3350339.209  17031815702  17031815702  POLYGON ((-87.83844 41.83299, -87.75021 41.807...       1437894.643    701.0    400.0  1.102272e+07
7   4972527.191  17031820700  17031820700  POLYGON ((-87.83844 41.83299, -87.75021 41.807...       2539565.470    716.0    475.0  1.102272e+07
8   5695045.378  17031820800  17031820800  POLYGON ((-87.83844 41.83299, -87.75021 41.807...        665047.899    351.0    484.0  1.102272e+07
9   1446863.928  17031815200  17031815200  POLYGON ((-87.83844 41.83299, -87.75021 41.807...        189613.313    617.0    854.0  1.102272e+07
10   907071.423  17031814300  17031814300  POLYGON ((-87.83844 41.83299, -87.75021 41.807...        453221.562    368.0    926.0  1.102272e+07
11  2446805.274  17031814200  17031814200  POLYGON ((-87.83844 41.83299, -87.75021 41.807...        155633.289    548.0   1471.0  1.102272e+07

  • Branch develop

    • Enriched DataFrame

              do_area     do_geoid        geoid                                           geometry  intersected_area  one_car  poverty     user_area
0   1.883076e+06  17031815701  17031815701  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      3.012791e+05    397.0    101.0  1.102272e+07
1   1.982065e+06  17031815800  17031815800  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      5.555265e+04    220.0    107.0  1.102272e+07
2   6.534082e+05  17031560300  17031560300  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      2.165794e+03    314.0    279.0  1.102272e+07
3   7.784560e+05  17031815300  17031815300  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      3.004092e+04    374.0    297.0  1.102272e+07
4   1.156222e+06  17031814500  17031814500  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      8.358102e+05    410.0    298.0  1.102272e+07
5   6.615876e+05  17031560200  17031560200  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      1.600955e+05    213.0    351.0  1.102272e+07
6   3.350339e+06  17031815702  17031815702  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      1.437895e+06    701.0    400.0  1.102272e+07
7   4.972527e+06  17031820700  17031820700  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      2.539565e+06    716.0    475.0  1.102272e+07
8   5.695045e+06  17031820800  17031820800  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      6.650479e+05    351.0    484.0  1.102272e+07
9   1.446864e+06  17031815200  17031815200  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      1.896133e+05    617.0    854.0  1.102272e+07
10  9.070714e+05  17031814300  17031814300  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      4.532216e+05    368.0    926.0  1.102272e+07
11  2.446805e+06  17031814200  17031814200  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      1.556333e+05    548.0   1471.0  1.102272e+07

    • Expected DataFrame

              do_area     do_geoid        geoid                                           geometry  intersected_area  one_car  poverty     user_area
0   1.883076e+06  17031815701  17031815701  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      3.012791e+05    397.0    101.0  1.102272e+07
1   1.982065e+06  17031815800  17031815800  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      5.555265e+04    220.0    107.0  1.102272e+07
2   6.534082e+05  17031560300  17031560300  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      2.165794e+03    314.0    279.0  1.102272e+07
3   7.784560e+05  17031815300  17031815300  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      3.004092e+04    374.0    297.0  1.102272e+07
4   1.156222e+06  17031814500  17031814500  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      8.358102e+05    410.0    298.0  1.102272e+07
5   6.615876e+05  17031560200  17031560200  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      1.600955e+05    213.0    351.0  1.102272e+07
6   3.350339e+06  17031815702  17031815702  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      1.437895e+06    701.0    400.0  1.102272e+07
7   4.972527e+06  17031820700  17031820700  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      2.539565e+06    716.0    475.0  1.102272e+07
8   5.695045e+06  17031820800  17031820800  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      6.650479e+05    351.0    484.0  1.102272e+07
9   1.446864e+06  17031815200  17031815200  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      1.896133e+05    617.0    854.0  1.102272e+07
10  9.070714e+05  17031814300  17031814300  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      4.532216e+05    368.0    926.0  1.102272e+07
11  2.446805e+06  17031814200  17031814200  POLYGON ((-87.83844 41.83299, -87.75021 41.807...      1.556333e+05    548.0   1471.0  1.102272e+07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment