[zhaoyongjie]

Summary

Superset uses Pandas Dataframe for query results processing. Now, these Dataframe between different Operators only support 1-dimensional Dataframe. This has caused confusion in some calculations. This PR introduces MultiIndex Dataframe for Operators. This is useful for simplifying calculations.

Current PostProcessing

Typical PostProcessing Operators are used in time-series charts for getting queries. e.g.:

        post_processing: [
          resampleOperator(formData, baseQueryObject),
          timeCompareOperator(formData, baseQueryObject),
          sortOperator(formData, { ...baseQueryObject, is_timeseries: true }),
          pivotOperatorInRuntime,
          rollingWindowOperator(formData, baseQueryObject),
          contributionOperator(formData, baseQueryObject),
          prophetOperator(formData, baseQueryObject),
        ],

We face these challenges for existing designs:

1. Such query can't be adapted to both rolling calculation and time compared calculation. In the other words, rollingWindowOperator should before timeCompareOperator.
2. We also need to consider how to face series in Dataframe. for example, in the resample.py. We need to "flat" multidimensional Dataframe to adapt this one.

After Changed

The new operators support calculate on multidimensional Dataframe. So, operators in QueryObject can change its position at will. eg:

        post_processing: [
          pivotOperatorInRuntime,
          resampleOperator(formData, baseQueryObject),
          rollingWindowOperator(formData, baseQueryObject),
          timeCompareOperator(formData, baseQueryObject),
          flatOperator(formData, baseQueryObject),
          contributionOperator(formData, baseQueryObject),
          prophetOperator(formData, baseQueryObject),
        ],

For example, The compare() operator supports the Dataframe shape like the following:

                   count_metric    sum_metric
    country              UK US         UK US
    dttm
    2019-01-01            1  2          5  6
    2019-01-02            3  4          7  8

adding new operator flat.py, this new operator will transform the multidimensional dataframe to flatten dataframe.

The full examples:

    pivot_df = pp.pivot(
        df=multiple_metrics_df,
        index=["dttm"],
        columns=["country"],
        aggregates={
            "sum_metric": {"operator": "sum"},
            "count_metric": {"operator": "sum"},
        },
        flatten_columns=False,
        reset_index=False,
    )
    """
                   count_metric    sum_metric
    country              UK US         UK US
    dttm
    2019-01-01            1  2          5  6
    2019-01-02            3  4          7  8
    """
    compared_df = pp.compare(
        pivot_df,
        source_columns=["count_metric"],
        compare_columns=["sum_metric"],
        compare_type=PPC.DIFF,
        drop_original_columns=True,
    )
    """
               difference__count_metric__sum_metric
    country                                      UK US
    dttm
    2019-01-01                                    4  4
    2019-01-02                                    4  4
    """
    flat_df = pp.flat(compared_df)
    """
            dttm  difference__count_metric__sum_metric, UK  difference__count_metric__sum_metric, US
    0 2019-01-01                                         4                                         4
    1 2019-01-02                                         4                                         4
    """

TESTING INSTRUCTIONS

Let's use a new dataset to test.

1. download DailyDelhiClimateTrain.csv to your local.
2. import DailyDelhiClimateTrain into Superset
   a) click upload CSV to database on top blue plus sign of Superset
   b) fill in DailyDelhiClimateTrain on table name
   c) select DailyDelhiClimateTrain.csv
   d) fill in date in Parse Dates line
   e) click Save
3. open explore page and select DailyDelhiClimateTrain dataset
4. change viz type to Line Chart, notice that you should pick up echart version
5. drag date in X-Axis
6. drag meantemp in metrics and select max for aggregate
7. select a time range
   
8. select day as time grainularity
9. select a time shift in AA section 1 year ago and actual values
10. validate data in 2015 and data in 2014 are correct (use SQLLab to validate)
11. select different calculation types in time comparison and validate data
12. select different rolling types in rolling window and validate data
13. select different time granularity and validate data.
14. you can find some "time hole" and validate Resample

Daily PCT with cumsum

Monthly comparison with quarter sum rolling

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[codecov]

Codecov Report

Merging #19116 (b7bc340) into master (6083545) will decrease coverage by 0.00%.
The diff coverage is 84.67%.

@@            Coverage Diff             @@
##           master   #19116      +/-   ##
==========================================
- Coverage   66.65%   66.64%   -0.01%     
==========================================
  Files        1672     1674       +2     
  Lines       64611    64602       -9     
  Branches     6505     6498       -7     
==========================================
- Hits        43066    43057       -9     
  Misses      19862    19862              
  Partials     1683     1683              

Flag	Coverage Δ
hive	52.67% <32.29%> (+0.02%)	⬆️
javascript	51.31% <96.42%> (-0.02%)	⬇️
mysql	81.65% <81.25%> (-0.01%)	⬇️
postgres	81.70% <81.25%> (-0.01%)	⬇️
presto	52.51% <32.29%> (+0.02%)	⬆️
python	82.12% <81.25%> (-0.01%)	⬇️
sqlite	81.47% <81.25%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...superset-ui-core/src/query/types/PostProcessing.ts	100.00% <ø> (ø)
...src/BigNumber/BigNumberWithTrendline/buildQuery.ts	11.11% <ø> (+2.02%)	⬆️
...in-chart-echarts/src/MixedTimeseries/buildQuery.ts	0.00% <ø> (ø)
.../plugin-chart-echarts/src/Timeseries/buildQuery.ts	66.66% <0.00%> (ø)
superset/charts/schemas.py	99.33% <ø> (ø)
superset/utils/pandas_postprocessing/aggregate.py	90.90% <ø> (ø)
superset/utils/pandas_postprocessing/diff.py	100.00% <ø> (ø)
superset/utils/pandas_postprocessing/select.py	100.00% <ø> (ø)
superset/utils/pandas_postprocessing/sort.py	100.00% <ø> (ø)
superset/utils/pandas_postprocessing/geography.py	82.85% <25.00%> (ø)
... and 24 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6083545...b7bc340. Read the comment docs.

[villebro]

Really nice refactor! Code LGTM and makes the AA/post processing functionality more uniform. My main concern is the assumption that before we've run the pivot operation, the data is not indexed, but after the pivot operation the df is indexed. This could be slightly confusing for viz developers, as they need to be aware of this. But we can keep iterating on this later (maybe do a big breaking change on the post processing API on 3.0)

[villebro]

Just a thought (doesn't need to be done here): Should we introduce a separate operation index that sets the index without needing to do a full pivot along with aggregations on the metrics? Then we could perhaps call them set_index and flatten_index or something that clearly communicates that we're specifically changing the index.

[zhaoyongjie]

It seems like stack and unstack function in Pandas. I will add those in future.

[jinghua-qa]

/testenv up

[github-actions]

@jinghua-qa Ephemeral environment spinning up at http://18.237.75.154:8080. Credentials are admin/admin. Please allow several minutes for bootstrapping and startup.

[jinghua-qa]

Thank you for the detail test steps~ LGTM

[zhaoyongjie]

Really nice refactor! Code LGTM and makes the AA/post processing functionality more uniform. My main concern is the assumption that before we've run the pivot operation, the data is not indexed, but after the pivot operation the df is indexed. This could be slightly confusing for viz developers, as they need to be aware of this. But we can keep iterating on this later (maybe do a big breaking change on the post processing API on 3.0)

I'm thinking that the sort operator could use on this case(may be), and if you have related examples I'd love to continue iterating.

[github-actions]

Ephemeral environment shutdown and build artifacts deleted.