[Feature Request]: Add hotkey shuffle/fanout support to BeamSQL

[Polber]

What would you like to happen?

Currently BeamSQL does not have a way to reshuffle data that contains hotkeys during sub-transforms. For example, if a pipeline performs a JOIN operation as part of a SQLTransform, there is a good chance that a given key contains a disproportionate amount of data when it is fed into the GroupByKey that is performed as part of the expanded JOIN operation. Adding hotkey detection and reshuffling/fanout similar to Combine.PerKey.withHotKeyFanout should greatly increase performance.

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[github-actions]

This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.

[github-actions]

This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.