IGNITE-23799 : Calcite. Hash join. #11770

Vladsz83 · 2024-12-23T20:32:14Z

No description provided.

# Conflicts: # modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite/exec/rel/ExecutionTest.java # modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite/planner/JoinCommutePlannerTest.java

This reverts commit 3337e98.

...ain/java/org/apache/ignite/internal/processors/query/calcite/rule/HashJoinConverterRule.java

alex-plekhanov · 2025-07-10T16:27:53Z

...rc/test/java/org/apache/ignite/internal/processors/query/calcite/exec/rel/ExecutionTest.java


+    /** Tests 'Select e.id, e.name, d.name as dep_name from DEP d <JOIN TYPE> join EMP e on e.DEPNO = d.DEPNO'. */
+    @Test
+    public void testHashJoin() {


ExecutionTest already contains a lot of unsorted tests, maybe it's time to move join tests to another class. But we also have MergeJoinExecutionTest/NestedLoopJoinExecutionTest they look almost identical and looks like they cover more cases than testHashJoin test. Maybe it's better to extract common part and add new HashJoinExecutionTest class.

I think you are right. But this might be other tests refactoring ticket.

...internal/processors/query/calcite/integration/CalciteBasicSecondaryIndexIntegrationTest.java

...st/java/org/apache/ignite/internal/processors/query/calcite/planner/HashJoinPlannerTest.java

alex-plekhanov · 2025-07-10T17:22:03Z

...st/java/org/apache/ignite/internal/processors/query/calcite/planner/HashJoinPlannerTest.java

+    private static List<List<Object>> testJoinIsAppliedParameters() {
+        return F.asList(
+            F.asList("select t1.c1 from t1 %s join t1 t2 using(c1)", true),
+            F.asList("select t1.c1 from t1 %s join t1 t2 on t1.c1 = t2.c1", true),


Please add two columns test.
Also, in cases like t1.c1 = t2.c2 and t1.id = 1 second condition can be pushed down to table scan (at least for some types of join), perhaps these cases should be tested too.

There are two-column tests below. Yes, some scans are pushed down and join uses only one equi join pair.

...a/org/apache/ignite/internal/processors/query/calcite/planner/JoinColocationPlannerTest.java

...src/main/java/org/apache/ignite/internal/processors/query/calcite/exec/rel/HashJoinNode.java

# Conflicts: # modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite/exec/rel/ExecutionTest.java # modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite/planner/JoinCommutePlannerTest.java

# Conflicts: # modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite/planner/PlannerTest.java

…lcite_hash_join

# Conflicts: # modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/exec/RuntimeHashIndex.java # modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/exec/rel/NestedLoopJoinNode.java # modules/calcite/src/test/java/org/apache/ignite/testsuites/ExecutionTestSuite.java

# Conflicts: # modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/exec/LogicalRelImplementor.java # modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/rel/IgniteJoinInfo.java # modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/rule/MergeJoinConverterRule.java

...te/src/main/java/org/apache/ignite/internal/processors/query/calcite/rel/IgniteHashJoin.java

...rc/main/java/org/apache/ignite/internal/processors/query/calcite/exec/MappingRowHandler.java

...src/main/java/org/apache/ignite/internal/processors/query/calcite/exec/rel/HashJoinNode.java

alex-plekhanov · 2026-01-16T11:41:14Z

...src/main/java/org/apache/ignite/internal/processors/query/calcite/exec/rel/HashJoinNode.java

+                                RowT right = rightIt.next();
+
+                                if (nonEqCond != null && !nonEqCond.test(left, right))
+                                    continue;


We can block the thread for a long time here, not allowing other queries to proceed. We should leave after some count of rows and schedule the next task (the same for other join types).

But may block thread similary in many places. We could take known https://issues.apache.org/jira/browse/IGNITE-23960 later and adjust it. WDYT?

alex-plekhanov · 2026-01-16T12:36:21Z

...ain/java/org/apache/ignite/internal/processors/query/calcite/rule/HashJoinConverterRule.java

+            return false;
+
+        // IS NOT DISTINCT is currently not supported simultaneously with equi conditions.
+        if (joinInfo.hasMatchingNulls() && joinInfo.matchingNullsCnt() != joinInfo.pairs().size())


Can't understand how it worked for example for JoinIntegrationTest#testIsNotDistinctWithEquiConditionFrom (and other) it's two key pairs in the test (IS NOT DISTINCT FROM and EQUALS) and other join converter rules are disabled.

It didn't work. There were no these tests yet.

...src/main/java/org/apache/ignite/internal/processors/query/calcite/exec/RuntimeHashIndex.java

Vladsz83 added 15 commits December 18, 2024 15:55

planner test works

e478873

fixed tests. + memory tracking

2d43502

Merge branch 'master' into IGNITE-23799_calcite_hash_join

4e0f987

Minor manual review fixes

c30cb44

Minorities

e43c971

Merge branch 'master' into IGNITE-23799_calcite_hash_join

a8db16e

Merge branch 'master' into IGNITE-23799_calcite_hash_join

eb70474

Merge branch 'master' into IGNITE-23799_calcite_hash_join

2fd447a

fix

91c9fec

+IGNITE-24675, +IGNITE-23970, +IGNITE-24050

72987ea

+minor ai3 ticket

3337e98

Revert "+minor ai3 ticket"

48016a9

This reverts commit 3337e98.

Merge branch 'master' into IGNITE-23799_calcite_hash_join_v2

aba8467

+ lost HashJoin exec test

2acd9b3

alex-plekhanov reviewed Jul 10, 2025

View reviewed changes

Vladsz83 added 14 commits July 16, 2025 13:33

merged master

cc4526c

Merge branch 'master' into IGNITE-23799_calcite_hash_join_v2

719aaee

# Conflicts: # modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite/planner/PlannerTest.java

Merge branch 'IGNITE-23799_calcite_hash_join_v2' into IGNITE-23799_ca…

78d1882

…lcite_hash_join

review fixes

75eb1e3

raw

354937b

rawWithOldStore

d7fe527

raw with new storage. Some tests.

d493a91

self-review fixes

8ef4118

optimized inputs requesting

3fb141c

self-review fixes

a29c239

test fix

0d838bf

Merge branch 'master' into IGNITE-23799_calcite_hash_join

e4c4464

Vladsz83 added 3 commits January 16, 2026 15:06

+ master

9b2036b

allow mix with is not distinct

414f886

alex-plekhanov reviewed Jan 16, 2026

View reviewed changes

Vladsz83 added 4 commits January 17, 2026 17:43

review fixes

304c43a

refactor to checkJoinFinished() / tryToRequestInputs();

630a213

+ rows number processing limitation

e42f8ca

refactor join reschedule

f1da1c1

IGNITE-23799 : Calcite. Hash join. #11770

Are you sure you want to change the base?

IGNITE-23799 : Calcite. Hash join. #11770

Uh oh!

Conversation

Vladsz83 commented Dec 23, 2024

Uh oh!

Uh oh!

alex-plekhanov Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

Vladsz83 Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alex-plekhanov Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

Vladsz83 Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alex-plekhanov Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Vladsz83 Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alex-plekhanov Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Vladsz83 Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Vladsz83 Jan 17, 2026 •

edited

Loading