Fix MSQ compaction state and native interval locking, add test coverage by cecemei · Pull Request #18950 · apache/druid

cecemei · 2026-01-24T02:49:48Z

Description

This PR fixes several issues with MSQ compaction state and native compaction interval locking, and adds comprehensive test coverage for MSQ compaction tasks.

Fixed compaction state issues

Filter propagation: MSQ compaction runner now propagates filter to compaction state
Query granularity propagation: MSQ compaction runner now reads query granularity from inputs and propagates it to compaction state

Fixed native compaction interval locking

Native runner's index task now locks intervals from CompactionTask instead of existing segments' intervals

Added MSQCompactionTaskRunTest

New test suite covering MSQ compaction task execution
Tests various compaction scenarios including partition handling, granularity changes, and metrics aggregation

Refactored test infrastructure

Extracted CompactionTaskRunBase to support both native and MSQ compaction tests
Added NativeCompactionTaskRunTest to provide dedicated test coverage for native compaction
Created TestSpyTaskActionClient utility for better test observability

Added builder pattern support

Added toBuilder() method to CompactionState with a new Builder class
Added toBuilder() method to DimensionsSpec

Key changed/added classes in this PR

MSQCompactionTaskRunTest
NativeCompactionTaskRunTest
CompactionTaskRunBase
CompactionState.Builder
DimensionsSpec.toBuilder()
TestSpyTaskActionClient

This PR has:

processing/src/test/java/org/apache/druid/data/input/impl/DimensionsSpecSerdeTest.java

+    Assert.assertEquals(original.isIncludeAllDimensions(), rebuilt.isIncludeAllDimensions());
+    Assert.assertEquals(original.useSchemaDiscovery(), rebuilt.useSchemaDiscovery());
+    Assert.assertEquals(original.isForceSegmentSortByTimeConfigured(), rebuilt.isForceSegmentSortByTimeConfigured());
+    Assert.assertEquals(original.getSpatialDimensions(), rebuilt.getSpatialDimensions());


processing/src/test/java/org/apache/druid/data/input/impl/DimensionsSpecSerdeTest.java

+    Assert.assertEquals(original.isIncludeAllDimensions(), rebuilt.isIncludeAllDimensions());
+    Assert.assertEquals(original.useSchemaDiscovery(), rebuilt.useSchemaDiscovery());
+    Assert.assertEquals(original.isForceSegmentSortByTimeConfigured(), rebuilt.isForceSegmentSortByTimeConfigured());
+    Assert.assertEquals(original.getSpatialDimensions(), rebuilt.getSpatialDimensions());


capistrant

Thank you for taking the time to extend all the existing testing code to work with MSQ, this is awesome!

Left some comments, I think my main questions/concerns are around the locking change as well as MSQ and dropExisting enforcement

indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java

capistrant · 2026-01-30T15:01:56Z

indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionRunner.java

+   */
+  default boolean forceDropExisting()
+  {
+    return true;


Was this a nasty bug that MSQ compaction didn't enforce dropExisting being true in the past? cuz like I'm sure there are people using it in production and not setting their io config explicitly.. so I guess their tasks would now fail until they set the config? I always thought dropExisting was a native only config since MSQ always creates tombstones regardless of this value. But I guess if REPLACE_LEGACY is used to make any other decisions behind the scenes it could be doing something unexpected?

MSQ actually never read this config, but instead just run with dropExisting, even though it's set to false (which is also the default) in the config. I thought this would be somewhat confusing if ppl really look into what this field means.

So it's not a real change for compaction runner, the result would still be the same but instead we just want the config to be honest. I was also hesistant about this might break anything, so also considering maybe just revert this change in case it breaks something.

In my test cluster, my msq compact supervisor does start failing to create tasks after deploying your feature branch. I think this would be a sad change to force people into making.

Caused by: org.apache.druid.java.util.common.IAE: Invalid config: runner[class org.apache.druid.msq.indexing.MSQCompactionRunner] must run with dropExisting

yea i made the dropExisting change in another pr: https://github.com/apache/druid/pull/18968/changes#diff-280dab320bc1341105272f99d3ec8e7f8626c9d117f7affa933bc1a522bd465aL368, but the check here is probably not all that necessary. anyway i removed this check here so the task wont fail.

capistrant · 2026-01-30T15:42:09Z

indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java

  {
-    final List<DataSegment> segments = segmentProvider.findSegments(taskActionClient);
-    return determineLockGranularityAndTryLockWithSegments(taskActionClient, segments, segmentProvider::checkSegments);
+    return determineLockGranularityAndTryLock(taskActionClient, List.of(segmentProvider.interval));


Is this what you refer to in Native runner's index task now locks intervals from CompactionTask instead of existing segments' intervals? Isn't this code on the path for both native and MSQ compaction? that is at least how I interpret it. Not that I think that makes this wrong, but I'm trying to make sure I understand the scope of the change.

an aside, I think this would make determineLockGranularityAndTryLockWithSegments unused now. do we want to leave that around for future use or remove it?

Yes this is on both path. This would make a different say compaction is for a day, but maybe we only have segments for 1 hour. This actually broke msq runner since the task lock is not that smart to figure out it's just the same thing, but then i felt this might be just before concurrent append & replace times that we try to lock less, we should probably move things to the concurrent append & replace world.

removed determineLockGranularityAndTryLockWithSegments

I am not sure if it safe to remove this. IIUC, the original logic was to fetch all the segments that overlap (even if partially) with the semgentProvider.interval, then try to lock the umbrella interval of the segments. So the final interval locked may be bigger than the original interval. The idea is that a task will be able to replace only those segments which are fully contained in the interval that is locked by that task.

@cecemei , could you share some more details on the issue that you encountered with the MSQ runner and the hour granularity? Maybe there could be an alternative solution.

ah i was thinking the case when the umbrella interval is smaller than the original interval (which is the case in CompactionTaskRunBase, the segment is hourly and there's only segments in hour 0 - hour 3, so compaction task only locks for 3 hours, but the interval can be all day (see NativeCompactionTaskRunTest). MSQ runner failed when getting a task lock with the following change:

diff.patch

Cannot create a new taskLockPosse for request[TimeChunkLockRequest{lockType=REPLACE, groupId='compact_test_mgmlcmch_2026-02-03T18:37:51.661Z', dataSource='test', interval=2014-01-01T00:00:00.000Z/2014-01-02T00:00:00.000Z, preferredVersion='null', priority=25, revoked=false}] because existing locks[[TaskLockPosse{taskLock=TimeChunkLock{type=REPLACE, groupId='compact_test_mgmlcmch_2026-02-03T18:37:51.661Z', dataSource='test', interval=2014-01-01T00:00:00.000Z/2014-01-01T03:00:00.000Z, version='2026-02-03T18:37:51.673Z', priority=25, revoked=false}, taskIds=[compact_test_mgmlcmch_2026-02-03T18:37:51.661Z]}]] have same or higher priorities

i'm not sure i quite understand the part replace only those segments which are fully contained in the locked interval? do you mean when append segments get upgrade? for compaction, i think testPartialIntervalCompactWithFinerSegmentGranularity proves that a smaller interval lock works?

...ge-query/src/main/java/org/apache/druid/msq/indexing/destination/SegmentGenerationUtils.java

…n/task/CompactionTask.java Co-authored-by: Lucas Capistrant <capistrant@users.noreply.github.com>

capistrant

Thanks for responding to my questions @cecemei

I think this should be ready then. Could you add a release notes section to the PR description that spec's out the two bug fixes you did for the lastCompactionState in MSQ compaction. Those may be of interest to have in druid 37 release notes. I'll leave it up to you if you also think adding a dedicated note about the locking changes.

kfaraz

Left some questions/suggestions based on first glance.

processing/src/main/java/org/apache/druid/timeline/CompactionState.java

kfaraz · 2026-02-02T06:07:44Z

processing/src/main/java/org/apache/druid/java/util/common/granularity/Granularities.java

  public static final Granularity FIFTEEN_MINUTE = GranularityType.FIFTEEN_MINUTE.getDefaultGranularity();
  public static final Granularity THIRTY_MINUTE = GranularityType.THIRTY_MINUTE.getDefaultGranularity();
  public static final Granularity HOUR = GranularityType.HOUR.getDefaultGranularity();
+  public static final Granularity THREE_HOUR = GranularityType.THREE_HOUR.getDefaultGranularity();


Why is this new granularity needed?

added this for the test because compaction task automatically picks the finest gran (hourly) if it's not specified, but msq runner can only process one interval per task (there's 3 hours).

Let's use one of the existing standard granularities then, like SIX_HOUR or EIGHT_HOUR and update the tests accordingly. Would that work?

updated to use SIX_HOUR and reverted THREE_HOUR change

indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionRunner.java

indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java

kfaraz · 2026-02-03T07:57:14Z

indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java

  {
-    final List<DataSegment> segments = segmentProvider.findSegments(taskActionClient);
-    return determineLockGranularityAndTryLockWithSegments(taskActionClient, segments, segmentProvider::checkSegments);
+    return determineLockGranularityAndTryLock(taskActionClient, List.of(segmentProvider.interval));


I am not sure if it safe to remove this. IIUC, the original logic was to fetch all the segments that overlap (even if partially) with the semgentProvider.interval, then try to lock the umbrella interval of the segments. So the final interval locked may be bigger than the original interval. The idea is that a task will be able to replace only those segments which are fully contained in the interval that is locked by that task.

@cecemei , could you share some more details on the issue that you encountered with the MSQ runner and the hour granularity? Maybe there could be an alternative solution.

indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

...-tests/src/test/java/org/apache/druid/testing/embedded/compact/CompactionSupervisorTest.java

indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java

kfaraz · 2026-02-05T05:53:14Z

indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionRunner.java

+   *
+   * @return true if aligned intervals are required by this runner, false otherwise.
+   */
+  boolean requireAlignedInterval();


Nit: I wonder if we ever allow non-aligned intervals even for native compaction.
I don't see that field being set while creating auto-compaction tasks.

I feel adding this new method to the CompactionRunner interface for a field which is never used is overkill.
Let's just continue ignoring this field in the MSQ compaction runner as we do today and just log a warning message in MSQ compaction runner if it is ever set to true.

I think we will just go ahead and deprecate this field and remove it altogether in a future release.

The docs seem to think so too:
https://druid.apache.org/docs/latest/data-management/manual-compaction/#compaction-io-configuration

#14127

I think we have had the parameter for backwards compat for a while, we can probably deprecate it now since auto-compaction does not use it and MSQ does not use it either.

kfaraz · 2026-02-05T05:58:48Z

indexing-service/src/test/java/org/apache/druid/indexing/common/actions/TaskActionTestKit.java

+    return this;
+  }
+
+  public TaskActionTestKit setBatchSegmentAllocation(boolean batchSegmentAllocation)


Just curious, have we added new tests which need this?

ah i think i had some issues with batch segment allocation with segment lock, and some pending segments cache was not removed properly or something (maybe i should have written it down), and have decided to disable the batch allocation to avoid this issue

yea so it's set to false in CompactionTaskRunBase

Oh, if that is the case, we would need to dig deeper into it since it sounds like a bug and batch segment allocation is the default. It should work for all cases.

it's only for segment lock, and i wasn't sure it's due to test setup or anything.

Okay, in that case, let's skip the test case for segment lock and add a comment or add a test case and mark it disabled. That way, we would be aware that there is a bug and can address it later.

i could not reproduce the test cases any more, maybe it disappeared after we switched to use the widen interval. anyways, so i added the segmentQueue test param.

...service/src/test/java/org/apache/druid/indexing/common/task/NativeCompactionTaskRunTest.java

…terval

kfaraz

Thanks for your patience, @cecemei !
I have left some final thoughts. We should be good to merge once these are addressed.

kfaraz · 2026-02-05T07:34:59Z

indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionIOConfig.java

    return inputSpec;
  }

+  @Deprecated


Nit: Adding a short javadoc indicating why this is deprecated would be nice.

kfaraz · 2026-02-05T07:40:27Z

multi-stage-query/src/main/java/org/apache/druid/msq/guice/MSQIndexingModule.java

  }
+
+  @Provides
+  IndexerControllerContext.Builder providesContextBuilder(Injector injector)


We need not use the Injector directly here.

Suggested change

IndexerControllerContext.Builder providesContextBuilder(Injector injector)

IndexerControllerContext.Builder getControllerContextBuilder(

@EscalatedGlobal ServiceClientFactory serviceClientFactory,

OverlordClient overlordClient

)

Also, is there any functional change here or is the code just being moved from MSQControllerTask?

there's no functional change, it's just easier for testing so that we can inject the context in tests

controller context need the injector... i originally added separate class as well but i think some tests breaks since the guice module asks for class instance even before it asks to provide context instance. it's not ideal but injector is in many places.

kfaraz · 2026-02-05T07:41:13Z

multi-stage-query/src/main/java/org/apache/druid/msq/indexing/IndexerControllerContext.java

 */
 public class IndexerControllerContext implements ControllerContext
 {
+  public interface Builder


It might be cleaner to move this into a separate file and call it IndexerControllerContextFactory to align with other similar factory classes in Druid.

kfaraz · 2026-02-05T07:43:36Z

multi-stage-query/src/main/java/org/apache/druid/msq/indexing/IndexerControllerContext.java

    this.overlordClient = overlordClient;
    this.memoryIntrospector = injector.getInstance(MemoryIntrospector.class);
-    final StorageConnectorProvider storageConnectorProvider = injector.getInstance(Key.get(StorageConnectorProvider.class, MultiStageQuery.class));
+    final StorageConnectorProvider storageConnectorProvider = injector.getInstance(Key.get(


Suggestion: Change doesn't seem necessary. It's best to avoid formatting changes unless they really help with readability.

this has been reverted

kfaraz · 2026-02-05T07:47:06Z

indexing-service/src/test/java/org/apache/druid/indexing/common/actions/TaskActionTestKit.java

+    return this;
+  }
+
+  public TaskActionTestKit setBatchSegmentAllocation(boolean batchSegmentAllocation)


Okay, in that case, let's skip the test case for segment lock and add a comment or add a test case and mark it disabled. That way, we would be aware that there is a bug and can address it later.

kfaraz · 2026-02-05T14:37:29Z

...ing-service/src/test/java/org/apache/druid/indexing/common/task/TestSpyTaskActionClient.java

+ * <p>
+ * Useful for verifying that tasks publish the expected segments and schemas in integration tests.
+ */
+public class TestSpyTaskActionClient implements TaskActionClient


This action client is very specific to segment transactions and might not have a wide usage.

I would suggest the following:

Move this test class to CompactionTaskRunBase itself

Rename it to something simpler TestTaskActionClient or WrappingTaskActionClient. (The "spy" is a little misleading since it suggests some usage of Mockito spy utilities).

Avoid javadocs since this code is unit-test-only and seems self explanatory. Add only 1-line javadocs or regular comments where necessary.

this is done

kfaraz · 2026-02-05T14:51:05Z

multi-stage-query/src/test/java/org/apache/druid/msq/exec/MSQCompactionTaskRunTest.java

+    for (LockGranularity lockGranularity : new LockGranularity[]{LockGranularity.TIME_CHUNK}) {
+      for (boolean useCentralizedDatasourceSchema : new boolean[]{false}) {
+        for (boolean useSegmentMetadataCache : new boolean[]{false, true}) {
+          for (boolean useConcurrentLocks : new boolean[]{false, true}) {
+            for (Interval inputInterval : new Interval[]{TEST_INTERVAL}) {
+              for (Granularity segmentGran : new Granularity[]{Granularities.SIX_HOUR}) {


I think only 2 for loops (for useSegmentMetadataCache and useConcurrentLocks) should be enough here. The other parameters seem to be taking a single value only.

kfaraz · 2026-02-05T14:51:37Z

multi-stage-query/src/test/java/org/apache/druid/msq/exec/MSQCompactionTaskRunTest.java

+          for (boolean useConcurrentLocks : new boolean[]{false, true}) {
+            for (Interval inputInterval : new Interval[]{TEST_INTERVAL}) {
+              for (Granularity segmentGran : new Granularity[]{Granularities.SIX_HOUR}) {
+                String name = StringUtils.format(


This should be done in the @Paramaters annotation itself.

kfaraz · 2026-02-05T14:55:51Z

indexing-service/src/test/java/org/apache/druid/indexing/common/task/CompactionTaskRunBase.java

-  }
-
-  @Test
-  public void testWithSegmentGranularity() throws Exception


Is this test not needed anymore?

segment granularity has been tested very throughly in this test after this change, so we dont need it any more.

kfaraz · 2026-02-05T14:59:21Z

indexing-service/src/test/java/org/apache/druid/indexing/common/task/CompactionTaskRunBase.java

 import java.util.stream.Collectors;

-@RunWith(Parameterized.class)
-public class CompactionTaskRunTest extends IngestionTestBase
+public abstract class CompactionTaskRunBase


The diff for this class seems big and it is a little difficult to ensure that no assertions have actually changed.
Since we are changing some core logic in this PR, I would advise minimizing the changes to this test class so that we can be certain that the new code works correctly with all existing tests.

If any refactor is needed in this test, we can do it in a follow up PR.
For the time being, if some methods need to be reused for the MSQCompactionTaskRunTest, they may be copied over or we may make them public static where applicable.

motivation to change this test is that it doesnt cover msq runner, and while re-writing it i find some tests too long to read, and some issues in msq runner have been discovered due to making it run with msq runner. the test coverage has improved a lot in this class, priori to this pr, this test only tests native compaction runner and also dont use a real task action client. i was surprised by how little test coverage we have on msq compaction runner, wanted to use this test file for any msq compaction runner related change in the future.

the changes in this pr actually was mostly not covered by this test (before this pr), only the interval locking is related. i re-ran the old test with the interval lock change, the failures are due to the interval diff in compaction state, and another test failed due to lock interval covers the input interval now. so all seems expected.

cecemei

Thanks for reviewing this @kfaraz ! Addressed all your comments.

cecemei · 2026-02-05T16:20:04Z

indexing-service/src/test/java/org/apache/druid/indexing/common/actions/TaskActionTestKit.java

+    return this;
+  }
+
+  public TaskActionTestKit setUseCentralizedDatasourceSchema(boolean useCentralizedDatasourceSchema)


it's not supported by msq engine, schema is not saved

cecemei · 2026-02-05T16:23:26Z

indexing-service/src/test/java/org/apache/druid/indexing/common/actions/TaskActionTestKit.java

+
+  public TaskActionTestKit setBatchSegmentAllocation(boolean batchSegmentAllocation)
+  {
+    if (configFinalized.get()) {


the before method was called before @test, wont that mean we can't change the config between tests, but in setup only?

cecemei · 2026-02-05T16:27:15Z

processing/src/main/java/org/apache/druid/timeline/CompactionState.java

  }
+
+
+  public Builder toBuilder()


ah i thought this toBuilder() is more fluent style? like:

getDefaultCompactionState().toBuilder().dimensionSpec().build()

cecemei · 2026-02-05T16:41:36Z

multi-stage-query/src/test/java/org/apache/druid/msq/exec/MSQReplaceTest.java

                             GranularityType.HOUR,
-                             Intervals.of("2016-06-27T01:00:00.000Z/2016-06-27T02:00:00.000Z")
+                             Intervals.of("2016-06-27T01:00:00.000Z/2016-06-27T02:00:00.000Z"),
+                             new CompactionTransformSpec(


tests without filters are still the same (without transform spec).

cecemei · 2026-02-05T16:50:19Z

multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQCompactionRunner.java

-    } else {
-      // The changed granularity would result in a new virtual column that needs to be aggregated upon.
-      dimensionSpecs.add(new DefaultDimensionSpec(TIME_VIRTUAL_COLUMN, TIME_VIRTUAL_COLUMN, ColumnType.LONG));
+    if (!dataSchema.getDimensionsSpec().getDimensionNames().contains(ColumnHolder.TIME_COLUMN_NAME)) {


cecemei · 2026-02-05T18:28:13Z

indexing-service/src/test/java/org/apache/druid/indexing/common/task/CompactionTaskRunBase.java

 import java.util.stream.Collectors;

-@RunWith(Parameterized.class)
-public class CompactionTaskRunTest extends IngestionTestBase
+public abstract class CompactionTaskRunBase


motivation to change this test is that it doesnt cover msq runner, and while re-writing it i find some tests too long to read, and some issues in msq runner have been discovered due to making it run with msq runner. the test coverage has improved a lot in this class, priori to this pr, this test only tests native compaction runner and also dont use a real task action client. i was surprised by how little test coverage we have on msq compaction runner, wanted to use this test file for any msq compaction runner related change in the future.

the changes in this pr actually was mostly not covered by this test (before this pr), only the interval locking is related. i re-ran the old test with the interval lock change, the failures are due to the interval diff in compaction state, and another test failed due to lock interval covers the input interval now. so all seems expected.

cecemei · 2026-02-05T18:41:21Z

indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionIOConfig.java

    return inputSpec;
  }

+  @Deprecated


cecemei · 2026-02-05T18:43:12Z

indexing-service/src/test/java/org/apache/druid/indexing/common/actions/TaskActionTestKit.java

+    return this;
+  }
+
+  public TaskActionTestKit setBatchSegmentAllocation(boolean batchSegmentAllocation)


i could not reproduce the test cases any more, maybe it disappeared after we switched to use the widen interval. anyways, so i added the segmentQueue test param.

cecemei · 2026-02-05T18:46:23Z

indexing-service/src/test/java/org/apache/druid/indexing/common/actions/TaskActionTestKit.java

+
+  public TaskActionTestKit setBatchSegmentAllocation(boolean batchSegmentAllocation)
+  {
+    if (configFinalized.get()) {


well this is just to prevent ppl from accidently calling it inside test method

cecemei · 2026-02-05T18:47:35Z

indexing-service/src/test/java/org/apache/druid/indexing/common/task/CompactionTaskRunBase.java

-  }
-
-  @Test
-  public void testWithSegmentGranularity() throws Exception


segment granularity has been tested very throughly in this test after this change, so we dont need it any more.

MSQ compaction runner run test

dd3293a

github-actions bot added Area - Batch Ingestion Area - Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Jan 24, 2026

fix test

f723153

github-actions bot added the Area - Querying label Jan 26, 2026

cecemei added 4 commits January 26, 2026 16:35

fix test 2

4004d08

lock input interval

2531140

test

a335f9d

Merge branch 'master' into compact-test

02f8030

cecemei changed the title ~~MSQ compaction runner run test~~ Add MSQCompactionTaskRunTest Jan 28, 2026

test coverage

9050d70

github-advanced-security bot found potential problems Jan 28, 2026

View reviewed changes

cecemei changed the title ~~Add MSQCompactionTaskRunTest~~ Fix MSQ compaction state and native interval locking, add test coverage Jan 28, 2026

cecemei marked this pull request as ready for review January 28, 2026 05:05

cecemei added 3 commits January 28, 2026 23:13

allowNonAlignedInterval and forceDropExisting

e43ec58

fix test

80ead3d

Merge branch 'master' into compact-test

c3de9ea

capistrant reviewed Jan 30, 2026

View reviewed changes

cecemei and others added 5 commits February 1, 2026 22:14

Update indexing-service/src/main/java/org/apache/druid/indexing/commo…

5d17e68

…n/task/CompactionTask.java Co-authored-by: Lucas Capistrant <capistrant@users.noreply.github.com>

Update indexing-service/src/main/java/org/apache/druid/indexing/commo…

95f4e09

…n/task/CompactionTask.java Co-authored-by: Lucas Capistrant <capistrant@users.noreply.github.com>

update

f918259

style

58c7197

drop-existing

4024d87

capistrant added Bug Release Notes labels Feb 2, 2026

capistrant approved these changes Feb 2, 2026

View reviewed changes

kfaraz reviewed Feb 3, 2026

View reviewed changes

cecemei and others added 2 commits February 3, 2026 10:21

Apply suggestion from @kfaraz

d56410f

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

format

c79ca10

cecemei added 8 commits February 3, 2026 11:43

aligned

05c8688

build

8e44e7c

mis-aligned

529dcd7

format

c99ee19

test

27fa2ed

lock-interval

f4af751

lock

c19202e

test

d85d0fc

kfaraz reviewed Feb 5, 2026

View reviewed changes

...-tests/src/test/java/org/apache/druid/testing/embedded/compact/CompactionSupervisorTest.java Outdated Show resolved Hide resolved

kfaraz mentioned this pull request Feb 5, 2026

Add compaction eligibility concept and incremental compaction support #18968

Open

10 tasks

kfaraz reviewed Feb 5, 2026

View reviewed changes

indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java Outdated Show resolved Hide resolved

kfaraz reviewed Feb 5, 2026

View reviewed changes

indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java Outdated Show resolved Hide resolved

kfaraz reviewed Feb 5, 2026

View reviewed changes

...service/src/test/java/org/apache/druid/indexing/common/task/NativeCompactionTaskRunTest.java Outdated Show resolved Hide resolved

cecemei added 3 commits February 4, 2026 22:31

force drop existing, revert non-aligned, deprecated allowNonAlignedIn…

1fb1342

…terval

revert THREE_HOUR

c38fbe1

revert format change

7221970

cecemei requested a review from kfaraz February 5, 2026 06:56

test

782d2ac

kfaraz reviewed Feb 5, 2026

View reviewed changes

cecemei added 3 commits February 5, 2026 09:13

comment

241b2a7

use-queue

1079703

reduce test

3c7df1a

cecemei commented Feb 5, 2026

View reviewed changes

cecemei requested a review from kfaraz February 5, 2026 19:34

capistrant self-requested a review February 5, 2026 21:12

-  IndexerControllerContext.Builder providesContextBuilder(Injector injector)
+  IndexerControllerContext.Builder getControllerContextBuilder(
+      @EscalatedGlobal ServiceClientFactory serviceClientFactory,
+      OverlordClient overlordClient
+  )

Conversation

cecemei commented Jan 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Fixed compaction state issues

Fixed native compaction interval locking

Added MSQCompactionTaskRunTest

Refactored test infrastructure

Added builder pattern support

Key changed/added classes in this PR

Uh oh!

Check notice

Uh oh!

Check notice

Uh oh!

capistrant left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cecemei Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

capistrant left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfaraz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfaraz Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

cecemei commented Jan 24, 2026 •

edited

Loading

cecemei Feb 3, 2026 •

edited

Loading

capistrant left a comment •

edited

Loading

kfaraz Feb 5, 2026 •

edited

Loading