Add config for dimension table upsert #17606
Open
+288
−42
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Description
enableUpsertboolean toDimensionTableConfig(JSON propertyenableUpsert) and exposedisUpsertEnabled()inpinot-spi.DimensionTableDataManagerand gate upsert-related logic behind_enableUpsert, including using queryable-doc-id snapshots when sizing/iterating segments and applying per-segment bitmaps.RecordLocationtype and helper methodsapplyQueryableDocIdsForRecordLocations,applyQueryableDocIdsForLookupTable,applyQueryableDocIdsToSegments, andgetQueryableDocIdsSnapshotinDimensionTableDataManagerto compute and apply per-segmentMutableRoaringBitmapsets and callImmutableSegmentImpl.enableUpsert(...)when appropriate.DimensionTableConfigto pass the new flag, and added integration coverage that creates a small OFFLINE upsert dimension table and asserts deduplicated selection/count results (testDimensionTableUpsertSelection), as well as a unit testtestLookupRespectsQueryableDocIdsthat verifies lookup respects queryable doc ids when upsert is enabled.Testing
mvn/CI) were executed as part of this change.MultiStageEngineIntegrationTest.testDimensionTableUpsertSelection(integration) andDimensionTableDataManagerTest.testLookupRespectsQueryableDocIds(unit), but these tests were added and not run in this rollout.Codex Task