Support OR with subqueries using DNF #3681
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
commit: |
❌ 28 Tests Failed:
View the top 3 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
This comment has been minimized.
This comment has been minimized.
Contributor
|
Found 75 test failures on Blacksmith runners: Failures
|
robacourt
changed the title
robacourt
changed the title
``` WHERE x IN (subquery1) OR status = 'active' WHERE x IN (subquery1) AND (status = 'active' OR y IN (subquery2)) ``` Key requirement: **Rows must only receive tags for disjuncts they actually satisfy**, including non‑sublink predicates. This prevents phantom tags that keep rows alive after move‑outs. --- **File:** `lib/electric/shapes/shape/subquery_moves.ex` **Add:** a helper that, for each DNF disjunct, builds a SQL expression that represents the *non‑sublink* predicate portion of that disjunct (i.e., replace each `sublink_membership_check(...)` node with `TRUE` and keep the rest). Pseudo‑steps: 1. Add a function: ```elixir @SPEC disjunct_predicates(Shape.t()) :: [%{index: non_neg_integer(), predicate_sql: String.t() | nil}] ``` 2. Implement by reusing the DNF disjunct ASTs (from `extract_dnf_structure/1`): - For each disjunct AST, walk it and replace `%Eval.Parser.Func{name: "sublink_membership_check", ...}` with a literal `true` node. - Then serialize the modified AST to SQL. 3. If the disjunct has **no non‑sublink predicates** (i.e., it was *only* sublinks), return `predicate_sql = nil` to avoid adding `CASE` conditions later. **Notes:** - Reuse any existing SQL serializer if available (search for an Eval → SQL renderer). - If none exists, add a tiny dedicated converter for the limited AST you need: `and`, `or`, comparisons, literals, refs, and the new `true` literal. **File:** `lib/electric/shapes/querying.ex` **Change:** `make_tags/4` in the DNF branch should wrap each tag expression in a `CASE` that checks its disjunct predicate **and** the sublink membership values. Concrete steps: 1. Extend `Shape` to store disjunct predicate SQL (or compute it on demand via `SubqueryMoves`). 2. In `make_tags/4` (DNF path): - For each disjunct: - Build `value_parts_expr` as now. - Build `predicate_sql` (may be `nil`). - If `predicate_sql` is `nil`, keep current tag expression. - If present, wrap: ```sql CASE WHEN (<predicate_sql>) THEN <tag_expr> ELSE NULL END ``` 3. Ensure `build_headers_part/3` filters out NULL tags (if it doesn’t already). If needed, wrap tag array with `array_remove(tags, NULL)` in `build_headers_part`. **Result:** rows only receive tags for the disjuncts they actually satisfy. --- **File:** `lib/electric/shapes/shape.ex` **Change:** `make_tags_from_dnf/5` must require both: - all sublinks in the disjunct are satisfied **and** - the non‑sublink predicate for the disjunct is true for the record. Steps: 1. Extend `dnf_structure` to include a compiled Eval expression for the disjunct predicate: ```elixir %{ sublinks: [...], patterns: %{...}, comparison_expressions: %{...}, predicate_expr: Electric.Replication.Eval.Expr.t() | nil } ``` 2. When building DNF in `SubqueryMoves.extract_dnf_structure/1`, compute `predicate_expr` by: - replacing each `sublink_membership_check` with literal `true` in the AST - wrapping in `Eval.Expr.wrap_parser_part` - if the result is just `true`, set `predicate_expr = nil` 3. In `make_tags_from_dnf/5`, add: ```elixir predicate_ok = case disjunct.predicate_expr do nil -> true expr -> WhereClause.includes_record?(expr, record, extra_refs_for_predicate) end ``` Then require `predicate_ok && all_satisfied` before tagging. 4. Use the same `extra_refs` input. If predicate evaluation doesn’t use extra refs, pass `%{}`. --- **File:** `test/electric/shapes/shape/subquery_moves_test.exs` Add tests for: 1. `x IN subq OR status = 'active'` - DNF returns 2 disjuncts - Disjunct 0 predicate is `nil` (only sublink) - Disjunct 1 predicate is `status = 'active'` 2. `x IN subq AND (status = 'active' OR y IN subq2)` - DNF returns 2 disjuncts - First disjunct has predicate `status = 'active'` - Second disjunct predicate `nil` **File:** `test/electric/shapes/shape_test.exs` (or add to existing shape tests) Case: - Shape: `x IN (subq) OR status = 'active'` - Record: `x = 1, status = 'inactive'` - Extra refs include `sublink 0` contains `1` Expect: - Only tag for disjunct 0 is present (no tag for disjunct 1) Then: - Move‑out for `x=1` should remove the row (no phantom tag keeps it). **File:** `test/electric/plug/router_test.exs` Add a test: 1. Create tables `parent(id, include_parent)` and `child(id, parent_id, status)` 2. Shape: `parent_id IN (SELECT id FROM parent WHERE include_parent = true) OR status = 'active'` 3. Start shape with one child where `parent_id` matches and `status = 'inactive'`. 4. Toggle `include_parent` to false (move‑out for subquery). Expect: - A move‑out event is emitted. - The child is removed (no lingering tag from `status = 'active'`). Also include the inverse case where `status = 'active'` stays, so row remains even after subquery move‑out. --- If time allows: - Extract tag building to `Electric.Shapes.Shape.Tagging`. - Extract predicate stripping to `Electric.Shapes.Shape.Predicate`. These are not required for correctness. --- - [ ] DNF disjunct predicate extraction added - [ ] SQL tag generation conditional on predicate - [ ] Runtime tag generation conditional on predicate - [ ] Unit tests for predicate extraction - [ ] Unit tests for runtime tags - [ ] Integration tests for OR + non‑sublink predicate
✅ Deploy Preview for electric-next ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
We already support tagged move-ins/outs for shapes with a single subquery dependency. Allowing multiple same-level subqueries currently results in 409/must-refetch on move-ins/outs because:
Desired behavior
Proposed approach (Option 1): multi-tags + client-side tag tracking + INSERT-as-UPSERT
Key idea
Treat each OR branch (or dependency) as an independent “reason” a row is in the shape:
This avoids 409/must-refetch and correctly handles overlap.
MVP scope (initial)**
✅ Support: multiple same-level subquery membership checks connected by OR:
🚫 Out of scope (initially):
Implementation plan (high-level tasks)
A) Server (sync-service)
B) Internal client in sync-service: Materializer
C) External clients (TypeScript at minimum)
D) Tests
Acceptance criteria