⚠ This page is served via a proxy. Original site: https://github.com
This service does not collect credentials or authentication data.
Skip to content

Conversation

@teskje
Copy link
Contributor

@teskje teskje commented Jan 29, 2026

This PR changes the order of builtin objects in BUILTIN_STATIC to move all indexes as far up as possible, i.e. immediately below the objects they index. The BUILTIN_STATIC order determines in which order objects are created during bootstrapping, so moving indexes up ensures that they can be used by all other objects depending on the indexed relations. Previously, builtin objects would commonly read data from persist again and duplicate view fragments, instead of using readily available indexes.

On main, this change reduces the number of dataflow operators in mz_catalog_server from 11863 to 9948 and the number of arrangements from 842 to 763.

Motivation

  • This PR fixes a previously unreported bug.

mz_catalog_server has a bunch of dataflows that don't make use of available indexes. Slack thread.

Tips for reviewer

Checklist

  • This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
  • If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

@teskje teskje force-pushed the catalog-fix-builtin-index-order branch from 0c0fd13 to a39008c Compare January 29, 2026 11:32
This commit changes the order of builtin objects in `BUILTIN_STATIC` to
move all indexes as far up as possible, i.e. immediately below the
objects they index. The `BUILTIN_STATIC` order determines in which order
objects are created during bootstrapping, so moving indexes up ensures
that they can be used by all other objects depending on the indexed
relations. Previously, builtin objects would commonly read data from
persist again and duplicate view fragments, instead of using readily
available indexes.
@teskje teskje force-pushed the catalog-fix-builtin-index-order branch from a39008c to 9ea31d8 Compare January 29, 2026 11:37
@teskje teskje marked this pull request as ready for review January 29, 2026 11:47
@teskje teskje requested a review from a team as a code owner January 29, 2026 11:47
@teskje teskje requested a review from ohbadiah January 29, 2026 11:47
Copy link
Member

@antiguru antiguru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes a lot of sense!

@teskje teskje marked this pull request as draft January 29, 2026 12:39
@teskje
Copy link
Contributor Author

teskje commented Jan 29, 2026

It actually doesn't work 🙈 From Slack:

Ok, I think the BUILTINS_STATIC fix is the wrong fix. It works in new environments because it makes it so that the catalog IDs end up in the order we want them. But in existing environments the catalog IDs have already been assigned. Reordering BUILTINS_STATIC changes the order in which the objects are applied to the in-memory catalog, but applying doesn't create the dataflow plans. That happens in bootstrap_dataflow_plans here. That function just creates the plans in the order provided, and the order provided is just the ID order.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants