release: embed mode + structlog agg #54
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduce CallTracker to assign sequential frame IDs and maintain call paths during opcode traversal. This enables accurate identification of which contract call each opcode belongs to, even when the same contract is called multiple times. Extend Structlog with CallFrameID and CallFramePath fields to persist the tracking information alongside each opcode record. Update extractCallAddress to handle all CALL-type opcodes (CALL, CALLCODE, DELEGATECALL, STATICCALL) for complete call target extraction.
feat: detect CREATE/CREATE2 opcodes and fetch contract address from receipt refactor: replace extractCallAddress with extractCallAddressWithCreate to handle contract creation addresses
… trace-based computation - Remove fetchCreateAddress and hasCreateOpcode helpers - Introduce ComputeCreateAddresses to extract addresses directly from trace - Update extractCallAddressWithCreate signature to accept index and map - Rename queue name from "transaction-structlog" to "transaction_structlog" - Expand test coverage for nested and failed CREATE scenarios
The root frame now starts at depth 1 instead of 0 to match the actual EVM structlog output, where execution begins at depth 1. All tests updated to reflect the new depth semantics.
…om child gas Introduce GasSelf to represent the gas consumed by an opcode *excluding* any gas spent in child frames. For CALL/CREATE opcodes this yields the pure call overhead (warm/cold access, memory expansion, value transfer); for all other opcodes GasSelf equals GasUsed. This allows accurate aggregation of total execution gas without double counting. - Add ComputeGasSelf() helper that subtracts the sum of *direct* child GasUsed values from each CALL/CREATE opcode’s GasUsed. - Extend Structlog struct and ClickHouse schema with new GasSelf column. - Update both ProcessTransaction() and ExtractStructlogs() to populate the new field. - Provide extensive unit tests covering nested, sibling and edge cases.
…odes test(structlog): add comprehensive format_address tests for 20-byte padding refactor(structlog): introduce formatAddress to normalize stack values to 42-char addresses
…ic frames Replace heuristic precompile check with go-ethereum's canonical list to avoid mis-classifying early EOAs/contracts as precompiles. Tighten the EOA detection logic to only create synthetic frames when call depth remains unchanged (depth == nextDepth) instead of the previous nextDepth <= depth, eliminating phantom frames for failed calls and out-of-gas scenarios.
Introduce abstraction layer for execution nodes to support both RPC-based and embedded usage patterns. - Add Node interface defining the contract for execution data providers - Add DataSource interface for hosts to provide data directly - Add EmbeddedNode implementation that delegates to a DataSource - Rename existing implementation to RPCNode - Add NewPoolWithNodes() for creating pools with pre-created nodes - Update pool to work with Node interface instead of concrete types
Replace go-ethereum types (Block, Transaction, Receipt) with abstract interfaces to enable library embedding without CGO. Move geth-specific RPC implementation to pkg/ethereum/execution/geth/ subpackage with build tags, allowing clean separation between embedded and RPC modes.
…ack allocations Introduce dual-mode processing (RPC vs embedded) in StructLog to reduce memory pressure. Embedded mode pre-computes GasUsed and CallToAddress in the tracer, avoiding post-processing passes and stack allocations. - Add GasUsed and CallToAddress fields to StructLog - Document operation modes and field semantics - Extract call address via new helper, supporting both modes - Add opcode constants and isCallOpcode utility
… for embedded mode Reduces memory pressure by skipping full stack capture; the tracer already extracts CallToAddress directly for CALL-family opcodes.
Introduce hasPrecomputedGasUsed() to detect when the tracer already populates GasUsed (embedded mode). Skip expensive post-processing computation in that case, falling back to RPC mode calculation only when needed. This maintains backward compatibility while optimizing performance for embedded mode traces.
…eation for readability
* master: fix(ethereum): improve error message for unsupported chain IDs refactor(ethereum): change ChainID from int32 to int64 across services and processors
…matty refactor(structlog): dual-mode processing (RPC vs embedded)
* master: fix(ethereum): improve error message for unsupported chain IDs refactor(ethereum): change ChainID from int32 to int64 across services and processors
docs: remove trailing whitespace in README
…esToDataSource_ChainID test
* feat/call-frames: refactor: remove unused ParityTrace types and helper from execution package chore: accidental commit fix(structlog): correct EOA call detection to prevent phantom synthetic frames WIP: current changes test(structlog): fix and expand address extraction tests for CALL opcodes test(structlog): add comprehensive format_address tests for 20-byte padding refactor(structlog): introduce formatAddress to normalize stack values to 42-char addresses feat(structlog): add GasSelf field to isolate CALL/CREATE overhead from child gas fix(call_tracker): align root frame depth with EVM traces (depth 1) refactor(processor): replace receipt-based CREATE address lookup with trace-based computation style(transaction_processing.go): move comment to line above log to match Go style test: add unit tests for CREATE/CREATE2 address extraction feat: detect CREATE/CREATE2 opcodes and fetch contract address from receipt refactor: replace extractCallAddress with extractCallAddressWithCreate to handle contract creation addresses test: add comprehensive unit tests for extractCallAddress function feat(structlog): add call frame tracking to identify EVM call contexts
Eliminates race condition where MarkReady() could fire before callbacks were registered, causing missed ready events.
…tion refactor(pool): add debug logging for OnReady callback registration and execution
…rom structlogs in embedded mode instead of skipping post-processing
…scan Introduce hasPrecomputedCreateAddresses() to check whether the tracer already populated CallToAddress for CREATE/CREATE2 opcodes. When true (embedded mode), we skip the costly ComputeCreateAddresses() pass, improving performance for large traces.
docs: update comments and error messages to match actual processor name
feat: introduce transaction call_frame processor for EVM execution analysis
…tions test(aggregator): add unit tests for failed vs successful tx gas handling Co-Authored-By: Claude Opus 4.5 <[email protected]>
…ic gas for failed txs - Add underflow guards in ComputeGasUsed and computeGasUsed when structlog gas values are corrupted or out of order, falling back to GasCost. - Reorder arithmetic in computeIntrinsicGas to avoid underflow when receiptGas < gasCumulative. - Always compute intrinsic gas for root frame, even on failed transactions, since it is charged before EVM execution starts. - Add StartBlock/EndBlock config options to allow reprocessing ranges. - Extend test coverage for all underflow scenarios and failed-tx behavior.
fix: prevent uint64 underflow in gas calculations and correct intrinsic gas for failed txs
fix(aggregator): skip gas refund and intrinsic gas for failed transactions
* master: ci: add alpha-release workflow for tagged builds Introduces a new GitHub Actions workflow that triggers on any tag push. It derives a release suffix from the tag, sets up Go, Node, Docker/QEMU, and runs GoReleaser inside a container to build and publish pre-release artifacts without marking them as latest.
* master: ci(alpha-release.yml): replace goreleaser job with tag-release job ci(alpha-release.yml): trigger on push to release/* branches instead of tags ci(alpha-release.yml): compute next alpha version from latest matching tag ci(alpha-release.yml): push new tag via epobot token for automated releases
Erigon's debug_traceTransaction can return huge gasCost values when an unsigned integer underflow occurs in callGas(). Detect and clamp any gasCost that exceeds the available gas to the actual gas left, matching Reth's behavior. Prevents downstream consumers from seeing implausibly large gas costs.
…field REVERT executes successfully (no opcode error) but still causes the transaction to fail. The failure is indicated by trace.Failed=true. Update Finalize to set ErrorCount=1 in this case and use the corrected value when deciding whether to apply gas refunds. Add test TestFrameAggregator_RevertWithoutOpcodeError to ensure REVERT transactions are correctly flagged as failed and receive no refund.
fix(aggregator): count REVERT as error even when opcode has no error field
fix(structlog): sanitize corrupted gasCost values from Erigon underflow bug
Use toUInt64OrZero to cast nullable UInt32 to UInt64, returning 0 for NULL. Treat 0 as "no data" and return genesis block, preventing nil pointer panic.
fix(state): handle nullable UInt32 column in getLimiterMaxBlock
…UInt32 The new expression avoids potential type-casting issues in ch-go by explicitly handling NULL values before casting to UInt64.
…elivery The channel-based leadership notification was dropping events when the buffer (size 10) was full, causing instances to get stuck thinking they were still leader when they had lost leadership. This adds OnLeadershipChange(callback) to the Elector interface for guaranteed delivery via synchronous callback invocation. The channel API is preserved for backward compatibility but marked as deprecated. Changes: - Add LeadershipCallback type and OnLeadershipChange method to interface - Implement callback storage and notification in RedisElector - Update handlers to call callbacks first, then channel (best-effort) - Update Manager to use callbacks instead of monitorLeadership goroutine - Add tests for callback invocation, multiple callbacks, guaranteed delivery, and slow callback handling
…ication fix(leaderelection): add callback-based notification for guaranteed delivery
Move block registration (ClickHouse & Redis) before enqueueing tasks to guarantee the block record exists before any task can complete. Calculate expected task count upfront and warn if enqueueing fails.
fix: reorder block processing steps to prevent race condition?
…ync (#60) * refactor(leaderelection): replace hand-rolled Redis locking with redsync Replace manual SetNX/Lua script implementation with github.com/go-redsync/redsync/v4 for distributed mutex handling. Changes: - Add redsync v4.15.0 dependency - Simplify Elector interface: remove LeadershipChannel() and GetLeaderID() - Rewrite RedisElector using redsync.Mutex with: - WithSetNXOnExtend() for Redis restart resilience - WithDriftFactor(0.01) for clock drift safety - WithTries(1) for non-blocking acquisition attempts - Update tests to use callback-based notification only - Reduce implementation from ~374 to ~260 lines Breaking changes: - LeadershipChannel() removed (use OnLeadershipChange callback instead) - GetLeaderID() removed (redsync uses opaque internal values) * fix logging
Replace synchronous, per-block metrics collection with dedicated background goroutines that refresh heavy metrics every 15 s. This removes costly ClickHouse queries from the hot path and reduces block-processing latency. - Remove updateBlockMetrics() from both structlog & structlog_agg - Add start/stopMetricsWorker() helpers and runMetricsWorker() loop - Keep only lightweight BlockHeight update in ProcessNextBlock - Introduce limiter cache in state.Manager to avoid repeated MAX() queries; refresh every 6 s via background goroutine - Update limiter query to use ORDER BY … LIMIT 1 for index usage - Add comprehensive tests for cache hit/miss, concurrent access, single-start guarantees and refresh behaviour
to avoid full table scan and improve performance
…processing (#64) feat(pending.go): add ErrBlockAlreadyBeingProcessed sentinel error refactor(block_processing): acquire Redis lock before ClickHouse mark to prevent races fix(state): include milliseconds in ClickHouse timestamp format
Change default processing behavior to run as fast as possible with no delay between cycles. When no work is available, a 10ms backoff prevents CPU spin. - Remove DefaultInterval (was 10s), add DefaultNoWorkBackoff (10ms) - Remove automatic interval assignment in config validation - Change processBlocks to return bool indicating if work was done - Rewrite runBlockProcessing to use default case for continuous processing - Fixed interval mode (interval > 0) still supported for rate limiting
- Add configurable asyncInsert and waitForAsyncInsert settings to each processor's ClickHouse writes to reduce part creation pressure - Use *bool pointers to distinguish omitted config (defaults to true) from explicit false values - Remove unused chunkSize and progressLogThreshold config options - Simplify structlog ProcessTransaction by removing OnInput streaming pattern in favor of simple batch insert - Remove DefaultChunkSize and DefaultProgressLogThreshold from tracker - Update tests to remove chunk-related assertions
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.