Add interactive segmentation and text query features #1582
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add prediction-error event handler in useModeManager.ts - Show prompt dialog when segmentation service fails to initialize - Subscribe and unsubscribe to prediction-error event properly
- Add editorMenuRef and onTextQueryServiceReady to Viewer.vue - Forward text-query-init, text-query, text-query-all-frames events from EditorMenu through Viewer - Add text query handlers to web ViewerLoader.vue - Errors now properly propagate to show error dialog instead of infinite loading
When text query fails to load, show 'Unable to load text query model' instead of 'Unable to load segmentation module' for better clarity.
When users clicked the segmentation button and then quickly switched to polygon mode, the async initialization would complete ~2 seconds later and call completeActivation(), which emitted an 'activate' event that kicked the user out of polygon mode. Added a pendingActivation flag that: - Is set when async initialization starts - Is checked before completing activation - Is cleared in deactivate() to cancel in-flight async activation This ensures switching modes during segmentation loading doesn't cause the mode to be overridden when the async init completes. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Point the desktop segmentation service to interactive_segmenter_default.conf and the stereo service to interactive_stereo_default.conf, both under the VIAME install configs/pipelines directory. The config files are approach-agnostic, so the command-line args no longer specify model details. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Rename vital_segmentation_service.py to viame_segmentation_service.py and VitalSegmentationService to ViameSegmentationService - Remove all sam2/sam3 references: backend detection, hardcoded algorithm types, SAM-specific env vars, and backward-compat aliases - Algorithm creation is now config-driven, reading the type from the VIAME config file rather than probing for specific Python modules - Consolidate duplicate sam2/sam3 API endpoints into generic ones: segmentation_predict, segmentation_status, text_query - Update web client endpoint URLs to match Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Register Electron IPC handlers for the stereo service (enable, disable, set-frame, transfer-line, transfer-points, calibration, status) and expose corresponding async functions from the desktop frontend API. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add stereoSettings (interactiveModeEnabled, loading state) to client settings with a toggle in TrackSettingsPanel for stereo datasets - Emit stereo annotation-complete events from useModeManager when interactive stereo mode is enabled (box, line, polygon, segmentation) - Include control points in segmentation prediction results so they can be forwarded for stereo warping - Pass isStereoDataset prop through Sidebar and Viewer to conditionally show the stereo settings section - Wire up ViewerLoader to handle stereo annotation completion Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…outs The stereo interactive mode toggle was only visible in the vertical sidebar layout. The horizontal (Sidebar.vue) and bottom (Viewer.vue) layouts were not passing the isStereoDataset prop to TrackSettingsPanel. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- ViewerLoader: skip stereo transfer if other camera already has a feature for the track+frame (only-once warp guard) - ViewerLoader: add getOrCreateStereoTrack helper, error dialogs, initial frame disparity kick-off on enable - LineLayer: add click/right-click handlers and selectionAPI so line annotations are interactive (matching RectangleLayer pattern) - LayerManager: wire up lineLayer bus events; route right-click editing through trackEdit for geometry auto-detection - useModeManager: auto-detect LineString/Polygon geometry in handleTrackEdit so the correct editor type activates Note: line editing on the second camera still enters creation mode instead of editing mode — further investigation needed into why EditAnnotationLayer.formatData receives null features or the track is not in enabledTracks for the target camera. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…mplete after 2 points When right-clicking to edit an existing line, the selectedKey was not set to match the stored feature key (e.g. 'HeadTails'), causing EditAnnotationLayer to enter creation mode instead of editing mode with drag handles. Also auto- completes LineString creation after 2 points to prevent multi-point lines. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…rack Right-clicking outside detections in point segmentation mode now properly finalizes the current prediction (if any) and deselects the track. The segmentation recipe also deactivates even when there is no pending prediction, so right-click always exits the mode cleanly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ing geometry The auto-detection was unconditionally overriding the user's chosen editing mode (e.g. switching from rectangle to polygon). Now it only falls back to a different geometry type when the current mode can't edit the track, such as being in Polygon mode on a track that only has a LineString. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The editing mode should always stay as whatever the user last selected. If the geometry doesn't exist for that mode, the edit layer enters creation mode to let the user draw it. Only the LineString selectedKey logic is kept since lines are stored with a recipe key. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When editing a detection that lacks the current geometry type (e.g. line mode on a detection with no line), the edit layer enters creation mode. Right-clicking in empty space now cancels creation and exits editing, matching the behavior when the geometry already exists. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ctions Right-clicking empty space in creation mode now fully deselects the track instead of just exiting editing. Right-clicking on another detection while in creation mode now cancels creation and switches to editing the clicked detection in a single click. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Re-activate the segmentation recipe after confirming so it stays available for the next detection. Reset accumulated points/mask state when switching tracks to prevent stale data from interfering. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Restore the Reset button in EditorMenu that was lost during the Composition API conversion. When pressed, it undoes the segmentation polygon and restores the detection to its pre-segmentation state by saving original track features before prediction modifies them. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Right-click in LineString creation mode was adding a vertex because setShapeInProgress (actiondown) fires before the mouseclick handler that cancels creation. Add a right-click guard to prevent vertex creation on right-click for all annotation types. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When creating a line on a track with no pre-existing bounding box, generate bounds 20% larger than the tight box around both vertices. Uses unionWithoutBounds to replace interim first-point bounds with the final expanded box, ensuring symmetric padding on all sides. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
🤖 Generated with Claude Code