Scene Alignment

An accurate alignment between the real world and its digital twin is crucial as Flowstate uses the digital twin for example for collision avoidance. The Scene Alignment dialog facilitates accurate alignment of a physical object in a real-world scene with its corresponding digital representation. It can be used either during setup phase to accurate align the digital twin to the built workcell, or at runtime to validate that the real world to digital twin alignment is correct. Alignment can be done either manually or semi-automatically.

The dialog presents a live camera view alongside a 3D world view. It enables the following actions:

• Selecting a camera for the live view, which displays either the real world or the simulation, depending on the application's state.
• Selecting a world (Initial or Execute) to overlay with the camera view.
• Selecting an object within the 3D world.
• Adding multiple cameras to improve alignment accuracy.
• Aligning the objects pose: Either manually using 6 degree-of-freedom transformations, or semi-automatically using perception based refinement (note that the initial object's pose has to be already close to the real pose in order for refinement to work).
• Adjusting the overlay opacity.
• Saving changes to the Initial world.

Prerequisites

• The camera's intrinsic calibration has to be accurate.
• Camera-to-robot calibration has to be accurate.
• (If using a multi-view setup) Camera-to-camera calibration has to be accurate.

How to Perform Scene Alignment

1. Open the Scene alignment dialog

   Initiate the Scene Alignment dialog from the desired camera view.

   • Select a camera in your 3D world. In the Properties panel, click on Control -> Open Scene alignment dialog.

2. Select a world

   Within the Setup section, use the 3D world dropdown to select the world to overlay with the camera view. Changing the selection will load the corresponding world (Initial or Execute), ensuring that all subsequent operations, such as object selection and pose refinement, are performed within that specific world context.

3. Select a camera

   Within the Setup section, use the Reference camera dropdown to select the camera intended for the alignment process. It is initially set to the selected camera of the previous step.

4. (Optional) Add Refinement Cameras

   To improve the alignment accuracy, add more cameras to capture images from multiple viewpoints. Ensure the object to be aligned is visible in all cameras intended to add.
   Steps:

   1. Navigate to the Setup section and expand Refinement cameras (Optional).
   2. Click + to open a new camera selection list.
   3. Select an additional camera. Repeat for all desired cameras.

5. Select an object

   Select the object to align in the 3D world view.

6. Align the object

   • Open the Align section.
   • Rough Positioning: Roughly position the selected object in the 3D world to match its position in the camera view. A rough alignment of the object's pose is crucial for the alignment algorithm to converge properly. Since this algorithm does not use a trained pose estimator under the hood, it requires an initial object pose to work properly.
   • (Optional) Add Occluders: If other objects in the scene might be blocking the view of the object to be aligned, then specify these as "Occluders". Expand the Occluders block and click the + icon to add one or more objects from the world.
   • Click the Align button.
     • The pose of the selected object will be refined.
     • An annotated image, showing the refined pose, will be displayed instead of the camera view. Check the annotated image carefully to make sure the algorithm converged successfully. If refinement worked properly, the rendered edges of the object's pose should match it's counterpart in the camera image.
     • The Align button will change to Accept.
     • Use the Overlay opacity slider to adjust the transparency between the 3D world overlay and the annotated image. This can aid in visualizing the alignment.
     • The object's pose in the 3D world will be updated.
     • A score chip will display the alignment score. A score above 0.80 is generally considered a good alignment, while lower scores may indicate a poor result or that the object is not clearly visible in the camera image, but this depends on a case by case basis, so it is important to check the visualizations for each refined object independently.

7. Accept or reset changes

   • Accept: If the alignment is satisfactory, click the Accept button. This will clear the annotated image and start the live camera view again, indicating that the current refined pose is accepted. Make sure you have checked the annotated image for correctness before clicking Accept.
   • Reset: If the alignment is not satisfactory, click the Reset button. This will revert the object to its initial position in the selected world.

Changes of one or multiple refinements can be saved to the world by clicking the Save button in the bottom right corner.

• If in Initial world: Clicking Save will permanently save the current changes to the application's initial state.
• If in Belief world: Clicking Save will transfer the refined objects poses to the "Initial" world.