Skip to main content

estimate_point_cloud

Skill class for ai.intrinsic.estimate_point_cloud skill.

The estimate_point_cloud skill estimates the 3D point cloud of a scene using observations from at least two cameras. We recommend using the Intrinsic Plenoptic System (IPS) cameras for improved reconstruction.

The goal of this skill is to dynamically reconstruct the scene in case the digital twin does not accurately represent the real world scene. The skill also features collision avoidance with objects that are either misrepresented or entirely missing from the simulated environment.

The skill uses input images and user-defined stereo pairs to compute the scene's 3D reconstruction.

Prerequisites​

Required inputs:

  • capture_data : This parameter must be linked to the output of the capture_images skill. This provides image data for each camera that is intended to be used for point cloud estimation.
  • stereo_pair_ids : A list of stereo pairs required for point cloud estimation. Each pair is defined by two cameras, where each camera specifies a camera slot id (integer value) and an optional sensor_id (integer value). The camera_slot_id refers to the ID of the camera provided via capture_data parameter, while the sensor_id indicates which specific sensor of that camera to use. For single-sensor cameras (e.g., Basler), the sensor_id can be omitted. For IPS cameras the sensor_ids are defined as follows:
    • sensor_id 1: RGB camera
    • sensor_id 2: Polarized camera
    • sensor_id 3: Polarized camera
    • sensor_id 4: Infra-red camera
  • point_cloud_service_name : the name of the service to be used for point cloud estimation. It has to be an existing service in the solution. Based on this, the algorithm will be automatically chosen. Available services for point cloud estimation can be selected from the Service Catalog.
  • Min Depth and Max Depth define the depth range of the scene..

The cameras must be extrinsically calibrated w.r.t each other. This means that, if the solution is running in real world, the relative position and orientation of the cameras should match (up to the calibration accuracy) the real world position. Follow camera-to-camera calibration process for more information on how to calibrate multiple cameras.

Usage Example​

  • Add capture_images skill for each camera you want to use for the point cloud estimation.
  • Link the CaptureData output of different capture_images skills to the list of CaptureData in estimate_point_cloud.
  • Add one of the point cloud estimation services available from the catalog: Multi-View Point Cloud Estimator (for high performance despite slower runtime) or Point Cloud Estimator (for faster runtime with a trade-off in performance). Give an arbitrary name to the service.
  • Add the name to the point_cloud_service_name parameter of the estimate_point_cloud skill.

(Optional parameters):

  • image_scale define the scale of the images to run the algorithm. Should be in the range [0, 1]. A lower scale results in a faster estimation but can degrade performance. The default value is 1.0.
  • point_cloud_rendering contains additional parameters to optionally add the point cloud to the world.
  • return_params contains additional parameters to optionally add return values to the skill (e.g point_cloud proto or image 3d).
  • debug_params contains additional parameters to help in debug purposes. Can be used to log debug data to reproduce the results.

Parameters​

capture_data​

A list of capture data that can be linked to the output of capture_images skill for multiple cameras.

stereo_pair_ids​

A list of stereo pair that are defined by camera slot id and (optionally) sensor_id for both the first and the second camera in the stereo pair.

point_cloud_service_name​

Name of the available service that should be used for point cloud estimation.

min_depth​

Minimum depth in the working volume observed by the cameras.

max_depth​

Maxium depth in the working volume observed by the cameras.

[Optional] image_scale​

How to scale the input images. Should be in the range [0, 1]. The smaller the scale, the faster the algorithm can run, but the accuracy could be lower. For example, for scale 0.5 all images would be downsampled by a factor of 2 (in both dimensions). Default value is 1.0.

[Otional] point_cloud_rendering​

Optional parameters to render the estimated point cloud in the world.

[Otional] return_params​

Optional parameters to return specific additional values (e.g point cloud proto or image3D).

[Otional] debug_params​

Optional parameters to help debugging.

Error Code​

The skill does not have error codes yet.