Train Service

The train service is responsible for handling pose estimator training jobs, including the creation, saving, listing, status monitoring, deletion, cancellation, and validation of training jobs. HMIs meant for training pose estimators typically interact with the Train API.

The Train Service can be instructed via its gRPC API.

Life of a pose estimator training job

A new training job is created with CreateTrainingJob of TrainService. In the CreateTrainingJobRequest, the user can:

• Specify a list of scene objects on which a new pose estimator will be trained
• Configure the pose estimator parameters
• Set up various metadata, such as asset name, package and version, which will be useful for saving the pose estimator upon completion of its training.

If in doubt about the selected parameters of CreateTrainingJobRequest, the user can call VerifyTrainingJob prior to CreateTrainingJob with the same CreateTrainingJobRequest as input argument. VerifyTrainingJob checks an incoming request and provides feedback in case the selection of certain parameters can be improved. CreateTrainingJob returns a long-running operation.

Training a pose estimator may take from a few minutes to several hours depending on the type of pose estimator, scene object properties, selected pose range, and other training parameters. During training of the pose estimator, the training status can be queried through GetTrainingJob, which returns progress information and/or encountered errors. ListTrainingJobs can be used to list all ongoing training jobs and also enables getting the status of each job.

An ongoing training job can be cancelled with CancelTrainingJob. This is an asynchronous operation without guaranteed success. On successful cancellation, the long-running operation corresponding to the training job is not deleted, but its longrunning.Operation.error field is set to gRPC error code Code.CANCELLED. A training job that has been cancelled can be explicitly deleted with DeleteTrainingJob.

On successful completion of the training job, the user may select to save the pose estimator to their current solution by calling SaveTrainingJob. This saves the results of the training job into a data asset that is now installed to the solution, and it deletes the long-running operation. Once the training job has been saved, it can be deleted with DeleteTrainingJob.

Example using the gRPC API with grpcurl

The following examples use grpcurl to illustrate how various gRPCs of the train API can be called. The instructions on how to set up an environment for calling gRPC services via grpcurl can be found here.

In order to build the transitively closed protobuf descriptor file set of the train service, the following command can be run from the SDK directory:

bazel build //intrinsic/perception/proto/v1:train_service_proto_descriptor_set

In the following, we use the grpc-train alias for brevity:

alias grpcurl-train="grpcurl -H 'x-server-name:${VM}' -H 'authorization:Bearer ${TOKEN}' -protoset 'bazel-bin/intrinsic/perception/proto/v1/train_service_proto_descriptor_set_transitive_set_sci.proto.bin' -d @ 'www.endpoints.${PROJECT}.cloud.goog:443'"

VM, TOKEN, and PROJECT are defined with the same values as for grpcurl-executive.

Create pose estimator training job

The following gRPC creates a training job for a new surface-based pose estimator:

grpcurl-train intrinsic_proto.perception.v1.TrainService/CreateTrainingJob <<EOM
{
  "pose_estimation_config": {
    "targets": {
      "id": "box",
      "scene_object": {
        "name": "box",
        ...
      },
      "pose_range": {
        "icosahedron": {
          "depth": 2,
          "vertex": 23,
          "vertex": 142,
          "vertex": 143
        },
        "min_distance": 1.5,
        "max_distance": 2.5
      },
      "symmetry": {
        "finite_non_trivial_symmetry": {
          "center": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
          },
        "axis_angle": {
          "axis": {
            "x": 1
          },
          "angle": 3.1415927
        }
      }
    },
    "intrinsic_params": {
      "focal_length_x": 1409.2594805825083,
      "focal_length_y": 1409.2594805825083,
      "principal_point_x": 967.5,
      "principal_point_y": 607.5,
      "dimensions": {
        "cols": 1936,
        "rows": 1216
      }
    }
  },
  "pose_estimator_type": "POSE_ESTIMATOR_TYPE_SURFACE_BASED",
  "custom_data": {
    "value": "some custom data"
  },
  "asset_metadata": {
    "asset_name": "box_pose_estimator"
  }
}
EOM

• The pose estimator should detect a target with id box. The scene_object field defines the corresponding scene object in the running solution. To obtain the scene object protobuf, we recommend calling the GetInstalledAsset gRPC of the InstalledAssets service. The scene object can be retrieved through the installed_asset.deployment_data.scene_object.manifest.assets.scene_object_model field of the returned InstalledAsset. Please note the following constraints:
  • Only single-target training is supported; you cannot train on multiple targets or scene objects at once.
  • Targets with multiple meshes are not yet supported.
• The pose_range field defines potential view angles from the camera towards the target. In this case, the pose range is defined by three points on top of an icosahedron. The target is expected to be within a distance of [1.5, 2.5] meters from the camera.
• The target has a finite symmetry, defined by an angle of pi rad (or 180 deg) along the x axis.
• The camera that will be used for training the pose estimator is defined through its intrinsic parameters, which can be obtained from the camera configuration after completing the camera's intrinsic calibration. You can retrieve the current configuration by calling the GetResourceInstance gRPC of the ResourceRegistry service. This call returns a ResourceInstance object. The intrinsic parameters are located in the object's configuration field, which is a CameraConfig protobuf.
• pose_estimator_type specifies the type of pose estimator to be trained; in this case, a surface-based pose estimator.
• custom_data may include any data that the user wishes to persist during and after training. This data is returned to the caller via the TrainingJobStatus message when we call GetTrainingJob or ListTrainingJobs. As an example, custom_data may hold the name of the pose estimator that is being trained. This name is needed for saving the pose estimator upon successful training completion. If many pose estimators are being trained in parallel, custom_data provides a convenient way to keep track of their names along with other useful information that the user wishes to associate with each pose estimator.
• Finally, asset_metadata includes useful information about the data asset that will be used to store the pose estimator once training is successfully completed.

info

The symmetry of an object and the pose range that needs to be considered while training a new pose estimator may be tricky to define for certain objects and/or workcell set-ups. In this case, it may help to use the Flowstate Editor, specifically the pose estimation wizard, to visualize various symmetries and pose range options. If a training is kicked off from the UI, the pose estimation configuration can be accessed through the list of pose estimators, by clicking on Open configuration file. Those parameters can be then copied to the pose_estimation_config defined in the CreateTrainingJobRequest protobuf.

If access to Flowstate is not possible, another option to gain confidence in the selected symmetry and pose range parameters is to call intrinsic_proto.perception.v1.TrainService/VerifyTrainingJob prior to intrinsic_proto.perception.v1.TrainService/CreateTrainingJob using the same input argument. VerifyTrainingJob may provide feedback in case there are a priori concerns about the selected training parameters.

warning

Please note that for training ML-based pose estimators (single-view, multi-view), your organization needs to have the required ML credits. If the ML credits have been exhausted, CreateTrainingJob will return an error.

Monitor training job status

Information on the status of an ongoing training job can be obtained through:

grpcurl-train intrinsic_proto.perception.v1.TrainService/ListTrainingJobs <<< ""

for all ongoing training operations, or

grpcurl-train intrinsic_proto.perception.v1.TrainService/GetTrainingJob <<EOM
{
  "name": "perception_ml_train/perception-2025-05-05-10-00-00"
}
EOM

for a specific operation that has been previously listed.

Cancel a training job

The cancellation of a training job can be triggered through:

grpcurl-train intrinsic_proto.perception.v1.TrainService/CancelTrainingJob<<EOM
{
  "name": "perception_ml_train/perception-2025-05-05-10-00-00"
}
EOM

The name in the CancelTrainingJobRequest can be obtained through the response of ListTrainingJobs.

Delete a training job

A cancelled or completed training job can be deleted through:

grpcurl-train intrinsic_proto.perception.v1.TrainService/DeleteTrainingJob<<EOM
{
  "name": "perception_ml_train/perception-2025-05-05-10-00-00"
}
EOM

Save a training job

A training job that has been successfully completed can be saved, i.e., the corresponding data asset holding the pose estimator result can be installed to the running solution, persisting the pose estimator through:

grpcurl-train intrinsic_proto.perception.v1.TrainService/SaveTrainingJob<<EOM
{
  "name": "perception_ml_train/perception-2025-05-05-10-00-00",
  "asset_id": {
    "name": "box_pose_estimator",
    "package": "ai.intrinsic"
  }
}
EOM

Sharing pose estimators across solutions

For a saved pose estimator which has been tested and validated with respect to its pose estimation accuracy, you may wish to install it to other solutions within the same organization, for instance for deployment on multiple workcells in parallel. In this case, a user with the role of "releaser" may release the data asset which holds the pose estimator to the production catalog. This is possible through the CreateAsset gRPC of the AssetCatalog service.

Once a pose estimator data asset has been released to the catalog, users can install it to various solutions on different IPCs/VMs by calling the CreateInstalledAsset gRPC of the InstalledAssets service.