Skip to main content

Skill Unit Testing

This guide teaches you how to create a Python or C++ unit test.

Before attempting this guide you must:

  1. Set up your development environment
  2. Know how to create a skill

Create a new unit test​

All new Python and C++ skills come with an automatically generated unit test. Create a new Python or C++ skill.

  • When asked to enter a skill ID, use com.example.stop_stopwatch.
  • When asked to enter a folder, use skills/stop_stopwatch.

You now have several files in the skills/stop_stopwatch folder, including one file that contains unit tests.

Run a skill unit test​

There are two ways to run a skill unit test:

  • By running bazel test on the command line, or
  • By clicking the CodeLens above the test target in VS code.

Run a bazel test on the command line​

Use bazel test to run tests when:

  • You prefer command line based workflows, or
  • You want to run multiple tests at the same time.

If you are using the dev container, open a new terminal in VS Code. If you are not using the dev container, open a new terminal and go to the directory containing your MODULE.bazel.

You must tell Bazel which targets it should test by giving it the name of the target. The skill you created has a unit test target named //skills/stop_stopwatch:stop_stopwatch_test Run that test by executing the following command in the terminal:

bazel test //skills/stop_stopwatch:stop_stopwatch_test

Run a bazel test through the VS Code extension​

Use the VS code extention to run a test when:

  • You are using the dev container, and
  • You want to run a single test.
  1. Open the file skills/stop_stopwatch/BUILD.
  2. Click the Test CodeLens above the py_test or cc_test target named stop_stopwatch_test.

CodeLens above stop_stopwatch_test

Understand the new unit test​

Open the file skills/stop_stopwatch/stop_stopwatch_test.py.

import unittest

from intrinsic.skills.testing import skill_test_utils as stu

from skills.stop_stopwatch.stop_stopwatch import StopStopwatch
from skills.stop_stopwatch.stop_stopwatch_pb2 import StopStopwatchParams


class StopStopwatchTest(unittest.TestCase):

    def test_get_footprint(self):
        skill = StopStopwatch()

        params = StopStopwatchParams(
            text="hello world",
        )
        context = stu.make_test_get_footprint_context()
        request = stu.make_test_get_footprint_request(params)

        result = skill.get_footprint(request, context)
        self.assertTrue(result.lock_the_universe)

    def test_preview(self):
        skill = StopStopwatch()

        params = StopStopwatchParams(
            text="hello world",
        )
        context = stu.make_test_preview_context()
        request = stu.make_test_preview_request(params)

        # Update this test when you implement preview
        with self.assertRaises(NotImplementedError):
            skill.preview(request, context)

    def test_execute(self):
        skill = StopStopwatch()

        params = StopStopwatchParams(
            text="hello world",
        )
        context = stu.make_test_execute_context()
        request = stu.make_test_execute_request(params)

        with self.assertLogs() as log_output:
            skill.execute(request, context)

        output = log_output[0][0].message
        self.assertEqual(output, '"text" parameter passed in skill params: hello world')


if __name__ == '__main__':
    unittest.main()

The file contains a unit test for each of the 3 most important methods on a skill:

  • Get Footprint
  • Preview
  • Execute

Each method takes two parameters:

  • A context object that gives access to services in the solution, like the world service or a custom service
  • A request object that includes the skill's parameters

Every test follows the same structure:

  1. Create an instance of the skill.
  2. Set the skill's parameters.
  3. Create a context object.
  4. Create a request object.
  5. Call a method on the skill with those objects.
  6. Check the results.

Implement the StopStopwatch skill​

Most skills depend on services to perform some work for them. Follow the instructions in this section to implement the StopStopwatch such that it depends on a stopwatch service.

Define the parameters and return type​

How will the unit test know if the skill behaves correctly? The skill should return a result that a unit test can check. Modify the skill's proto file skills/stop_stopwatch/stop_stopwatch.proto as follows:

  1. The skill won't need any parameters, so delete the text field from StopStopwatchParams.
  2. The skill needs a result message, so add a new message StopStopwatchResult with a field time_elapsed.

The two messages should look like this:

message StopStopwatchParams {
}

message StopStopwatchResult {
  double time_elapsed = 1;
}

Make the skill declare that it returns StopStopwatchResult when it is executed.

  1. Open skills/stop_stopwatch/stop_stopwatch.manifest.textproto.
  2. Copy the following return type declaration to the end of stop_stopwatch.manifest.textproto. 
    return_type {
      message_full_name: "com.example.StopStopwatchResult"
    }

Depend on the stopwatch service​

Make stop_stopwatch depend on the Stopwatch Service from the SDK Examples. There are multiple ways to depend on a service.

  • If the service is in the same Bazel workspace as the skill, make the py_proto_library, cc_proto_library, py_grpc_library, and cc_grpc_library targets visible to your skill. Then, make your skill targets depend on the service targets directly.
  • If the service is in a different Bazel workspace and that workspace is open source, add an external dependency to your MODULE.bazel. Then, make your skill targets depend on the service targets from the external dependency.
  • If all you have is the service's proto definition, then add it to your own Bazel workspace.

Pretend that all you have is the service's proto definition. Follow these instructions to add it to your Bazel workspace.

  1. Copy the stopwatch_service.proto file into the skills/stop_stopwatch folder.
  2. Add the following proto_library() target to skills/stop_stopwatch/BUILD.
proto_library(
  name = "stopwatch_service_proto",
  srcs = ["stopwatch_service.proto"],
)
  1. Add a load() statement for the py_grpc_library() rules to skills/stop_stopwatch/BUILD.
load("@com_github_grpc_grpc//bazel:python_rules.bzl", "py_grpc_library")
  1. Create two targets for the stopwatch service proto definition.
py_proto_library(
    name = "stopwatch_service_py_pb2",
    visibility = ["//visibility:public"],
    deps = [":stopwatch_service_proto"],
)

py_grpc_library(
    name = "stopwatch_service_py_pb2_grpc",
    srcs = [":stopwatch_service_proto"],
    visibility = ["//visibility:public"],
    deps = [":stopwatch_service_py_pb2"],
)
  1. Add these dependencies to the list of deps of the py_library target named stop_stopwatch_py.
py_library(
    name = "stop_stopwatch_py",
    # ...
    deps = [
        # ... 
        ":stopwatch_service_py_pb2_grpc",
        ":stopwatch_service_py_pb2",
        "@ai_intrinsic_sdks//intrinsic/util/grpc:connection",
        "@ai_intrinsic_sdks//intrinsic/util/grpc:interceptor",
    # ...
  1. Add the following imports to the top of skills/stop_stopwatch/stop_stopwatch.py.
import grpc
from intrinsic.util.grpc import connection
from intrinsic.util.grpc import interceptor
from skills.stop_stopwatch import stopwatch_service_pb2 as stopwatch_proto
from skills.stop_stopwatch import stopwatch_service_pb2_grpc as stopwatch_grpc
  1. Add a function to create a gRPC client for the stopwatch service above the skill class definition in skills/stop_stopwatch/stop_stopwatch.py.
def make_grpc_stub(resource_handle):
    logging.info(f"Address: {resource_handle.connection_info.grpc.address}")
    logging.info(f"Server Instance: {resource_handle.connection_info.grpc.server_instance}")
    logging.info(f"Header: {resource_handle.connection_info.grpc.header}")

    # Create a gRPC channel without using TLS
    grpc_info = resource_handle.connection_info.grpc
    grpc_channel = grpc.insecure_channel(grpc_info.address)
    connection_params = connection.ConnectionParams(
        grpc_info.address, grpc_info.server_instance, grpc_info.header
    )

    intercepted_channel = grpc.intercept_channel(
        grpc_channel,
        interceptor.HeaderAdderInterceptor(connection_params.headers),
    )
    return stopwatch_grpc.StopwatchServiceStub(
        intercepted_channel
    )
  1. Replace the execute method in skills/stop_stopwatch/stop_stopwatch.py with the following to make the skill stop the stopwatch.
  @overrides(skill_interface.Skill)
  def execute(
      self,
      request: skill_interface.ExecuteRequest[stop_stopwatch_pb2.StopStopwatchParams],
      context: skill_interface.ExecuteContext
  ) -> stop_stopwatch_pb2.StopStopwatchResult:
      stub = make_grpc_stub(context.resource_handles["stopwatch_service"])

      logging.info("Stopping the stopwatch")
      response = stub.Stop(stopwatch_proto.StopRequest())
      if not response.success:
          raise skill_interface.SkillError(1, f"Failed to stop stopwatch {response.error}")

      logging.info("Successfully stopped the stopwatch")
      result = stop_stopwatch_pb2.StopStopwatchResult(time_elapsed=response.time_elapsed)
      return result
  1. Lastly, add a dependency on the stopwatch service to stop_stopwatch.manifest.textproto.
dependencies {
  required_equipment {
    key: "stopwatch_service"
    value {
      capability_names: "stopwatch.StopwatchService"
    }
  }
}

The skill now stops the stopwatch when it is executed.

Provide a service to a skill in a unit test​

Now that the skill depends on the stopwatch service, every unit test must provide the service to the skill being tested.

Choose between using the real service or a fake service​

If you have the source code of the service your skill depends on, you have a choice to make:

  • Instantiate the real service in the unit test
  • Instantiate a fake service in the unit test

Instantiate the real service in a unit test when:

  • You have access to the source code of the real service, and
  • The service is easy to instantiate, and
  • The test conditions needed can be reached determinisitcally with the real service.

Instantiate a fake service in a unit test when:

  • You don't have access to the source code of the real service, or
  • The service has many dependencies or is difficult to instantiate, or
  • The test is checking edge cases that cannot be reached deterministically with the real service.

Using real services makes unit tests more realistic, but it may limit what parts of your skill can be tested. Using fake services can test more of a skills behavoir, but it can miss real bugs in your skill if the fake service behaves differently from the real service.

Use a fake service in a test​

Let's assume you have chosen to use a fake service in your unit test. Follow these instructions to create a fake stopwatch service that returns a constant value as the time elapsed.

  1. Add the following imports to skills/stop_stopwatch/stop_stopwatch_test.py. 
    from skills.stop_stopwatch import stopwatch_service_pb2 as stopwatch_proto
    from skills.stop_stopwatch import stopwatch_service_pb2_grpc as stopwatch_grpc
  2. Add this FakeStopwatchServicer class to the top of skills/stop_stopwatch/stop_stopwatch_test.py. 
    class FakeStopwatchServicer(stopwatch_grpc.StopwatchServiceServicer):

      def Stop(self, request, context):
        response = stopwatch_proto.StopResponse()
        response.time_elapsed = 42
        response.success = True
        return response

The test file now has a fake stopwatch service. Next, modify each test case to provide the fake service to the skill.

The intrinsic SDK includes a function make_grpc_server_with_resource_handle. Use it to create a gRPC service and resource handle in the unit tests.

  1. Remove the setting of the text field in the parameters message from the test_get_footprint, test_preview, and test_execute functions. 
    params = StopStopwatchParams()
  2. Add these lines to the three functions test_get_footprint, test_preview, and test_execute after the line skill = StopStopwatch(). 
    server, handle = stu.make_grpc_server_with_resource_handle("stopwatch_service")
    stopwatch_grpc.add_StopwatchServiceServicer_to_server(FakeStopwatchServicer(), server)
    server.start()
  3. Modify the call to make_test_get_footprint_context to include the newly created resource handle. 
    context = stu.make_test_get_footprint_context(
        resource_handles={handle.name: handle},
    )
  4. Modify the call to make_test_preview_context to include the newly created resource handle. 
    context = stu.make_test_preview_context(
        resource_handles={handle.name: handle},
    )
  5. Modify the call to make_test_execute_context to include the newly created resource handle. 
    context = stu.make_test_execute_context(
        resource_handles={handle.name: handle},
    )
  6. Modify the test expecations of test_execute_context to expect the the time elapsed value returned by the fake stopwatch service. 
    result = skill.execute(request, context)

    self.assertEqual(result.time_elapsed, 42)

Run the unit test one more time; it should pass.

Conclusion​

You now know how to create skill unit tests, even if the skill depends on other services. However, unit tests are limited by the realism of the fake services they use. Read the Skill Integration Testing guide to learn how to create tests that use real services.