Skill Integration Testing

This guide shows how to create an integration test. Integration tests check that your skill works with other skills and services in a solution.

Before attempting this guide you must:

Overview

In this guide you will create an integration test. Integration tests are semi-automated. They require a user to:

Run the test
Add skills, services, and assets to a solution.

This guide walks you through creating an integration test that checks if two skills can start and stop a stopwatch provided by a Stopwatch service.

Download the SDK Examples

You need a Bazel workspace with the stopwatch skills and services. Get one by downloading the SDK Examples repository.

Opening a teriminal on your system

Using git clone to download the SDK Examples repository.

git clone https://github.com/intrinsic-ai/sdk-examples.git

Open the folder in the dev container

Create an Integration test

Create a minimal test as a starting point.

Create a new folder at the root of the Bazel workspace called integration_test_guide.
Create a file called BUILD in that folder.
Add this py_binary rule into BUILD
```
py_binary(
    name = "stopwatch_integration_test",
    srcs = ["stopwatch_integration_test.py"],
    main = "stopwatch_integration_test.py",
    deps = [
    ],
)
```
note
You might be surprised to use a py_binary rule instead of a py_test rule. Use a py_binary rule because integration tests require user input, and must be run manually.
Create another file called stopwatch_integration_test.py.

Put the following code into stopwatch_integration_test.py.

import unittest

class TestStopwatch(unittest.TestCase):

    def test_start_stop_stopwatch(self):
        raise NotImplementedError("TODO, write the test!")


if __name__ == '__main__':
    unittest.main()

note

When writing integration tests, you must write the test in Python. However, skills and services used in an integration test may be written in any supported language.

Use bazel run to run the test.

bazel run //integration_test_guide:stopwatch_integration_test

Right now, the test fails with NotImplementedError. Continue reading to learn how to implement it.

Connect to a solution

Integration tests run skills and services in a solution. Specifically, integration tests use the target solution. Create a new solution through Flowstate, and then set it as your target solution.

Make the test connect to the target solution.

Import the deployments library near the top of stopwatch_integration_test.py
```
from intrinsic.solutions import deployments
```
Add a classmethod called setUpClass to the TestStopwatch class, and make it use the deployments library to connect to the target solution.
```
@classmethod
def setUpClass(cls):
    cls.solution = deployments.connect_to_selected_solution()
```

Add a dependency on @ai_intrinsic_sdks//intrinsic/solutions:deployments to the py_binary target in BUILD.

py_binary(
    name = "stopwatch_integration_test",
    # ...
    deps = [
        "@ai_intrinsic_sdks//intrinsic/solutions:deployments",
    ],
)

Ask the user to add the skills and services to the solution

Use the input function to ask the user to add the skills and service in the setUpClass method. The user must hit [Enter] after each input call to advance the test.

need_skills = ['com.example.start_stopwatch', 'com.example.stop_stopwatch_py']
need_services = [('com.example.stopwatch_service', 'stopwatch')]
for skill_id in need_skills:
    input(f'Please install the skill {skill_id} into the target solution')
for service_id, name in need_services:
    input(f'Please install the service {service_id} into the target solution, then add an instance named "{name}"')

Finally, store references to the two newly added skills at the end of the setUpClass method.

cls.solution.skills.update()
cls.solution.resources.update()
cls.start_skill = cls.solution.skills.com.example.start_stopwatch
cls.stop_skill = cls.solution.skills.com.example.stop_stopwatch
cls.stopwatch_service = cls.solution.resources.stopwatch

Create a behavior tree

In Flowstate, the Executive executes skills. A behavior tree tells the Executive what order to execute skills.

The integration test must create skill instances, and then put them into a behavior tree. Then the integration test must give the behavior tree to the Executive.

Delete the line raise NotImplementedError("TODO, write the test!") from the test_start_stop_stopwatch method.

Create an instance of each of the skills needed in the test_start_stop_stopwatch method.

start_skill = self.start_skill(
    stopwatch_service=self.stopwatch_service
)
stop_skill = self.stop_skill(
    stopwatch_service=self.stopwatch_service
)

Create a behavior tree using the two skills. Use a Sequence Node to run the Start Stopwatch skill followed by the Stop Stopwatch skill.

Import the behavior_tree at the top of stopwatch_integration_test.py.
```
from intrinsic.solutions import behavior_tree as bt
```

Create the behavior tree instance in the test_start_stop_stopwatch method.

tree = bt.BehaviorTree(
    name="StopStopwatch Integration Test",
    root=bt.Sequence([
        bt.Task(action=start_skill, name="Start stopwatch"),
        bt.Task(action=stop_skill, name="Stop stopwatch"),
    ]
  ),
)

Next, make the executive run the behavior tree.
```
self.solution.executive.run(tree)
```

Finally, add a dependency on @ai_intrinsic_sdks//intrinsic/solutions:behavior_tree to the py_binary target in BUILD.

py_binary(
    name = "stopwatch_integration_test",
    # ...
    deps = [
        "@ai_intrinsic_sdks//intrinsic/solutions:behavior_tree",
        # ...
    ],
)

Add test expectations

The integration test now causes the Start Stopwatch and Stop Stopwatch skills to get executed, but how can the test be sure that they really worked?

First, if any skill fails to run, then the line self.solution.executive.run(tree) will raise an exception. The lack of an exception means the Executive successfully executed the whole behavior tree.

Secondly, the Executive stores the results of the skills. Use the get_value method on the executive to get the time elapsed from the Stop Stopwatch skill. Add an assertion to make sure the time elapsed is more than zero to the end of the test_start_stop_stopwatch method.

self.assertGreater(self.solution.executive.get_value(stop_skill.result).time_elapsed, 0)

Run the test

Run the test one more time.

bazel run //integration_test_guide:stopwatch_integration_test

Add each of the skills and service to the solution when the test prompts you to. The test should pass.

Conclusion

You now know how to create integration tests to check that multiple skills and services work in a solution. If you want to check the behavior of a skill in isolation, read the Skill Unit Testing guide.

Overview​

Download the SDK Examples​

Create an Integration test​

Connect to a solution​

Ask the user to add the skills and services to the solution​

Create a behavior tree​

Add test expectations​

Run the test​

Conclusion​