Skip to main content

Log options

Structured logging behavior can be configured on a per-event-source basis using log options. This allows for fine-grained control over aspects like cloud synchronization, on-premise buffering, rate limiting, and data retention.

These options are defined by the LogOptions proto message and set on the DataLogger service running on the IPC, using the structured logging clients in the SDK.

note

There are defaults are already set on the service. Look at default_log_options.pbtxt to see what they are.

Managing log options​

This guide covers how to manage log options using the C++ StructuredLoggingClient.

INTR_ASSIGN_OR_RETURN(
    StructuredLoggingClient client,
    StructuredLoggingClient::Create("logger.app-intrinsic-base.svc.cluster.local:8080", absl::Now() + absl::Seconds(5)));

Set log options​

To set or update log options, use the SetLogOptions RPC. This RPC takes a map where the key is a unique identifier for the configuration and the value is the LogOptions proto to apply.

The event_source field within the LogOptions proto is an RE2-compliant regex that determines which event sources the configuration applies to.

For example, "ai.intrinsic/specific-event-source" will match only that event source (ai.intrinsic/specific-event-source), while "robot-.*" would match robot-telemetry and robot-state.

tip

If multiple LogOptions configurations match a single event source, the one with the highest log_options_precedence_value is used. This allows you to create a low-precedence default configuration and override it for more specific event sources with a higher precedence value.

tip

The key in the SetLogOptions map is a unique identifier for the configuration. A common and recommended practice is to use the event_source regex itself as the key.

You can then use this key to easily update or overwrite a specific configuration later by calling SetLogOptions again with the same key.

// Match only ai.intrinsic/specific-event-source.
intrinsic_proto::data_logger::LogOptions specific_options;
specific_options.set_event_source("ai.intrinsic/specific-event-source");
specific_options.set_log_options_precedence_value(100);
specific_options.set_sync_active(true);

// Match all event sources.
intrinsic_proto::data_logger::LogOptions default_options;
default_options.set_event_source(".*");
default_options.set_log_options_precedence_value(10);
default_options.set_sync_active(false);

// Set the options in a single call, using the event source as the key.
INTR_RETURN_IF_ERROR(client.SetLogOptions({
    {"ai.intrinsic/specific-event-source", specific_options},
    {".*", default_options},
}));

In this case, ai.intrinsic/specific-event-source will use the specific_options because it has a higher log_options_precedence_value (100) than default_options (10). Any other event source will use default_options.

Get log options​

To retrieve the currently applied log options for a specific event source, use the GetLogOptions RPC. The service will return the LogOptions with the highest precedence that matches the given event source name.

INTR_ASSIGN_OR_RETURN(
    intrinsic_proto::data_logger::LogOptions current_options,
    client.GetLogOptions("ai.intrinsic/specific-event-source"));

LOG(INFO) << "sync_active: " << current_options.sync_active();

Options for certain workflows​

Here are some common configuration workflows you might want to use, and the corresponding options to set.

Configure a logging budget​

To manage network bandwidth, you can set a rate limit on the data being logged. The DataLogger service uses a token bucket algorithm to do rate limiting, and you can configure the algorithm using the logging_budget field.

  • refresh: The rate in bytes per second that the bucket refills
  • burst: The maximum capacity (in bytes) of the bucket, which handles bursts of data
intrinsic_proto::data_logger::LogOptions budget_options;
budget_options.set_event_source("ai.intrinsic/high-frequency-sensor");
budget_options.set_log_options_precedence_value(50);
budget_options.mutable_logging_budget()->set_refresh(1024 * 10); // 10 KiB/s.
budget_options.mutable_logging_budget()->set_burst(1024 * 100);  // 100 KiB burst.

INTR_RETURN_IF_ERROR(client.SetLogOptions({{"ai.intrinsic/high-frequency-sensor", budget_options}}));

Configure and prioritize syncing to the cloud​

You can configure whether logs are synced to the cloud and specify the priority level for each event source.

  • sync_active: If true, logs for matching event sources are synced to the cloud. Set to false for data that should only be stored locally.
  • priority: A higher value gives the event source a higher upload priority. When bandwidth is limited and the system needs to drop data, logs with a lower priority will be dropped before logs with a higher priority.
// Prioritize critical data.
intrinsic_proto::data_logger::LogOptions critical_options;
critical_options.set_event_source("ai.intrinsic/some_critical_topic_prefix-.*");
critical_options.set_log_options_precedence_value(100);
critical_options.set_sync_active(true);
critical_options.set_priority(100);

// Disable syncing for debug data.
intrinsic_proto::data_logger::LogOptions debug_options;
debug_options.set_event_source("ai.intrinsic/some_debug_prefix-.*");
debug_options.set_log_options_precedence_value(50);
debug_options.set_sync_active(false);

INTR_RETURN_IF_ERROR(client.SetLogOptions({
    {"ai.intrinsic/some_critical_topic_prefix-.*", critical_options},
    {"ai.intrinsic/some_debug_prefix-.*", debug_options},
}));

Configure disk retention for recordings and log fetches​

For data that will be used to create Solution Recordings, it's critical to ensure the data is retained on disk, and is retained long enough to be included in a recording, especially if the device has limited connectivity.

  • retain_on_disk: Set to true to save logs to the on-premise logging disk, this is a pre-requisite for enabling logs from an event source to be saved in a recording.
  • retain_on_disk_retention_duration: Specifies how long the data should be kept. If not set, a server-side default is used (typically 24 hours).

These configurations will also affect retention for log fetches within the IPC.

note

The retention duration is the maximum lookback you can have on an event source to create a recording with.

Once a recording is created, the logs are copied to a separate archive with extended retention, allowing uploads to continue over limited bandwidth for potentially several weeks.

intrinsic_proto::data_logger::LogOptions recording_options;
recording_options.set_event_source("ai.intrinsic/some_event_source_to_record");
recording_options.set_log_options_precedence_value(100);
recording_options.set_retain_on_disk(true);
recording_options.mutable_retain_on_disk_retention_duration()->set_seconds(7 * 24 * 60 * 60);

INTR_RETURN_IF_ERROR(client.SetLogOptions({{"ai.intrinsic/some_event_source_to_record", recording_options}}));