Skip to content

Commit 8105f2a

Browse files
Google APIscopybara-github
Google APIs
authored and
copybara-github
committed
fix!: An existing oauth_scope `https://www.googleapis.com/auth/userinfo.email` is removed from service `JobsV1Beta3` fix!: An existing oauth_scope `https://www.googleapis.com/auth/compute.readonly` is removed from service `MessagesV1Beta3` fix!: An existing oauth_scope `https://www.googleapis.com/auth/userinfo.email` is removed from service `MessagesV1Beta3` fix!: An existing oauth_scope `https://www.googleapis.com/auth/compute.readonly` is removed from service `MetricsV1Beta3` fix!: An existing oauth_scope `https://www.googleapis.com/auth/userinfo.email` is removed from service `MetricsV1Beta3` fix!: An existing oauth_scope `https://www.googleapis.com/auth/compute.readonly` is removed from service `SnapshotsV1Beta3` fix!: An existing oauth_scope `https://www.googleapis.com/auth/userinfo.email` is removed from service `SnapshotsV1Beta3` fix!: An existing oauth_scope `https://www.googleapis.com/auth/compute.readonly` is removed from service `TemplatesService` fix!: An existing oauth_scope `https://www.googleapis.com/auth/userinfo.email` is removed from service `TemplatesService` fix!: An existing oauth_scope `https://www.googleapis.com/auth/compute.readonly` is removed from service `FlexTemplatesService` fix!: An existing oauth_scope `https://www.googleapis.com/auth/userinfo.email` is removed from service `FlexTemplatesService feat: A new field `use_streaming_engine_resource_based_billing` is added to message `.google.dataflow.v1beta3.Environment` feat: A new field `streaming_mode` is added to message `.google.dataflow.v1beta3.Environment` feat: A new message `DataSamplingConfig` is added feat: A new field `data_sampling` is added to message `.google.dataflow.v1beta3.DebugOptions` feat: A new enum `StreamingMode` is added feat: A new method_signature `job,update_mask` is added to method `UpdateJob` in service `JobsV1Beta3` feat: A new field `runtime_updatable_params` is added to message `.google.dataflow.v1beta3.Job` feat: A new field `satisfies_pzi` is added to message `.google.dataflow.v1beta3.Job` feat: A new field `service_resources` is added to message `.google.dataflow.v1beta3.Job` feat: A new message `ServiceResources` is added feat: A new message `RuntimeUpdatableParams` is added feat: A new field `bugs` is added to message `.google.dataflow.v1beta3.SdkVersion` feat: A new message `SdkBug` is added feat: A new field `user_display_properties` is added to message `.google.dataflow.v1beta3.JobMetadata` feat: A new field `step_names_hash` is added to message `.google.dataflow.v1beta3.PipelineDescription` feat: A new field `update_mask` is added to message `.google.dataflow.v1beta3.UpdateJobRequest` feat: A new field `name` is added to message `.google.dataflow.v1beta3.ListJobsRequest` feat: A new field `trie` is added to message `.google.dataflow.v1beta3.MetricUpdate` feat: A new message `StragglerInfo` is added feat: A new message `StreamingStragglerInfo` is added feat: A new message `Straggler` is added feat: A new message `HotKeyDebuggingInfo` is added feat: A new message `StragglerSummary` is added feat: A new field `straggler_summary` is added to message `.google.dataflow.v1beta3.StageSummary` feat: A new field `straggler_info` is added to message `.google.dataflow.v1beta3.WorkItemDetails` feat: A new field `dynamic_destinations` is added to message `.google.dataflow.v1beta3.PubsubLocation` feat: A new field `image_repository_username_secret_id` is added to message `.google.dataflow.v1beta3.ContainerSpec` feat: A new field `image_repository_password_secret_id` is added to message `.google.dataflow.v1beta3.ContainerSpec` feat: A new field `image_repository_cert_path` is added to message `.google.dataflow.v1beta3.ContainerSpec` feat: A new field `enable_launcher_vm_serial_port_logging` is added to message `.google.dataflow.v1beta3.FlexTemplateRuntimeEnvironment` feat: A new field `streaming_mode` is added to message `.google.dataflow.v1beta3.FlexTemplateRuntimeEnvironment` feat: A new field `disk_size_gb` is added to message `.google.dataflow.v1beta3.RuntimeEnvironment` feat: A new field `streaming_mode` is added to message `.google.dataflow.v1beta3.RuntimeEnvironment` feat: A new message `ParameterMetadataEnumOption` is added feat: A new field `group_name` is added to message `.google.dataflow.v1beta3.ParameterMetadata` feat: A new field `parent_name` is added to message `.google.dataflow.v1beta3.ParameterMetadata` feat: A new field `parent_trigger_values` is added to message `.google.dataflow.v1beta3.ParameterMetadata` feat: A new field `enum_options` is added to message `.google.dataflow.v1beta3.ParameterMetadata` feat: A new field `default_value` is added to message `.google.dataflow.v1beta3.ParameterMetadata` feat: A new field `hidden_ui` is added to message `.google.dataflow.v1beta3.ParameterMetadata` feat: A new field `streaming` is added to message `.google.dataflow.v1beta3.TemplateMetadata` feat: A new field `supports_at_least_once` is added to message `.google.dataflow.v1beta3.TemplateMetadata` feat: A new field `supports_exactly_once` is added to message `.google.dataflow.v1beta3.TemplateMetadata` feat: A new field `default_streaming_mode` is added to message `.google.dataflow.v1beta3.TemplateMetadata` feat: A new value `GO` is added to enum `Language` feat: A new value `BIGQUERY_TABLE` is added to enum `ParameterType` feat: A new value `JAVASCRIPT_UDF_FILE` is added to enum `ParameterType` feat: A new value `SERVICE_ACCOUNT` is added to enum `ParameterType` feat: A new value `MACHINE_TYPE` is added to enum `ParameterType` feat: A new value `KMS_KEY_NAME` is added to enum `ParameterType` feat: A new value `WORKER_REGION` is added to enum `ParameterType` feat: A new value `WORKER_ZONE` is added to enum `ParameterType` feat: A new value `BOOLEAN` is added to enum `ParameterType` feat: A new value `ENUM` is added to enum `ParameterType` feat: A new value `NUMBER` is added to enum `ParameterType` feat: A new value `KAFKA_TOPIC` is added to enum `ParameterType` feat: A new value `KAFKA_READ_TOPIC` is added to enum `ParameterType` feat: A new value `KAFKA_WRITE_TOPIC` is added to enum `ParameterType` docs: A comment for field `service_options` in message `.google.dataflow.v1beta3.Environment` is changed docs: A comment for field `service_kms_key_name` in message `.google.dataflow.v1beta3.Environment` is changed docs: A comment for field `dataset` in message `.google.dataflow.v1beta3.Environment` is changed docs: A comment for field `service_account_email` in message `.google.dataflow.v1beta3.Environment` is changed docs: A comment for field `flex_resource_scheduling_goal` in message `.google.dataflow.v1beta3.Environment` is changed docs: A comment for field `worker_region` in message `.google.dataflow.v1beta3.Environment` is changed docs: A comment for field `worker_zone` in message `.google.dataflow.v1beta3.Environment` is changed docs: A comment for field `debug_options` in message `.google.dataflow.v1beta3.Environment` is changed docs: A comment for enum `WorkerIPAddressConfiguration` is changed docs: A comment for message `SdkHarnessContainerImage` is changed docs: A comment for field `capabilities` in message `.google.dataflow.v1beta3.SdkHarnessContainerImage` is changed docs: A comment for field `enable_hot_key_logging` in message `.google.dataflow.v1beta3.DebugOptions` is changed docs: A comment for method `CreateJob` in service `JobsV1Beta3` is changed docs: A comment for method `ListJobs` in service `JobsV1Beta3` is changed docs: A comment for method `AggregatedListJobs` in service `JobsV1Beta3` is changed docs: A comment for message `Job` is changed docs: A comment for field `id` in message `.google.dataflow.v1beta3.Job` is changed docs: A comment for field `project_id` in message `.google.dataflow.v1beta3.Job` is changed docs: A comment for field `name` in message `.google.dataflow.v1beta3.Job` is changed docs: A comment for field `type` in message `.google.dataflow.v1beta3.Job` is changed docs: A comment for field `environment` in message `.google.dataflow.v1beta3.Job` is changed docs: A comment for field `current_state` in message `.google.dataflow.v1beta3.Job` is changed docs: A comment for field `requested_state` in message `.google.dataflow.v1beta3.Job` is changed docs: A comment for field `transform_name_mapping` in message `.google.dataflow.v1beta3.Job` is changed docs: A comment for field `location` in message `.google.dataflow.v1beta3.Job` is changed docs: A comment for message `Step` is changed docs: A comment for enum `JobState` is changed docs: A comment for message `JobExecutionStageInfo` is changed docs: A comment for enum value `JOB_VIEW_ALL` in enum `JobView` is changed docs: A comment for message `MetricUpdate` is changed docs: A comment for field `set` in message `.google.dataflow.v1beta3.MetricUpdate` is changed docs: A comment for message `JobMetrics` is changed docs: A comment for method `CreateJobFromTemplate` in service `TemplatesService` is changed docs: A comment for method `LaunchTemplate` in service `TemplatesService` is changed docs: A comment for method `GetTemplate` in service `TemplatesService` is changed docs: A comment for service `FlexTemplatesService` is changed docs: A comment for field `dump_heap_on_oom` in message `.google.dataflow.v1beta3.FlexTemplateRuntimeEnvironment` is changed docs: A comment for field `save_heap_dumps_to_gcs_path` in message `.google.dataflow.v1beta3.FlexTemplateRuntimeEnvironment` is changed docs: A comment for field `num_workers` in message `.google.dataflow.v1beta3.RuntimeEnvironment` is changed docs: A comment for field `max_workers` in message `.google.dataflow.v1beta3.RuntimeEnvironment` is changed docs: A comment for field `zone` in message `.google.dataflow.v1beta3.RuntimeEnvironment` is changed docs: A comment for field `service_account_email` in message `.google.dataflow.v1beta3.RuntimeEnvironment` is changed docs: A comment for field `temp_location` in message `.google.dataflow.v1beta3.RuntimeEnvironment` is changed docs: A comment for field `bypass_temp_dir_validation` in message `.google.dataflow.v1beta3.RuntimeEnvironment` is changed docs: A comment for field `machine_type` in message `.google.dataflow.v1beta3.RuntimeEnvironment` is changed docs: A comment for field `additional_experiments` in message `.google.dataflow.v1beta3.RuntimeEnvironment` is changed docs: A comment for field `network` in message `.google.dataflow.v1beta3.RuntimeEnvironment` is changed docs: A comment for field `subnetwork` in message `.google.dataflow.v1beta3.RuntimeEnvironment` is changed docs: A comment for field `additional_user_labels` in message `.google.dataflow.v1beta3.RuntimeEnvironment` is changed docs: A comment for field `kms_key_name` in message `.google.dataflow.v1beta3.RuntimeEnvironment` is changed docs: A comment for field `ip_configuration` in message `.google.dataflow.v1beta3.RuntimeEnvironment` is changed docs: A comment for field `worker_region` in message `.google.dataflow.v1beta3.RuntimeEnvironment` is changed docs: A comment for field `worker_zone` in message `.google.dataflow.v1beta3.RuntimeEnvironment` is changed docs: A comment for field `enable_streaming_engine` in message `.google.dataflow.v1beta3.RuntimeEnvironment` is changed docs: A comment for message `LaunchTemplateParameters` is changed docs: A comment for field `job_name` in message `.google.dataflow.v1beta3.LaunchTemplateParameters` is changed docs: A comment for field `gcs_path` in message `.google.dataflow.v1beta3.LaunchTemplateRequest` is changed docs: A comment for field `dynamic_template` in message `.google.dataflow.v1beta3.LaunchTemplateRequest` is changed docs: A comment for field `launch_parameters` in message `.google.dataflow.v1beta3.LaunchTemplateRequest` is changed docs: A comment for message `DynamicTemplateLaunchParams` is changed docs: A comment for field `gcs_path` in message `.google.dataflow.v1beta3.DynamicTemplateLaunchParams` is changed PiperOrigin-RevId: 753588060
1 parent cccef41 commit 8105f2a

8 files changed

Lines changed: 643 additions & 206 deletions

File tree

‎google/dataflow/v1beta3/dataflow_v1beta3.yaml‎

Lines changed: 6 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -14,62 +14,35 @@ apis:
1414
documentation:
1515
summary: Manages Google Cloud Dataflow projects on Google Cloud Platform.
1616

17-
backend:
18-
rules:
19-
- selector: google.dataflow.v1beta3.FlexTemplatesService.LaunchFlexTemplate
20-
deadline: 30.0
21-
- selector: 'google.dataflow.v1beta3.JobsV1Beta3.*'
22-
deadline: 30.0
23-
- selector: google.dataflow.v1beta3.MessagesV1Beta3.ListJobMessages
24-
deadline: 30.0
25-
- selector: 'google.dataflow.v1beta3.MetricsV1Beta3.*'
26-
deadline: 30.0
27-
- selector: 'google.dataflow.v1beta3.SnapshotsV1Beta3.*'
28-
deadline: 30.0
29-
- selector: 'google.dataflow.v1beta3.TemplatesService.*'
30-
deadline: 30.0
31-
3217
authentication:
3318
rules:
3419
- selector: google.dataflow.v1beta3.FlexTemplatesService.LaunchFlexTemplate
3520
oauth:
3621
canonical_scopes: |-
3722
https://www.googleapis.com/auth/cloud-platform,
38-
https://www.googleapis.com/auth/compute,
39-
https://www.googleapis.com/auth/compute.readonly,
40-
https://www.googleapis.com/auth/userinfo.email
23+
https://www.googleapis.com/auth/compute
4124
- selector: 'google.dataflow.v1beta3.JobsV1Beta3.*'
4225
oauth:
4326
canonical_scopes: |-
4427
https://www.googleapis.com/auth/cloud-platform,
45-
https://www.googleapis.com/auth/compute,
46-
https://www.googleapis.com/auth/compute.readonly,
47-
https://www.googleapis.com/auth/userinfo.email
28+
https://www.googleapis.com/auth/compute
4829
- selector: google.dataflow.v1beta3.MessagesV1Beta3.ListJobMessages
4930
oauth:
5031
canonical_scopes: |-
5132
https://www.googleapis.com/auth/cloud-platform,
52-
https://www.googleapis.com/auth/compute,
53-
https://www.googleapis.com/auth/compute.readonly,
54-
https://www.googleapis.com/auth/userinfo.email
33+
https://www.googleapis.com/auth/compute
5534
- selector: 'google.dataflow.v1beta3.MetricsV1Beta3.*'
5635
oauth:
5736
canonical_scopes: |-
5837
https://www.googleapis.com/auth/cloud-platform,
59-
https://www.googleapis.com/auth/compute,
60-
https://www.googleapis.com/auth/compute.readonly,
61-
https://www.googleapis.com/auth/userinfo.email
38+
https://www.googleapis.com/auth/compute
6239
- selector: 'google.dataflow.v1beta3.SnapshotsV1Beta3.*'
6340
oauth:
6441
canonical_scopes: |-
6542
https://www.googleapis.com/auth/cloud-platform,
66-
https://www.googleapis.com/auth/compute,
67-
https://www.googleapis.com/auth/compute.readonly,
68-
https://www.googleapis.com/auth/userinfo.email
43+
https://www.googleapis.com/auth/compute
6944
- selector: 'google.dataflow.v1beta3.TemplatesService.*'
7045
oauth:
7146
canonical_scopes: |-
7247
https://www.googleapis.com/auth/cloud-platform,
73-
https://www.googleapis.com/auth/compute,
74-
https://www.googleapis.com/auth/compute.readonly,
75-
https://www.googleapis.com/auth/userinfo.email
48+
https://www.googleapis.com/auth/compute

‎google/dataflow/v1beta3/environment.proto‎

Lines changed: 105 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -56,17 +56,18 @@ message Environment {
5656
// field for service related experiments is service_options.
5757
repeated string experiments = 3;
5858

59-
// The list of service options to enable. This field should be used for
60-
// service related experiments only. These experiments, when graduating to GA,
61-
// should be replaced by dedicated fields or become default (i.e. always on).
62-
repeated string service_options = 16;
63-
64-
// If set, contains the Cloud KMS key identifier used to encrypt data
65-
// at rest, AKA a Customer Managed Encryption Key (CMEK).
59+
// Optional. The list of service options to enable. This field should be used
60+
// for service related experiments only. These experiments, when graduating to
61+
// GA, should be replaced by dedicated fields or become default (i.e. always
62+
// on).
63+
repeated string service_options = 16 [(google.api.field_behavior) = OPTIONAL];
64+
65+
// Optional. If set, contains the Cloud KMS key identifier used to encrypt
66+
// data at rest, AKA a Customer Managed Encryption Key (CMEK).
6667
//
6768
// Format:
6869
// projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY
69-
string service_kms_key_name = 12;
70+
string service_kms_key_name = 12 [(google.api.field_behavior) = OPTIONAL];
7071

7172
// The worker pools. At least one "harness" worker pool must be
7273
// specified in order for the job to have workers.
@@ -79,14 +80,14 @@ message Environment {
7980
// are required in order to run the job.
8081
google.protobuf.Struct version = 6;
8182

82-
// The dataset for the current project where various workflow
83+
// Optional. The dataset for the current project where various workflow
8384
// related tables are stored.
8485
//
8586
// The supported resource type is:
8687
//
8788
// Google BigQuery:
8889
// bigquery.googleapis.com/{dataset}
89-
string dataset = 7;
90+
string dataset = 7 [(google.api.field_behavior) = OPTIONAL];
9091

9192
// The Cloud Dataflow SDK pipeline options specified by the user. These
9293
// options are passed through the service and are used to recreate the
@@ -97,31 +98,47 @@ message Environment {
9798
// Experimental settings.
9899
google.protobuf.Any internal_experiments = 9;
99100

100-
// Identity to run virtual machines as. Defaults to the default account.
101-
string service_account_email = 10;
101+
// Optional. Identity to run virtual machines as. Defaults to the default
102+
// account.
103+
string service_account_email = 10 [(google.api.field_behavior) = OPTIONAL];
102104

103-
// Which Flexible Resource Scheduling mode to run in.
104-
FlexResourceSchedulingGoal flex_resource_scheduling_goal = 11;
105+
// Optional. Which Flexible Resource Scheduling mode to run in.
106+
FlexResourceSchedulingGoal flex_resource_scheduling_goal = 11
107+
[(google.api.field_behavior) = OPTIONAL];
105108

106-
// The Compute Engine region
109+
// Optional. The Compute Engine region
107110
// (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in
108111
// which worker processing should occur, e.g. "us-west1". Mutually exclusive
109112
// with worker_zone. If neither worker_region nor worker_zone is specified,
110113
// default to the control plane's region.
111-
string worker_region = 13;
114+
string worker_region = 13 [(google.api.field_behavior) = OPTIONAL];
112115

113-
// The Compute Engine zone
116+
// Optional. The Compute Engine zone
114117
// (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in
115118
// which worker processing should occur, e.g. "us-west1-a". Mutually exclusive
116119
// with worker_region. If neither worker_region nor worker_zone is specified,
117120
// a zone in the control plane's region is chosen based on available capacity.
118-
string worker_zone = 14;
121+
string worker_zone = 14 [(google.api.field_behavior) = OPTIONAL];
119122

120123
// Output only. The shuffle mode used for the job.
121124
ShuffleMode shuffle_mode = 15 [(google.api.field_behavior) = OUTPUT_ONLY];
122125

123-
// Any debugging options to be supplied to the job.
124-
DebugOptions debug_options = 17;
126+
// Optional. Any debugging options to be supplied to the job.
127+
DebugOptions debug_options = 17 [(google.api.field_behavior) = OPTIONAL];
128+
129+
// Output only. Whether the job uses the Streaming Engine resource-based
130+
// billing model.
131+
bool use_streaming_engine_resource_based_billing = 18
132+
[(google.api.field_behavior) = OUTPUT_ONLY];
133+
134+
// Optional. Specifies the Streaming Engine message processing guarantees.
135+
// Reduces cost and latency but might result in duplicate messages committed
136+
// to storage. Designed to run simple mapping streaming ETL jobs at the lowest
137+
// cost. For example, Change Data Capture (CDC) to BigQuery is a canonical use
138+
// case. For more information, see
139+
// [Set the pipeline streaming
140+
// mode](https://cloud.google.com/dataflow/docs/guides/streaming-modes).
141+
StreamingMode streaming_mode = 19 [(google.api.field_behavior) = OPTIONAL];
125142
}
126143

127144
// The packages that must be installed in order for a worker to run the
@@ -388,7 +405,10 @@ message AutoscalingSettings {
388405
int32 max_num_workers = 2;
389406
}
390407

391-
// Specifies how IP addresses should be allocated to the worker machines.
408+
// Specifies how to allocate IP addresses to worker machines. You can also use
409+
// [pipeline
410+
// options](https://cloud.google.com/dataflow/docs/reference/pipeline-options#security_and_networking)
411+
// to specify whether Dataflow workers use external IP addresses.
392412
enum WorkerIPAddressConfiguration {
393413
// The configuration is unknown, or unspecified.
394414
WORKER_IP_UNSPECIFIED = 0;
@@ -400,7 +420,7 @@ enum WorkerIPAddressConfiguration {
400420
WORKER_IP_PRIVATE = 2;
401421
}
402422

403-
// Defines a SDK harness container for executing Dataflow pipelines.
423+
// Defines an SDK harness container for executing Dataflow pipelines.
404424
message SdkHarnessContainerImage {
405425
// A docker container image that resides in Google Container Registry.
406426
string container_image = 1;
@@ -417,7 +437,7 @@ message SdkHarnessContainerImage {
417437
string environment_id = 3;
418438

419439
// The set of capabilities enumerated in the above Environment proto. See also
420-
// https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto
440+
// [beam_runner_api.proto](https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto)
421441
repeated string capabilities = 4;
422442
}
423443

@@ -546,9 +566,68 @@ enum ShuffleMode {
546566
SERVICE_BASED = 2;
547567
}
548568

569+
// Configuration options for sampling elements.
570+
message DataSamplingConfig {
571+
// The following enum defines what to sample for a running job.
572+
enum DataSamplingBehavior {
573+
// If given, has no effect on sampling behavior. Used as an unknown or unset
574+
// sentinel value.
575+
DATA_SAMPLING_BEHAVIOR_UNSPECIFIED = 0;
576+
577+
// When given, disables element sampling. Has same behavior as not setting
578+
// the behavior.
579+
DISABLED = 1;
580+
581+
// When given, enables sampling in-flight from all PCollections.
582+
ALWAYS_ON = 2;
583+
584+
// When given, enables sampling input elements when a user-defined DoFn
585+
// causes an exception.
586+
EXCEPTIONS = 3;
587+
}
588+
589+
// List of given sampling behaviors to enable. For example, specifying
590+
// behaviors = [ALWAYS_ON] samples in-flight elements but does not sample
591+
// exceptions. Can be used to specify multiple behaviors like,
592+
// behaviors = [ALWAYS_ON, EXCEPTIONS] for specifying periodic sampling and
593+
// exception sampling.
594+
//
595+
// If DISABLED is in the list, then sampling will be disabled and ignore the
596+
// other given behaviors.
597+
//
598+
// Ordering does not matter.
599+
repeated DataSamplingBehavior behaviors = 1;
600+
}
601+
549602
// Describes any options that have an effect on the debugging of pipelines.
550603
message DebugOptions {
551-
// When true, enables the logging of the literal hot key to the user's Cloud
552-
// Logging.
553-
bool enable_hot_key_logging = 1;
604+
// Optional. When true, enables the logging of the literal hot key to the
605+
// user's Cloud Logging.
606+
bool enable_hot_key_logging = 1 [(google.api.field_behavior) = OPTIONAL];
607+
608+
// Configuration options for sampling elements from a running pipeline.
609+
DataSamplingConfig data_sampling = 2;
610+
}
611+
612+
// Specifies the Streaming Engine message processing guarantees. Reduces cost
613+
// and latency but might result in duplicate messages written to storage.
614+
// Designed to run simple mapping streaming ETL jobs at the lowest cost.
615+
// For example, Change Data Capture (CDC) to BigQuery is a canonical use
616+
// case. For more information, see
617+
// [Set the pipeline streaming
618+
// mode](https://cloud.google.com/dataflow/docs/guides/streaming-modes).
619+
enum StreamingMode {
620+
// Run in the default mode.
621+
STREAMING_MODE_UNSPECIFIED = 0;
622+
623+
// In this mode, message deduplication is performed against persistent state
624+
// to make sure each message is processed and committed to storage exactly
625+
// once.
626+
STREAMING_MODE_EXACTLY_ONCE = 1;
627+
628+
// Message deduplication is not performed. Messages might be processed
629+
// multiple times, and the results are applied multiple times.
630+
// Note: Setting this value also enables Streaming Engine and
631+
// Streaming Engine resource-based billing.
632+
STREAMING_MODE_AT_LEAST_ONCE = 2;
554633
}

0 commit comments

Comments
 (0)