For AI agents: A markdown version of this page is available at https://docs.datadoghq.com/infrastructure/resource_catalog/gcp_dataproc_workflow_template.md. A documentation index is available at /llms.txt.

gcp_dataproc_workflow_template

ancestors

Type: UNORDERED_LIST_STRING

create_time

Type: TIMESTAMP
Provider name: createTime
Description: Output only. The time template was created.

dag_timeout

Type: STRING
Provider name: dagTimeout
Description: Optional. Timeout duration for the DAG of jobs, expressed in seconds (see JSON representation of duration (https://developers.google.com/protocol-buffers/docs/proto3#json)). The timeout duration must be from 10 minutes (“600s”) to 24 hours (“86400s”). The timer begins when the first job is submitted. If the workflow is running at the end of the timeout period, any remaining jobs are cancelled, the workflow is ended, and if the workflow was running on a managed cluster, the cluster is deleted.

encryption_config

Type: STRUCT
Provider name: encryptionConfig
Description: Optional. Encryption settings for encrypting workflow template job arguments.

id

Type: STRING
Provider name: id

jobs

Type: UNORDERED_LIST_STRUCT
Provider name: jobs
Description: Required. The Directed Acyclic Graph of Jobs to submit.

  • flink_job
    Type: STRUCT
    Provider name: flinkJob
    Description: Optional. Job is a Flink job.

    • args
      Type: UNORDERED_LIST_STRING
      Provider name: args
      Description: Optional. The arguments to pass to the driver. Do not include arguments, such as –conf, that can be set as job properties, since a collision might occur that causes an incorrect job submission.

    • jar_file_uris
      Type: UNORDERED_LIST_STRING
      Provider name: jarFileUris
      Description: Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Flink driver and tasks.

    • logging_config
      Type: STRUCT
      Provider name: loggingConfig
      Description: Optional. The runtime log config for job execution.

    • main_class
      Type: STRING
      Provider name: mainClass
      Description: The name of the driver’s main class. The jar file that contains the class must be in the default CLASSPATH or specified in jarFileUris.

    • main_jar_file_uri
      Type: STRING
      Provider name: mainJarFileUri
      Description: The HCFS URI of the jar file that contains the main class.

    • savepoint_uri
      Type: STRING
      Provider name: savepointUri
      Description: Optional. HCFS URI of the savepoint, which contains the last saved progress for starting the current job.

  • hadoop_job
    Type: STRUCT
    Provider name: hadoopJob
    Description: Optional. Job is a Hadoop job.

    • archive_uris
      Type: UNORDERED_LIST_STRING
      Provider name: archiveUris
      Description: Optional. HCFS URIs of archives to be extracted in the working directory of Hadoop drivers and tasks. Supported file types: .jar, .tar, .tar.gz, .tgz, or .zip.

    • args
      Type: UNORDERED_LIST_STRING
      Provider name: args
      Description: Optional. The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision might occur that causes an incorrect job submission.

    • file_uris
      Type: UNORDERED_LIST_STRING
      Provider name: fileUris
      Description: Optional. HCFS (Hadoop Compatible Filesystem) URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.

    • jar_file_uris
      Type: UNORDERED_LIST_STRING
      Provider name: jarFileUris
      Description: Optional. Jar file URIs to add to the CLASSPATHs of the Hadoop driver and tasks.

    • logging_config
      Type: STRUCT
      Provider name: loggingConfig
      Description: Optional. The runtime log config for job execution.

    • main_class
      Type: STRING
      Provider name: mainClass
      Description: The name of the driver’s main class. The jar file containing the class must be in the default CLASSPATH or specified in jar_file_uris.

    • main_jar_file_uri
      Type: STRING
      Provider name: mainJarFileUri
      Description: The HCFS URI of the jar file containing the main class. Examples: ‘gs://foo-bucket/analytics-binaries/extract-useful-metrics-mr.jar’ ‘hdfs:/tmp/test-samples/custom-wordcount.jar’ ‘file:///home/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar’

  • hive_job
    Type: STRUCT
    Provider name: hiveJob
    Description: Optional. Job is a Hive job.
    • continue_on_failure
      Type: BOOLEAN
      Provider name: continueOnFailure
      Description: Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
    • jar_file_uris
      Type: UNORDERED_LIST_STRING
      Provider name: jarFileUris
      Description: Optional. HCFS URIs of jar files to add to the CLASSPATH of the Hive server and Hadoop MapReduce (MR) tasks. Can contain Hive SerDes and UDFs.
    • query_file_uri
      Type: STRING
      Provider name: queryFileUri
      Description: The HCFS URI of the script that contains Hive queries.
    • query_list
      Type: STRUCT
      Provider name: queryList
      Description: A list of queries.
      • queries
        Type: UNORDERED_LIST_STRING
        Provider name: queries
        Description: Required. The queries to execute. You do not need to end a query expression with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of a Dataproc API snippet that uses a QueryList to specify a HiveJob: “hiveJob”: { “queryList”: { “queries”: [ “query1”, “query2”, “query3;query4”, ] } }
  • pig_job
    Type: STRUCT
    Provider name: pigJob
    Description: Optional. Job is a Pig job.

    • continue_on_failure
      Type: BOOLEAN
      Provider name: continueOnFailure
      Description: Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.

    • jar_file_uris
      Type: UNORDERED_LIST_STRING
      Provider name: jarFileUris
      Description: Optional. HCFS URIs of jar files to add to the CLASSPATH of the Pig Client and Hadoop MapReduce (MR) tasks. Can contain Pig UDFs.

    • logging_config
      Type: STRUCT
      Provider name: loggingConfig
      Description: Optional. The runtime log config for job execution.

    • query_file_uri
      Type: STRING
      Provider name: queryFileUri
      Description: The HCFS URI of the script that contains the Pig queries.

    • query_list
      Type: STRUCT
      Provider name: queryList
      Description: A list of queries.

      • queries
        Type: UNORDERED_LIST_STRING
        Provider name: queries
        Description: Required. The queries to execute. You do not need to end a query expression with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of a Dataproc API snippet that uses a QueryList to specify a HiveJob: “hiveJob”: { “queryList”: { “queries”: [ “query1”, “query2”, “query3;query4”, ] } }
  • prerequisite_step_ids
    Type: UNORDERED_LIST_STRING
    Provider name: prerequisiteStepIds
    Description: Optional. The optional list of prerequisite job step_ids. If not specified, the job will start at the beginning of workflow.
  • presto_job
    Type: STRUCT
    Provider name: prestoJob
    Description: Optional. Job is a Presto job.

    • client_tags
      Type: UNORDERED_LIST_STRING
      Provider name: clientTags
      Description: Optional. Presto client tags to attach to this query

    • continue_on_failure
      Type: BOOLEAN
      Provider name: continueOnFailure
      Description: Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.

    • logging_config
      Type: STRUCT
      Provider name: loggingConfig
      Description: Optional. The runtime log config for job execution.

    • output_format
      Type: STRING
      Provider name: outputFormat
      Description: Optional. The format in which query output will be displayed. See the Presto documentation for supported output formats

    • query_file_uri
      Type: STRING
      Provider name: queryFileUri
      Description: The HCFS URI of the script that contains SQL queries.

    • query_list
      Type: STRUCT
      Provider name: queryList
      Description: A list of queries.

      • queries
        Type: UNORDERED_LIST_STRING
        Provider name: queries
        Description: Required. The queries to execute. You do not need to end a query expression with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of a Dataproc API snippet that uses a QueryList to specify a HiveJob: “hiveJob”: { “queryList”: { “queries”: [ “query1”, “query2”, “query3;query4”, ] } }
  • pyspark_job
    Type: STRUCT
    Provider name: pysparkJob
    Description: Optional. Job is a PySpark job.

    • archive_uris
      Type: UNORDERED_LIST_STRING
      Provider name: archiveUris
      Description: Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.

    • args
      Type: UNORDERED_LIST_STRING
      Provider name: args
      Description: Optional. The arguments to pass to the driver. Do not include arguments, such as –conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.

    • file_uris
      Type: UNORDERED_LIST_STRING
      Provider name: fileUris
      Description: Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.

    • jar_file_uris
      Type: UNORDERED_LIST_STRING
      Provider name: jarFileUris
      Description: Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks.

    • logging_config
      Type: STRUCT
      Provider name: loggingConfig
      Description: Optional. The runtime log config for job execution.

    • main_python_file_uri
      Type: STRING
      Provider name: mainPythonFileUri
      Description: Required. The HCFS URI of the main Python file to use as the driver. Must be a .py file.

    • python_file_uris
      Type: UNORDERED_LIST_STRING
      Provider name: pythonFileUris
      Description: Optional. HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.

  • scheduling
    Type: STRUCT
    Provider name: scheduling
    Description: Optional. Job scheduling configuration.
  • spark_job
    Type: STRUCT
    Provider name: sparkJob
    Description: Optional. Job is a Spark job.

    • archive_uris
      Type: UNORDERED_LIST_STRING
      Provider name: archiveUris
      Description: Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.

    • args
      Type: UNORDERED_LIST_STRING
      Provider name: args
      Description: Optional. The arguments to pass to the driver. Do not include arguments, such as –conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.

    • file_uris
      Type: UNORDERED_LIST_STRING
      Provider name: fileUris
      Description: Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.

    • jar_file_uris
      Type: UNORDERED_LIST_STRING
      Provider name: jarFileUris
      Description: Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.

    • logging_config
      Type: STRUCT
      Provider name: loggingConfig
      Description: Optional. The runtime log config for job execution.

    • main_class
      Type: STRING
      Provider name: mainClass
      Description: The name of the driver’s main class. The jar file that contains the class must be in the default CLASSPATH or specified in SparkJob.jar_file_uris.

    • main_jar_file_uri
      Type: STRING
      Provider name: mainJarFileUri
      Description: The HCFS URI of the jar file that contains the main class.

  • spark_r_job
    Type: STRUCT
    Provider name: sparkRJob
    Description: Optional. Job is a SparkR job.

    • archive_uris
      Type: UNORDERED_LIST_STRING
      Provider name: archiveUris
      Description: Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.

    • args
      Type: UNORDERED_LIST_STRING
      Provider name: args
      Description: Optional. The arguments to pass to the driver. Do not include arguments, such as –conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.

    • file_uris
      Type: UNORDERED_LIST_STRING
      Provider name: fileUris
      Description: Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.

    • logging_config
      Type: STRUCT
      Provider name: loggingConfig
      Description: Optional. The runtime log config for job execution.

    • main_r_file_uri
      Type: STRING
      Provider name: mainRFileUri
      Description: Required. The HCFS URI of the main R file to use as the driver. Must be a .R file.

  • spark_sql_job
    Type: STRUCT
    Provider name: sparkSqlJob
    Description: Optional. Job is a SparkSql job.

    • jar_file_uris
      Type: UNORDERED_LIST_STRING
      Provider name: jarFileUris
      Description: Optional. HCFS URIs of jar files to be added to the Spark CLASSPATH.

    • logging_config
      Type: STRUCT
      Provider name: loggingConfig
      Description: Optional. The runtime log config for job execution.

    • query_file_uri
      Type: STRING
      Provider name: queryFileUri
      Description: The HCFS URI of the script that contains SQL queries.

    • query_list
      Type: STRUCT
      Provider name: queryList
      Description: A list of queries.

      • queries
        Type: UNORDERED_LIST_STRING
        Provider name: queries
        Description: Required. The queries to execute. You do not need to end a query expression with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of a Dataproc API snippet that uses a QueryList to specify a HiveJob: “hiveJob”: { “queryList”: { “queries”: [ “query1”, “query2”, “query3;query4”, ] } }
  • step_id
    Type: STRING
    Provider name: stepId
    Description: Required. The step id. The id must be unique among all jobs within the template.The step id is used as prefix for job id, as job goog-dataproc-workflow-step-id label, and in prerequisiteStepIds field from other steps.The id must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), and hyphens (-). Cannot begin or end with underscore or hyphen. Must consist of between 3 and 50 characters.
  • trino_job
    Type: STRUCT
    Provider name: trinoJob
    Description: Optional. Job is a Trino job.

    • client_tags
      Type: UNORDERED_LIST_STRING
      Provider name: clientTags
      Description: Optional. Trino client tags to attach to this query

    • continue_on_failure
      Type: BOOLEAN
      Provider name: continueOnFailure
      Description: Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.

    • logging_config
      Type: STRUCT
      Provider name: loggingConfig
      Description: Optional. The runtime log config for job execution.

    • output_format
      Type: STRING
      Provider name: outputFormat
      Description: Optional. The format in which query output will be displayed. See the Trino documentation for supported output formats

    • query_file_uri
      Type: STRING
      Provider name: queryFileUri
      Description: The HCFS URI of the script that contains SQL queries.

    • query_list
      Type: STRUCT
      Provider name: queryList
      Description: A list of queries.

      • queries
        Type: UNORDERED_LIST_STRING
        Provider name: queries
        Description: Required. The queries to execute. You do not need to end a query expression with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of a Dataproc API snippet that uses a QueryList to specify a HiveJob: “hiveJob”: { “queryList”: { “queries”: [ “query1”, “query2”, “query3;query4”, ] } }

labels

Type: UNORDERED_LIST_STRING

name

Type: STRING
Provider name: name
Description: Output only. The resource name of the workflow template, as described in https://cloud.google.com/apis/design/resource_names. For projects.regions.workflowTemplates, the resource name of the template has the following format: projects/{project_id}/regions/{region}/workflowTemplates/{template_id} For projects.locations.workflowTemplates, the resource name of the template has the following format: projects/{project_id}/locations/{location}/workflowTemplates/{template_id}

organization_id

Type: STRING

parameters

Type: UNORDERED_LIST_STRUCT
Provider name: parameters
Description: Optional. Template parameters whose values are substituted into the template. Values for parameters must be provided when the template is instantiated.

  • description
    Type: STRING
    Provider name: description
    Description: Optional. Brief description of the parameter. Must not exceed 1024 characters.
  • fields
    Type: UNORDERED_LIST_STRING
    Provider name: fields
    Description: Required. Paths to all fields that the parameter replaces. A field is allowed to appear in at most one parameter’s list of field paths.A field path is similar in syntax to a google.protobuf.FieldMask. For example, a field path that references the zone field of a workflow template’s cluster selector would be specified as placement.clusterSelector.zone.Also, field paths can reference fields using the following syntax: Values in maps can be referenced by key: labels’key’ placement.clusterSelector.clusterLabels’key’ placement.managedCluster.labels’key’ placement.clusterSelector.clusterLabels’key’ jobs’step-id’.labels’key’ Jobs in the jobs list can be referenced by step-id: jobs’step-id’.hadoopJob.mainJarFileUri jobs’step-id’.hiveJob.queryFileUri jobs’step-id’.pySparkJob.mainPythonFileUri jobs’step-id’.hadoopJob.jarFileUris0 jobs’step-id’.hadoopJob.archiveUris0 jobs’step-id’.hadoopJob.fileUris0 jobs’step-id’.pySparkJob.pythonFileUris0 Items in repeated fields can be referenced by a zero-based index: jobs’step-id’.sparkJob.args0 Other examples: jobs’step-id’.hadoopJob.properties’key’ jobs’step-id’.hadoopJob.args0 jobs’step-id’.hiveJob.scriptVariables’key’ jobs’step-id’.hadoopJob.mainJarFileUri placement.clusterSelector.zoneIt may not be possible to parameterize maps and repeated fields in their entirety since only individual map values and individual items in repeated fields can be referenced. For example, the following field paths are invalid: placement.clusterSelector.clusterLabels jobs’step-id’.sparkJob.args
  • name
    Type: STRING
    Provider name: name
    Description: Required. Parameter name. The parameter name is used as the key, and paired with the parameter value, which are passed to the template when the template is instantiated. The name must contain only capital letters (A-Z), numbers (0-9), and underscores (_), and must not start with a number. The maximum length is 40 characters.
  • validation
    Type: STRUCT
    Provider name: validation
    Description: Optional. Validation rules to be applied to this parameter’s value.
    • regex
      Type: STRUCT
      Provider name: regex
      Description: Validation based on regular expressions.
      • regexes
        Type: UNORDERED_LIST_STRING
        Provider name: regexes
        Description: Required. RE2 regular expressions used to validate the parameter’s value. The value must match the regex in its entirety (substring matches are not sufficient).
    • values
      Type: STRUCT
      Provider name: values
      Description: Validation based on a list of allowed values.
      • values
        Type: UNORDERED_LIST_STRING
        Provider name: values
        Description: Required. List of allowed values for the parameter.

parent

Type: STRING

placement

Type: STRUCT
Provider name: placement
Description: Required. WorkflowTemplate scheduling information.

  • cluster_selector
    Type: STRUCT
    Provider name: clusterSelector
    Description: Optional. A selector that chooses target cluster for jobs based on metadata.The selector is evaluated at the time each job is submitted.
    • zone
      Type: STRING
      Provider name: zone
      Description: Optional. The zone where workflow process executes. This parameter does not affect the selection of the cluster.If unspecified, the zone of the first cluster matching the selector is used.
  • managed_cluster
    Type: STRUCT
    Provider name: managedCluster
    Description: A cluster that is managed by the workflow.
    • cluster_name
      Type: STRING
      Provider name: clusterName
      Description: Required. The cluster name prefix. A unique cluster name will be formed by appending a random suffix.The name must contain only lower-case letters (a-z), numbers (0-9), and hyphens (-). Must begin with a letter. Cannot begin or end with hyphen. Must consist of between 2 and 35 characters.
    • config
      Type: STRUCT
      Provider name: config
      Description: Required. The cluster configuration.
      • autoscaling_config
        Type: STRUCT
        Provider name: autoscalingConfig
        Description: Optional. Autoscaling config for the policy associated with the cluster. Cluster does not autoscale if this field is unset.
      • auxiliary_node_groups
        Type: UNORDERED_LIST_STRUCT
        Provider name: auxiliaryNodeGroups
        Description: Optional. The node group settings.
        • node_group
          Type: STRUCT
          Provider name: nodeGroup
          Description: Required. Node group configuration.
          • name
            Type: STRING
            Provider name: name
            Description: The Node group resource name (https://aip.dev/122).
          • node_group_config
            Type: STRUCT
            Provider name: nodeGroupConfig
            Description: Optional. The node group instance group configuration.
            • accelerators
              Type: UNORDERED_LIST_STRUCT
              Provider name: accelerators
              Description: Optional. The Compute Engine accelerator configuration for these instances.
            • disk_config
              Type: STRUCT
              Provider name: diskConfig
              Description: Optional. Disk option config settings.
              • boot_disk_provisioned_iops
                Type: INT64
                Provider name: bootDiskProvisionedIops
                Description: Optional. Indicates how many IOPS to provision for the disk. This sets the number of I/O operations per second that the disk can handle. This field is supported only if boot_disk_type is hyperdisk-balanced.
              • boot_disk_provisioned_throughput
                Type: INT64
                Provider name: bootDiskProvisionedThroughput
                Description: Optional. Indicates how much throughput to provision for the disk. This sets the number of throughput mb per second that the disk can handle. Values must be greater than or equal to 1. This field is supported only if boot_disk_type is hyperdisk-balanced.
              • boot_disk_size_gb
                Type: INT32
                Provider name: bootDiskSizeGb
                Description: Optional. Size in GB of the boot disk (default is 500GB).
              • boot_disk_type
                Type: STRING
                Provider name: bootDiskType
                Description: Optional. Type of the boot disk (default is “pd-standard”). Valid values: “pd-balanced” (Persistent Disk Balanced Solid State Drive), “pd-ssd” (Persistent Disk Solid State Drive), or “pd-standard” (Persistent Disk Hard Disk Drive). See Disk types (https://cloud.google.com/compute/docs/disks#disk-types).
              • local_ssd_interface
                Type: STRING
                Provider name: localSsdInterface
                Description: Optional. Interface type of local SSDs (default is “scsi”). Valid values: “scsi” (Small Computer System Interface), “nvme” (Non-Volatile Memory Express). See local SSD performance (https://cloud.google.com/compute/docs/disks/local-ssd#performance).
              • num_local_ssds
                Type: INT32
                Provider name: numLocalSsds
                Description: Optional. Number of attached SSDs, from 0 to 8 (default is 0). If SSDs are not attached, the boot disk is used to store runtime logs and HDFS (https://hadoop.apache.org/docs/r1.2.1/hdfs_user_guide.html) data. If one or more SSDs are attached, this runtime bulk data is spread across them, and the boot disk contains only basic config and installed binaries.Note: Local SSD options may vary by machine type and number of vCPUs selected.
            • image_uri
              Type: STRING
              Provider name: imageUri
              Description: Optional. The Compute Engine image resource used for cluster instances.The URI can represent an image or image family.Image examples: https://www.googleapis.com/compute/v1/projects/[project_id]/global/images/[image-id] projects/[project_id]/global/images/[image-id] image-idImage family examples. Dataproc will use the most recent image from the family: https://www.googleapis.com/compute/v1/projects/[project_id]/global/images/family/[custom-image-family-name] projects/[project_id]/global/images/family/[custom-image-family-name]If the URI is unspecified, it will be inferred from SoftwareConfig.image_version or the system default.
            • instance_flexibility_policy
              Type: STRUCT
              Provider name: instanceFlexibilityPolicy
              Description: Optional. Instance flexibility Policy allowing a mixture of VM shapes and provisioning models.
              • instance_selection_list
                Type: UNORDERED_LIST_STRUCT
                Provider name: instanceSelectionList
                Description: Optional. List of instance selection options that the group will use when creating new VMs.
                • machine_types
                  Type: UNORDERED_LIST_STRING
                  Provider name: machineTypes
                  Description: Optional. Full machine-type names, e.g. “n1-standard-16”.
                • rank
                  Type: INT32
                  Provider name: rank
                  Description: Optional. Preference of this instance selection. Lower number means higher preference. Dataproc will first try to create a VM based on the machine-type with priority rank and fallback to next rank based on availability. Machine types and instance selections with the same priority have the same preference.
              • instance_selection_results
                Type: UNORDERED_LIST_STRUCT
                Provider name: instanceSelectionResults
                Description: Output only. A list of instance selection results in the group.
                • machine_type
                  Type: STRING
                  Provider name: machineType
                  Description: Output only. Full machine-type names, e.g. “n1-standard-16”.
                • vm_count
                  Type: INT32
                  Provider name: vmCount
                  Description: Output only. Number of VM provisioned with the machine_type.
              • provisioning_model_mix
                Type: STRUCT
                Provider name: provisioningModelMix
                Description: Optional. Defines how the Group selects the provisioning model to ensure required reliability.
                • standard_capacity_base
                  Type: INT32
                  Provider name: standardCapacityBase
                  Description: Optional. The base capacity that will always use Standard VMs to avoid risk of more preemption than the minimum capacity you need. Dataproc will create only standard VMs until it reaches standard_capacity_base, then it will start using standard_capacity_percent_above_base to mix Spot with Standard VMs. eg. If 15 instances are requested and standard_capacity_base is 5, Dataproc will create 5 standard VMs and then start mixing spot and standard VMs for remaining 10 instances.
                • standard_capacity_percent_above_base
                  Type: INT32
                  Provider name: standardCapacityPercentAboveBase
                  Description: Optional. The percentage of target capacity that should use Standard VM. The remaining percentage will use Spot VMs. The percentage applies only to the capacity above standard_capacity_base. eg. If 15 instances are requested and standard_capacity_base is 5 and standard_capacity_percent_above_base is 30, Dataproc will create 5 standard VMs and then start mixing spot and standard VMs for remaining 10 instances. The mix will be 30% standard and 70% spot.
            • instance_names
              Type: UNORDERED_LIST_STRING
              Provider name: instanceNames
              Description: Output only. The list of instance names. Dataproc derives the names from cluster_name, num_instances, and the instance group.
            • instance_references
              Type: UNORDERED_LIST_STRUCT
              Provider name: instanceReferences
              Description: Output only. List of references to Compute Engine instances.
              • instance_id
                Type: STRING
                Provider name: instanceId
                Description: The unique identifier of the Compute Engine instance.
              • instance_name
                Type: STRING
                Provider name: instanceName
                Description: The user-friendly name of the Compute Engine instance.
              • public_ecies_key
                Type: STRING
                Provider name: publicEciesKey
                Description: The public ECIES key used for sharing data with this instance.
              • public_key
                Type: STRING
                Provider name: publicKey
                Description: The public RSA key used for sharing data with this instance.
            • is_preemptible
              Type: BOOLEAN
              Provider name: isPreemptible
              Description: Output only. Specifies that this instance group contains preemptible instances.
            • machine_type_uri
              Type: STRING
              Provider name: machineTypeUri
              Description: Optional. The Compute Engine machine type used for cluster instances.A full URL, partial URI, or short name are valid. Examples: https://www.googleapis.com/compute/v1/projects/[project_id]/zones/[zone]/machineTypes/n1-standard-2 projects/[project_id]/zones/[zone]/machineTypes/n1-standard-2 n1-standard-2Auto Zone Exception: If you are using the Dataproc Auto Zone Placement (https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/auto-zone#using_auto_zone_placement) feature, you must use the short name of the machine type resource, for example, n1-standard-2.
            • managed_group_config
              Type: STRUCT
              Provider name: managedGroupConfig
              Description: Output only. The config for Compute Engine Instance Group Manager that manages this group. This is only used for preemptible instance groups.
              • instance_group_manager_name
                Type: STRING
                Provider name: instanceGroupManagerName
                Description: Output only. The name of the Instance Group Manager for this group.
              • instance_group_manager_uri
                Type: STRING
                Provider name: instanceGroupManagerUri
                Description: Output only. The partial URI to the instance group manager for this group. E.g. projects/my-project/regions/us-central1/instanceGroupManagers/my-igm.
              • instance_template_name
                Type: STRING
                Provider name: instanceTemplateName
                Description: Output only. The name of the Instance Template used for the Managed Instance Group.
            • min_cpu_platform
              Type: STRING
              Provider name: minCpuPlatform
              Description: Optional. Specifies the minimum cpu platform for the Instance Group. See Dataproc -> Minimum CPU Platform (https://cloud.google.com/dataproc/docs/concepts/compute/dataproc-min-cpu).
            • min_num_instances
              Type: INT32
              Provider name: minNumInstances
              Description: Optional. The minimum number of primary worker instances to create. If min_num_instances is set, cluster creation will succeed if the number of primary workers created is at least equal to the min_num_instances number.Example: Cluster creation request with num_instances = 5 and min_num_instances = 3: If 4 VMs are created and 1 instance fails, the failed VM is deleted. The cluster is resized to 4 instances and placed in a RUNNING state. If 2 instances are created and 3 instances fail, the cluster in placed in an ERROR state. The failed VMs are not deleted.
            • num_instances
              Type: INT32
              Provider name: numInstances
              Description: Optional. The number of VM instances in the instance group. For HA cluster master_config groups, must be set to 3. For standard cluster master_config groups, must be set to 1.
            • preemptibility
              Type: STRING
              Provider name: preemptibility
              Description: Optional. Specifies the preemptibility of the instance group.The default value for master and worker groups is NON_PREEMPTIBLE. This default cannot be changed.The default value for secondary instances is PREEMPTIBLE.
              Possible values:
            • startup_config
              Type: STRUCT
              Provider name: startupConfig
              Description: Optional. Configuration to handle the startup of instances during cluster create and update process.
              • required_registration_fraction
                Type: DOUBLE
                Provider name: requiredRegistrationFraction
                Description: Optional. The config setting to enable cluster creation/ updation to be successful only after required_registration_fraction of instances are up and running. This configuration is applicable to only secondary workers for now. The cluster will fail if required_registration_fraction of instances are not available. This will include instance creation, agent registration, and service registration (if enabled).
          • roles
            Type: UNORDERED_LIST_STRING
            Provider name: roles
            Description: Required. Node group roles.
        • node_group_id
          Type: STRING
          Provider name: nodeGroupId
          Description: Optional. A node group ID. Generated if not specified.The ID must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), and hyphens (-). Cannot begin or end with underscore or hyphen. Must consist of from 3 to 33 characters.
      • config_bucket
        Type: STRING
        Provider name: configBucket
        Description: Optional. A Cloud Storage bucket used to stage job dependencies, config files, and job driver console output. If you do not specify a staging bucket, Cloud Dataproc will determine a Cloud Storage location (US, ASIA, or EU) for your cluster’s staging bucket according to the Compute Engine zone where your cluster is deployed, and then create and manage this project-level, per-location bucket (see Dataproc staging and temp buckets (https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/staging-bucket)). This field requires a Cloud Storage bucket name, not a gs://… URI to a Cloud Storage bucket.
      • dataproc_metric_config
        Type: STRUCT
        Provider name: dataprocMetricConfig
        Description: Optional. The config for Dataproc metrics.
        • metrics
          Type: UNORDERED_LIST_STRUCT
          Provider name: metrics
          Description: Required. Metrics sources to enable.
          • metric_overrides
            Type: UNORDERED_LIST_STRING
            Provider name: metricOverrides
            Description: Optional. Specify one or more Custom metrics (https://cloud.google.com/dataproc/docs/guides/dataproc-metrics#custom_metrics) to collect for the metric course (for the SPARK metric source (any Spark metric (https://spark.apache.org/docs/latest/monitoring.html#metrics) can be specified).Provide metrics in the following format: METRIC_SOURCE: INSTANCE:GROUP:METRIC Use camelcase as appropriate.Examples: yarn:ResourceManager:QueueMetrics:AppsCompleted spark:driver:DAGScheduler:job.allJobs sparkHistoryServer:JVM:Memory:NonHeapMemoryUsage.committed hiveserver2:JVM:Memory:NonHeapMemoryUsage.used Notes: Only the specified overridden metrics are collected for the metric source. For example, if one or more spark:executive metrics are listed as metric overrides, other SPARK metrics are not collected. The collection of the metrics for other enabled custom metric sources is unaffected. For example, if both SPARK and YARN metric sources are enabled, and overrides are provided for Spark metrics only, all YARN metrics are collected.
          • metric_source
            Type: STRING
            Provider name: metricSource
            Description: Required. A standard set of metrics is collected unless metricOverrides are specified for the metric source (see Custom metrics (https://cloud.google.com/dataproc/docs/guides/dataproc-metrics#custom_metrics) for more information).
            Possible values:
            • METRIC_SOURCE_UNSPECIFIED - Required unspecified metric source.
            • MONITORING_AGENT_DEFAULTS - Monitoring agent metrics. If this source is enabled, Dataproc enables the monitoring agent in Compute Engine, and collects monitoring agent metrics, which are published with an agent.googleapis.com prefix.
            • HDFS - HDFS metric source.
            • SPARK - Spark metric source.
            • YARN - YARN metric source.
            • SPARK_HISTORY_SERVER - Spark History Server metric source.
            • HIVESERVER2 - Hiveserver2 metric source.
            • HIVEMETASTORE - hivemetastore metric source
            • FLINK - flink metric source
      • encryption_config
        Type: STRUCT
        Provider name: encryptionConfig
        Description: Optional. Encryption settings for the cluster.
      • endpoint_config
        Type: STRUCT
        Provider name: endpointConfig
        Description: Optional. Port/endpoint configuration for this cluster
        • enable_http_port_access
          Type: BOOLEAN
          Provider name: enableHttpPortAccess
          Description: Optional. If true, enable http access to specific ports on the cluster from external sources. Defaults to false.
      • gce_cluster_config
        Type: STRUCT
        Provider name: gceClusterConfig
        Description: Optional. The shared Compute Engine config settings for all instances in a cluster.
      • gke_cluster_config
        Type: STRUCT
        Provider name: gkeClusterConfig
        Description: Optional. BETA. The Kubernetes Engine config for Dataproc clusters deployed to The Kubernetes Engine config for Dataproc clusters deployed to Kubernetes. These config settings are mutually exclusive with Compute Engine-based options, such as gce_cluster_config, master_config, worker_config, secondary_worker_config, and autoscaling_config.
        • gke_cluster_target
          Type: STRING
          Provider name: gkeClusterTarget
          Description: Optional. A target GKE cluster to deploy to. It must be in the same project and region as the Dataproc cluster (the GKE cluster can be zonal or regional). Format: ‘projects/{project}/locations/{location}/clusters/{cluster_id}’
        • namespaced_gke_deployment_target
          Type: STRUCT
          Provider name: namespacedGkeDeploymentTarget
          Description: Optional. Deprecated. Use gkeClusterTarget. Used only for the deprecated beta. A target for the deployment.
          • cluster_namespace
            Type: STRING
            Provider name: clusterNamespace
            Description: Optional. A namespace within the GKE cluster to deploy into.
          • target_gke_cluster
            Type: STRING
            Provider name: targetGkeCluster
            Description: Optional. The target GKE cluster to deploy to. Format: ‘projects/{project}/locations/{location}/clusters/{cluster_id}’
        • node_pool_target
          Type: UNORDERED_LIST_STRUCT
          Provider name: nodePoolTarget
          Description: Optional. GKE node pools where workloads will be scheduled. At least one node pool must be assigned the DEFAULT GkeNodePoolTarget.Role. If a GkeNodePoolTarget is not specified, Dataproc constructs a DEFAULT GkeNodePoolTarget. Each role can be given to only one GkeNodePoolTarget. All node pools must have the same location settings.
      • initialization_actions
        Type: UNORDERED_LIST_STRUCT
        Provider name: initializationActions
        Description: Optional. Commands to execute on each node after config is completed. By default, executables are run on master and all worker nodes. You can test a node’s role metadata to run an executable on a master or worker node, as shown below using curl (you can also use wget): ROLE=$(curl -H Metadata-Flavor:Google http://metadata/computeMetadata/v1/instance/attributes/dataproc-role) if [[ “${ROLE}” == ‘Master’ ]]; then … master specific actions … else … worker specific actions … fi
      • lifecycle_config
        Type: STRUCT
        Provider name: lifecycleConfig
        Description: Optional. Lifecycle setting for the cluster.
      • master_config
        Type: STRUCT
        Provider name: masterConfig
        Description: Optional. The Compute Engine config settings for the cluster’s master instance.
        • accelerators
          Type: UNORDERED_LIST_STRUCT
          Provider name: accelerators
          Description: Optional. The Compute Engine accelerator configuration for these instances.
        • disk_config
          Type: STRUCT
          Provider name: diskConfig
          Description: Optional. Disk option config settings.
          • boot_disk_provisioned_iops
            Type: INT64
            Provider name: bootDiskProvisionedIops
            Description: Optional. Indicates how many IOPS to provision for the disk. This sets the number of I/O operations per second that the disk can handle. This field is supported only if boot_disk_type is hyperdisk-balanced.
          • boot_disk_provisioned_throughput
            Type: INT64
            Provider name: bootDiskProvisionedThroughput
            Description: Optional. Indicates how much throughput to provision for the disk. This sets the number of throughput mb per second that the disk can handle. Values must be greater than or equal to 1. This field is supported only if boot_disk_type is hyperdisk-balanced.
          • boot_disk_size_gb
            Type: INT32
            Provider name: bootDiskSizeGb
            Description: Optional. Size in GB of the boot disk (default is 500GB).
          • boot_disk_type
            Type: STRING
            Provider name: bootDiskType
            Description: Optional. Type of the boot disk (default is “pd-standard”). Valid values: “pd-balanced” (Persistent Disk Balanced Solid State Drive), “pd-ssd” (Persistent Disk Solid State Drive), or “pd-standard” (Persistent Disk Hard Disk Drive). See Disk types (https://cloud.google.com/compute/docs/disks#disk-types).
          • local_ssd_interface
            Type: STRING
            Provider name: localSsdInterface
            Description: Optional. Interface type of local SSDs (default is “scsi”). Valid values: “scsi” (Small Computer System Interface), “nvme” (Non-Volatile Memory Express). See local SSD performance (https://cloud.google.com/compute/docs/disks/local-ssd#performance).
          • num_local_ssds
            Type: INT32
            Provider name: numLocalSsds
            Description: Optional. Number of attached SSDs, from 0 to 8 (default is 0). If SSDs are not attached, the boot disk is used to store runtime logs and HDFS (https://hadoop.apache.org/docs/r1.2.1/hdfs_user_guide.html) data. If one or more SSDs are attached, this runtime bulk data is spread across them, and the boot disk contains only basic config and installed binaries.Note: Local SSD options may vary by machine type and number of vCPUs selected.
        • image_uri
          Type: STRING
          Provider name: imageUri
          Description: Optional. The Compute Engine image resource used for cluster instances.The URI can represent an image or image family.Image examples: https://www.googleapis.com/compute/v1/projects/[project_id]/global/images/[image-id] projects/[project_id]/global/images/[image-id] image-idImage family examples. Dataproc will use the most recent image from the family: https://www.googleapis.com/compute/v1/projects/[project_id]/global/images/family/[custom-image-family-name] projects/[project_id]/global/images/family/[custom-image-family-name]If the URI is unspecified, it will be inferred from SoftwareConfig.image_version or the system default.
        • instance_flexibility_policy
          Type: STRUCT
          Provider name: instanceFlexibilityPolicy
          Description: Optional. Instance flexibility Policy allowing a mixture of VM shapes and provisioning models.
          • instance_selection_list
            Type: UNORDERED_LIST_STRUCT
            Provider name: instanceSelectionList
            Description: Optional. List of instance selection options that the group will use when creating new VMs.
            • machine_types
              Type: UNORDERED_LIST_STRING
              Provider name: machineTypes
              Description: Optional. Full machine-type names, e.g. “n1-standard-16”.
            • rank
              Type: INT32
              Provider name: rank
              Description: Optional. Preference of this instance selection. Lower number means higher preference. Dataproc will first try to create a VM based on the machine-type with priority rank and fallback to next rank based on availability. Machine types and instance selections with the same priority have the same preference.
          • instance_selection_results
            Type: UNORDERED_LIST_STRUCT
            Provider name: instanceSelectionResults
            Description: Output only. A list of instance selection results in the group.
            • machine_type
              Type: STRING
              Provider name: machineType
              Description: Output only. Full machine-type names, e.g. “n1-standard-16”.
            • vm_count
              Type: INT32
              Provider name: vmCount
              Description: Output only. Number of VM provisioned with the machine_type.
          • provisioning_model_mix
            Type: STRUCT
            Provider name: provisioningModelMix
            Description: Optional. Defines how the Group selects the provisioning model to ensure required reliability.
            • standard_capacity_base
              Type: INT32
              Provider name: standardCapacityBase
              Description: Optional. The base capacity that will always use Standard VMs to avoid risk of more preemption than the minimum capacity you need. Dataproc will create only standard VMs until it reaches standard_capacity_base, then it will start using standard_capacity_percent_above_base to mix Spot with Standard VMs. eg. If 15 instances are requested and standard_capacity_base is 5, Dataproc will create 5 standard VMs and then start mixing spot and standard VMs for remaining 10 instances.
            • standard_capacity_percent_above_base
              Type: INT32
              Provider name: standardCapacityPercentAboveBase
              Description: Optional. The percentage of target capacity that should use Standard VM. The remaining percentage will use Spot VMs. The percentage applies only to the capacity above standard_capacity_base. eg. If 15 instances are requested and standard_capacity_base is 5 and standard_capacity_percent_above_base is 30, Dataproc will create 5 standard VMs and then start mixing spot and standard VMs for remaining 10 instances. The mix will be 30% standard and 70% spot.
        • instance_names
          Type: UNORDERED_LIST_STRING
          Provider name: instanceNames
          Description: Output only. The list of instance names. Dataproc derives the names from cluster_name, num_instances, and the instance group.
        • instance_references
          Type: UNORDERED_LIST_STRUCT
          Provider name: instanceReferences
          Description: Output only. List of references to Compute Engine instances.
          • instance_id
            Type: STRING
            Provider name: instanceId
            Description: The unique identifier of the Compute Engine instance.
          • instance_name
            Type: STRING
            Provider name: instanceName
            Description: The user-friendly name of the Compute Engine instance.
          • public_ecies_key
            Type: STRING
            Provider name: publicEciesKey
            Description: The public ECIES key used for sharing data with this instance.
          • public_key
            Type: STRING
            Provider name: publicKey
            Description: The public RSA key used for sharing data with this instance.
        • is_preemptible
          Type: BOOLEAN
          Provider name: isPreemptible
          Description: Output only. Specifies that this instance group contains preemptible instances.
        • machine_type_uri
          Type: STRING
          Provider name: machineTypeUri
          Description: Optional. The Compute Engine machine type used for cluster instances.A full URL, partial URI, or short name are valid. Examples: https://www.googleapis.com/compute/v1/projects/[project_id]/zones/[zone]/machineTypes/n1-standard-2 projects/[project_id]/zones/[zone]/machineTypes/n1-standard-2 n1-standard-2Auto Zone Exception: If you are using the Dataproc Auto Zone Placement (https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/auto-zone#using_auto_zone_placement) feature, you must use the short name of the machine type resource, for example, n1-standard-2.
        • managed_group_config
          Type: STRUCT
          Provider name: managedGroupConfig
          Description: Output only. The config for Compute Engine Instance Group Manager that manages this group. This is only used for preemptible instance groups.
          • instance_group_manager_name
            Type: STRING
            Provider name: instanceGroupManagerName
            Description: Output only. The name of the Instance Group Manager for this group.
          • instance_group_manager_uri
            Type: STRING
            Provider name: instanceGroupManagerUri
            Description: Output only. The partial URI to the instance group manager for this group. E.g. projects/my-project/regions/us-central1/instanceGroupManagers/my-igm.
          • instance_template_name
            Type: STRING
            Provider name: instanceTemplateName
            Description: Output only. The name of the Instance Template used for the Managed Instance Group.
        • min_cpu_platform
          Type: STRING
          Provider name: minCpuPlatform
          Description: Optional. Specifies the minimum cpu platform for the Instance Group. See Dataproc -> Minimum CPU Platform (https://cloud.google.com/dataproc/docs/concepts/compute/dataproc-min-cpu).
        • min_num_instances
          Type: INT32
          Provider name: minNumInstances
          Description: Optional. The minimum number of primary worker instances to create. If min_num_instances is set, cluster creation will succeed if the number of primary workers created is at least equal to the min_num_instances number.Example: Cluster creation request with num_instances = 5 and min_num_instances = 3: If 4 VMs are created and 1 instance fails, the failed VM is deleted. The cluster is resized to 4 instances and placed in a RUNNING state. If 2 instances are created and 3 instances fail, the cluster in placed in an ERROR state. The failed VMs are not deleted.
        • num_instances
          Type: INT32
          Provider name: numInstances
          Description: Optional. The number of VM instances in the instance group. For HA cluster master_config groups, must be set to 3. For standard cluster master_config groups, must be set to 1.
        • preemptibility
          Type: STRING
          Provider name: preemptibility
          Description: Optional. Specifies the preemptibility of the instance group.The default value for master and worker groups is NON_PREEMPTIBLE. This default cannot be changed.The default value for secondary instances is PREEMPTIBLE.
          Possible values:
        • startup_config
          Type: STRUCT
          Provider name: startupConfig
          Description: Optional. Configuration to handle the startup of instances during cluster create and update process.
          • required_registration_fraction
            Type: DOUBLE
            Provider name: requiredRegistrationFraction
            Description: Optional. The config setting to enable cluster creation/ updation to be successful only after required_registration_fraction of instances are up and running. This configuration is applicable to only secondary workers for now. The cluster will fail if required_registration_fraction of instances are not available. This will include instance creation, agent registration, and service registration (if enabled).
      • metastore_config
        Type: STRUCT
        Provider name: metastoreConfig
        Description: Optional. Metastore configuration.
        • dataproc_metastore_service
          Type: STRING
          Provider name: dataprocMetastoreService
          Description: Required. Resource name of an existing Dataproc Metastore service.Example: projects/[project_id]/locations/[dataproc_region]/services/[service-name]
      • secondary_worker_config
        Type: STRUCT
        Provider name: secondaryWorkerConfig
        Description: Optional. The Compute Engine config settings for a cluster’s secondary worker instances
        • accelerators
          Type: UNORDERED_LIST_STRUCT
          Provider name: accelerators
          Description: Optional. The Compute Engine accelerator configuration for these instances.
        • disk_config
          Type: STRUCT
          Provider name: diskConfig
          Description: Optional. Disk option config settings.
          • boot_disk_provisioned_iops
            Type: INT64
            Provider name: bootDiskProvisionedIops
            Description: Optional. Indicates how many IOPS to provision for the disk. This sets the number of I/O operations per second that the disk can handle. This field is supported only if boot_disk_type is hyperdisk-balanced.
          • boot_disk_provisioned_throughput
            Type: INT64
            Provider name: bootDiskProvisionedThroughput
            Description: Optional. Indicates how much throughput to provision for the disk. This sets the number of throughput mb per second that the disk can handle. Values must be greater than or equal to 1. This field is supported only if boot_disk_type is hyperdisk-balanced.
          • boot_disk_size_gb
            Type: INT32
            Provider name: bootDiskSizeGb
            Description: Optional. Size in GB of the boot disk (default is 500GB).
          • boot_disk_type
            Type: STRING
            Provider name: bootDiskType
            Description: Optional. Type of the boot disk (default is “pd-standard”). Valid values: “pd-balanced” (Persistent Disk Balanced Solid State Drive), “pd-ssd” (Persistent Disk Solid State Drive), or “pd-standard” (Persistent Disk Hard Disk Drive). See Disk types (https://cloud.google.com/compute/docs/disks#disk-types).
          • local_ssd_interface
            Type: STRING
            Provider name: localSsdInterface
            Description: Optional. Interface type of local SSDs (default is “scsi”). Valid values: “scsi” (Small Computer System Interface), “nvme” (Non-Volatile Memory Express). See local SSD performance (https://cloud.google.com/compute/docs/disks/local-ssd#performance).
          • num_local_ssds
            Type: INT32
            Provider name: numLocalSsds
            Description: Optional. Number of attached SSDs, from 0 to 8 (default is 0). If SSDs are not attached, the boot disk is used to store runtime logs and HDFS (https://hadoop.apache.org/docs/r1.2.1/hdfs_user_guide.html) data. If one or more SSDs are attached, this runtime bulk data is spread across them, and the boot disk contains only basic config and installed binaries.Note: Local SSD options may vary by machine type and number of vCPUs selected.
        • image_uri
          Type: STRING
          Provider name: imageUri
          Description: Optional. The Compute Engine image resource used for cluster instances.The URI can represent an image or image family.Image examples: https://www.googleapis.com/compute/v1/projects/[project_id]/global/images/[image-id] projects/[project_id]/global/images/[image-id] image-idImage family examples. Dataproc will use the most recent image from the family: https://www.googleapis.com/compute/v1/projects/[project_id]/global/images/family/[custom-image-family-name] projects/[project_id]/global/images/family/[custom-image-family-name]If the URI is unspecified, it will be inferred from SoftwareConfig.image_version or the system default.
        • instance_flexibility_policy
          Type: STRUCT
          Provider name: instanceFlexibilityPolicy
          Description: Optional. Instance flexibility Policy allowing a mixture of VM shapes and provisioning models.
          • instance_selection_list
            Type: UNORDERED_LIST_STRUCT
            Provider name: instanceSelectionList
            Description: Optional. List of instance selection options that the group will use when creating new VMs.
            • machine_types
              Type: UNORDERED_LIST_STRING
              Provider name: machineTypes
              Description: Optional. Full machine-type names, e.g. “n1-standard-16”.
            • rank
              Type: INT32
              Provider name: rank
              Description: Optional. Preference of this instance selection. Lower number means higher preference. Dataproc will first try to create a VM based on the machine-type with priority rank and fallback to next rank based on availability. Machine types and instance selections with the same priority have the same preference.
          • instance_selection_results
            Type: UNORDERED_LIST_STRUCT
            Provider name: instanceSelectionResults
            Description: Output only. A list of instance selection results in the group.
            • machine_type
              Type: STRING
              Provider name: machineType
              Description: Output only. Full machine-type names, e.g. “n1-standard-16”.
            • vm_count
              Type: INT32
              Provider name: vmCount
              Description: Output only. Number of VM provisioned with the machine_type.
          • provisioning_model_mix
            Type: STRUCT
            Provider name: provisioningModelMix
            Description: Optional. Defines how the Group selects the provisioning model to ensure required reliability.
            • standard_capacity_base
              Type: INT32
              Provider name: standardCapacityBase
              Description: Optional. The base capacity that will always use Standard VMs to avoid risk of more preemption than the minimum capacity you need. Dataproc will create only standard VMs until it reaches standard_capacity_base, then it will start using standard_capacity_percent_above_base to mix Spot with Standard VMs. eg. If 15 instances are requested and standard_capacity_base is 5, Dataproc will create 5 standard VMs and then start mixing spot and standard VMs for remaining 10 instances.
            • standard_capacity_percent_above_base
              Type: INT32
              Provider name: standardCapacityPercentAboveBase
              Description: Optional. The percentage of target capacity that should use Standard VM. The remaining percentage will use Spot VMs. The percentage applies only to the capacity above standard_capacity_base. eg. If 15 instances are requested and standard_capacity_base is 5 and standard_capacity_percent_above_base is 30, Dataproc will create 5 standard VMs and then start mixing spot and standard VMs for remaining 10 instances. The mix will be 30% standard and 70% spot.
        • instance_names
          Type: UNORDERED_LIST_STRING
          Provider name: instanceNames
          Description: Output only. The list of instance names. Dataproc derives the names from cluster_name, num_instances, and the instance group.
        • instance_references
          Type: UNORDERED_LIST_STRUCT
          Provider name: instanceReferences
          Description: Output only. List of references to Compute Engine instances.
          • instance_id
            Type: STRING
            Provider name: instanceId
            Description: The unique identifier of the Compute Engine instance.
          • instance_name
            Type: STRING
            Provider name: instanceName
            Description: The user-friendly name of the Compute Engine instance.
          • public_ecies_key
            Type: STRING
            Provider name: publicEciesKey
            Description: The public ECIES key used for sharing data with this instance.
          • public_key
            Type: STRING
            Provider name: publicKey
            Description: The public RSA key used for sharing data with this instance.
        • is_preemptible
          Type: BOOLEAN
          Provider name: isPreemptible
          Description: Output only. Specifies that this instance group contains preemptible instances.
        • machine_type_uri
          Type: STRING
          Provider name: machineTypeUri
          Description: Optional. The Compute Engine machine type used for cluster instances.A full URL, partial URI, or short name are valid. Examples: https://www.googleapis.com/compute/v1/projects/[project_id]/zones/[zone]/machineTypes/n1-standard-2 projects/[project_id]/zones/[zone]/machineTypes/n1-standard-2 n1-standard-2Auto Zone Exception: If you are using the Dataproc Auto Zone Placement (https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/auto-zone#using_auto_zone_placement) feature, you must use the short name of the machine type resource, for example, n1-standard-2.
        • managed_group_config
          Type: STRUCT
          Provider name: managedGroupConfig
          Description: Output only. The config for Compute Engine Instance Group Manager that manages this group. This is only used for preemptible instance groups.
          • instance_group_manager_name
            Type: STRING
            Provider name: instanceGroupManagerName
            Description: Output only. The name of the Instance Group Manager for this group.
          • instance_group_manager_uri
            Type: STRING
            Provider name: instanceGroupManagerUri
            Description: Output only. The partial URI to the instance group manager for this group. E.g. projects/my-project/regions/us-central1/instanceGroupManagers/my-igm.
          • instance_template_name
            Type: STRING
            Provider name: instanceTemplateName
            Description: Output only. The name of the Instance Template used for the Managed Instance Group.
        • min_cpu_platform
          Type: STRING
          Provider name: minCpuPlatform
          Description: Optional. Specifies the minimum cpu platform for the Instance Group. See Dataproc -> Minimum CPU Platform (https://cloud.google.com/dataproc/docs/concepts/compute/dataproc-min-cpu).
        • min_num_instances
          Type: INT32
          Provider name: minNumInstances
          Description: Optional. The minimum number of primary worker instances to create. If min_num_instances is set, cluster creation will succeed if the number of primary workers created is at least equal to the min_num_instances number.Example: Cluster creation request with num_instances = 5 and min_num_instances = 3: If 4 VMs are created and 1 instance fails, the failed VM is deleted. The cluster is resized to 4 instances and placed in a RUNNING state. If 2 instances are created and 3 instances fail, the cluster in placed in an ERROR state. The failed VMs are not deleted.
        • num_instances
          Type: INT32
          Provider name: numInstances
          Description: Optional. The number of VM instances in the instance group. For HA cluster master_config groups, must be set to 3. For standard cluster master_config groups, must be set to 1.
        • preemptibility
          Type: STRING
          Provider name: preemptibility
          Description: Optional. Specifies the preemptibility of the instance group.The default value for master and worker groups is NON_PREEMPTIBLE. This default cannot be changed.The default value for secondary instances is PREEMPTIBLE.
          Possible values:
        • startup_config
          Type: STRUCT
          Provider name: startupConfig
          Description: Optional. Configuration to handle the startup of instances during cluster create and update process.
          • required_registration_fraction
            Type: DOUBLE
            Provider name: requiredRegistrationFraction
            Description: Optional. The config setting to enable cluster creation/ updation to be successful only after required_registration_fraction of instances are up and running. This configuration is applicable to only secondary workers for now. The cluster will fail if required_registration_fraction of instances are not available. This will include instance creation, agent registration, and service registration (if enabled).
      • security_config
        Type: STRUCT
        Provider name: securityConfig
        Description: Optional. Security settings for the cluster.

        • identity_config
          Type: STRUCT
          Provider name: identityConfig
          Description: Optional. Identity related configuration, including service account based secure multi-tenancy user mappings.

        • kerberos_config
          Type: STRUCT
          Provider name: kerberosConfig
          Description: Optional. Kerberos related configuration.

          • cross_realm_trust_admin_server
            Type: STRING
            Provider name: crossRealmTrustAdminServer
            Description: Optional. The admin server (IP or hostname) for the remote trusted realm in a cross realm trust relationship.
          • cross_realm_trust_kdc
            Type: STRING
            Provider name: crossRealmTrustKdc
            Description: Optional. The KDC (IP or hostname) for the remote trusted realm in a cross realm trust relationship.
          • cross_realm_trust_realm
            Type: STRING
            Provider name: crossRealmTrustRealm
            Description: Optional. The remote realm the Dataproc on-cluster KDC will trust, should the user enable cross realm trust.
          • cross_realm_trust_shared_password_uri
            Type: STRING
            Provider name: crossRealmTrustSharedPasswordUri
            Description: Optional. The Cloud Storage URI of a KMS encrypted file containing the shared password between the on-cluster Kerberos realm and the remote trusted realm, in a cross realm trust relationship.
          • enable_kerberos
            Type: BOOLEAN
            Provider name: enableKerberos
            Description: Optional. Flag to indicate whether to Kerberize the cluster (default: false). Set this field to true to enable Kerberos on a cluster.
          • kdc_db_key_uri
            Type: STRING
            Provider name: kdcDbKeyUri
            Description: Optional. The Cloud Storage URI of a KMS encrypted file containing the master key of the KDC database.
          • key_password_uri
            Type: STRING
            Provider name: keyPasswordUri
            Description: Optional. The Cloud Storage URI of a KMS encrypted file containing the password to the user provided key. For the self-signed certificate, this password is generated by Dataproc.
          • keystore_password_uri
            Type: STRING
            Provider name: keystorePasswordUri
            Description: Optional. The Cloud Storage URI of a KMS encrypted file containing the password to the user provided keystore. For the self-signed certificate, this password is generated by Dataproc.
          • keystore_uri
            Type: STRING
            Provider name: keystoreUri
            Description: Optional. The Cloud Storage URI of the keystore file used for SSL encryption. If not provided, Dataproc will provide a self-signed certificate.
          • kms_key_uri
            Type: STRING
            Provider name: kmsKeyUri
            Description: Optional. The URI of the KMS key used to encrypt sensitive files.
          • realm
            Type: STRING
            Provider name: realm
            Description: Optional. The name of the on-cluster Kerberos realm. If not specified, the uppercased domain of hostnames will be the realm.
          • root_principal_password_uri
            Type: STRING
            Provider name: rootPrincipalPasswordUri
            Description: Optional. The Cloud Storage URI of a KMS encrypted file containing the root principal password.
          • tgt_lifetime_hours
            Type: INT32
            Provider name: tgtLifetimeHours
            Description: Optional. The lifetime of the ticket granting ticket, in hours. If not specified, or user specifies 0, then default value 10 will be used.
          • truststore_password_uri
            Type: STRING
            Provider name: truststorePasswordUri
            Description: Optional. The Cloud Storage URI of a KMS encrypted file containing the password to the user provided truststore. For the self-signed certificate, this password is generated by Dataproc.
          • truststore_uri
            Type: STRING
            Provider name: truststoreUri
            Description: Optional. The Cloud Storage URI of the truststore file used for SSL encryption. If not provided, Dataproc will provide a self-signed certificate.
      • software_config
        Type: STRUCT
        Provider name: softwareConfig
        Description: Optional. The config settings for cluster software.
      • temp_bucket
        Type: STRING
        Provider name: tempBucket
        Description: Optional. A Cloud Storage bucket used to store ephemeral cluster and jobs data, such as Spark and MapReduce history files. If you do not specify a temp bucket, Dataproc will determine a Cloud Storage location (US, ASIA, or EU) for your cluster’s temp bucket according to the Compute Engine zone where your cluster is deployed, and then create and manage this project-level, per-location bucket. The default bucket has a TTL of 90 days, but you can use any TTL (or none) if you specify a bucket (see Dataproc staging and temp buckets (https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/staging-bucket)). This field requires a Cloud Storage bucket name, not a gs://… URI to a Cloud Storage bucket.
      • worker_config
        Type: STRUCT
        Provider name: workerConfig
        Description: Optional. The Compute Engine config settings for the cluster’s worker instances.
        • accelerators
          Type: UNORDERED_LIST_STRUCT
          Provider name: accelerators
          Description: Optional. The Compute Engine accelerator configuration for these instances.
        • disk_config
          Type: STRUCT
          Provider name: diskConfig
          Description: Optional. Disk option config settings.
          • boot_disk_provisioned_iops
            Type: INT64
            Provider name: bootDiskProvisionedIops
            Description: Optional. Indicates how many IOPS to provision for the disk. This sets the number of I/O operations per second that the disk can handle. This field is supported only if boot_disk_type is hyperdisk-balanced.
          • boot_disk_provisioned_throughput
            Type: INT64
            Provider name: bootDiskProvisionedThroughput
            Description: Optional. Indicates how much throughput to provision for the disk. This sets the number of throughput mb per second that the disk can handle. Values must be greater than or equal to 1. This field is supported only if boot_disk_type is hyperdisk-balanced.
          • boot_disk_size_gb
            Type: INT32
            Provider name: bootDiskSizeGb
            Description: Optional. Size in GB of the boot disk (default is 500GB).
          • boot_disk_type
            Type: STRING
            Provider name: bootDiskType
            Description: Optional. Type of the boot disk (default is “pd-standard”). Valid values: “pd-balanced” (Persistent Disk Balanced Solid State Drive), “pd-ssd” (Persistent Disk Solid State Drive), or “pd-standard” (Persistent Disk Hard Disk Drive). See Disk types (https://cloud.google.com/compute/docs/disks#disk-types).
          • local_ssd_interface
            Type: STRING
            Provider name: localSsdInterface
            Description: Optional. Interface type of local SSDs (default is “scsi”). Valid values: “scsi” (Small Computer System Interface), “nvme” (Non-Volatile Memory Express). See local SSD performance (https://cloud.google.com/compute/docs/disks/local-ssd#performance).
          • num_local_ssds
            Type: INT32
            Provider name: numLocalSsds
            Description: Optional. Number of attached SSDs, from 0 to 8 (default is 0). If SSDs are not attached, the boot disk is used to store runtime logs and HDFS (https://hadoop.apache.org/docs/r1.2.1/hdfs_user_guide.html) data. If one or more SSDs are attached, this runtime bulk data is spread across them, and the boot disk contains only basic config and installed binaries.Note: Local SSD options may vary by machine type and number of vCPUs selected.
        • image_uri
          Type: STRING
          Provider name: imageUri
          Description: Optional. The Compute Engine image resource used for cluster instances.The URI can represent an image or image family.Image examples: https://www.googleapis.com/compute/v1/projects/[project_id]/global/images/[image-id] projects/[project_id]/global/images/[image-id] image-idImage family examples. Dataproc will use the most recent image from the family: https://www.googleapis.com/compute/v1/projects/[project_id]/global/images/family/[custom-image-family-name] projects/[project_id]/global/images/family/[custom-image-family-name]If the URI is unspecified, it will be inferred from SoftwareConfig.image_version or the system default.
        • instance_flexibility_policy
          Type: STRUCT
          Provider name: instanceFlexibilityPolicy
          Description: Optional. Instance flexibility Policy allowing a mixture of VM shapes and provisioning models.
          • instance_selection_list
            Type: UNORDERED_LIST_STRUCT
            Provider name: instanceSelectionList
            Description: Optional. List of instance selection options that the group will use when creating new VMs.
            • machine_types
              Type: UNORDERED_LIST_STRING
              Provider name: machineTypes
              Description: Optional. Full machine-type names, e.g. “n1-standard-16”.
            • rank
              Type: INT32
              Provider name: rank
              Description: Optional. Preference of this instance selection. Lower number means higher preference. Dataproc will first try to create a VM based on the machine-type with priority rank and fallback to next rank based on availability. Machine types and instance selections with the same priority have the same preference.
          • instance_selection_results
            Type: UNORDERED_LIST_STRUCT
            Provider name: instanceSelectionResults
            Description: Output only. A list of instance selection results in the group.
            • machine_type
              Type: STRING
              Provider name: machineType
              Description: Output only. Full machine-type names, e.g. “n1-standard-16”.
            • vm_count
              Type: INT32
              Provider name: vmCount
              Description: Output only. Number of VM provisioned with the machine_type.
          • provisioning_model_mix
            Type: STRUCT
            Provider name: provisioningModelMix
            Description: Optional. Defines how the Group selects the provisioning model to ensure required reliability.
            • standard_capacity_base
              Type: INT32
              Provider name: standardCapacityBase
              Description: Optional. The base capacity that will always use Standard VMs to avoid risk of more preemption than the minimum capacity you need. Dataproc will create only standard VMs until it reaches standard_capacity_base, then it will start using standard_capacity_percent_above_base to mix Spot with Standard VMs. eg. If 15 instances are requested and standard_capacity_base is 5, Dataproc will create 5 standard VMs and then start mixing spot and standard VMs for remaining 10 instances.
            • standard_capacity_percent_above_base
              Type: INT32
              Provider name: standardCapacityPercentAboveBase
              Description: Optional. The percentage of target capacity that should use Standard VM. The remaining percentage will use Spot VMs. The percentage applies only to the capacity above standard_capacity_base. eg. If 15 instances are requested and standard_capacity_base is 5 and standard_capacity_percent_above_base is 30, Dataproc will create 5 standard VMs and then start mixing spot and standard VMs for remaining 10 instances. The mix will be 30% standard and 70% spot.
        • instance_names
          Type: UNORDERED_LIST_STRING
          Provider name: instanceNames
          Description: Output only. The list of instance names. Dataproc derives the names from cluster_name, num_instances, and the instance group.
        • instance_references
          Type: UNORDERED_LIST_STRUCT
          Provider name: instanceReferences
          Description: Output only. List of references to Compute Engine instances.
          • instance_id
            Type: STRING
            Provider name: instanceId
            Description: The unique identifier of the Compute Engine instance.
          • instance_name
            Type: STRING
            Provider name: instanceName
            Description: The user-friendly name of the Compute Engine instance.
          • public_ecies_key
            Type: STRING
            Provider name: publicEciesKey
            Description: The public ECIES key used for sharing data with this instance.
          • public_key
            Type: STRING
            Provider name: publicKey
            Description: The public RSA key used for sharing data with this instance.
        • is_preemptible
          Type: BOOLEAN
          Provider name: isPreemptible
          Description: Output only. Specifies that this instance group contains preemptible instances.
        • machine_type_uri
          Type: STRING
          Provider name: machineTypeUri
          Description: Optional. The Compute Engine machine type used for cluster instances.A full URL, partial URI, or short name are valid. Examples: https://www.googleapis.com/compute/v1/projects/[project_id]/zones/[zone]/machineTypes/n1-standard-2 projects/[project_id]/zones/[zone]/machineTypes/n1-standard-2 n1-standard-2Auto Zone Exception: If you are using the Dataproc Auto Zone Placement (https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/auto-zone#using_auto_zone_placement) feature, you must use the short name of the machine type resource, for example, n1-standard-2.
        • managed_group_config
          Type: STRUCT
          Provider name: managedGroupConfig
          Description: Output only. The config for Compute Engine Instance Group Manager that manages this group. This is only used for preemptible instance groups.
          • instance_group_manager_name
            Type: STRING
            Provider name: instanceGroupManagerName
            Description: Output only. The name of the Instance Group Manager for this group.
          • instance_group_manager_uri
            Type: STRING
            Provider name: instanceGroupManagerUri
            Description: Output only. The partial URI to the instance group manager for this group. E.g. projects/my-project/regions/us-central1/instanceGroupManagers/my-igm.
          • instance_template_name
            Type: STRING
            Provider name: instanceTemplateName
            Description: Output only. The name of the Instance Template used for the Managed Instance Group.
        • min_cpu_platform
          Type: STRING
          Provider name: minCpuPlatform
          Description: Optional. Specifies the minimum cpu platform for the Instance Group. See Dataproc -> Minimum CPU Platform (https://cloud.google.com/dataproc/docs/concepts/compute/dataproc-min-cpu).
        • min_num_instances
          Type: INT32
          Provider name: minNumInstances
          Description: Optional. The minimum number of primary worker instances to create. If min_num_instances is set, cluster creation will succeed if the number of primary workers created is at least equal to the min_num_instances number.Example: Cluster creation request with num_instances = 5 and min_num_instances = 3: If 4 VMs are created and 1 instance fails, the failed VM is deleted. The cluster is resized to 4 instances and placed in a RUNNING state. If 2 instances are created and 3 instances fail, the cluster in placed in an ERROR state. The failed VMs are not deleted.
        • num_instances
          Type: INT32
          Provider name: numInstances
          Description: Optional. The number of VM instances in the instance group. For HA cluster master_config groups, must be set to 3. For standard cluster master_config groups, must be set to 1.
        • preemptibility
          Type: STRING
          Provider name: preemptibility
          Description: Optional. Specifies the preemptibility of the instance group.The default value for master and worker groups is NON_PREEMPTIBLE. This default cannot be changed.The default value for secondary instances is PREEMPTIBLE.
          Possible values:
        • startup_config
          Type: STRUCT
          Provider name: startupConfig
          Description: Optional. Configuration to handle the startup of instances during cluster create and update process.
          • required_registration_fraction
            Type: DOUBLE
            Provider name: requiredRegistrationFraction
            Description: Optional. The config setting to enable cluster creation/ updation to be successful only after required_registration_fraction of instances are up and running. This configuration is applicable to only secondary workers for now. The cluster will fail if required_registration_fraction of instances are not available. This will include instance creation, agent registration, and service registration (if enabled).

project_id

Type: STRING

project_number

Type: STRING

region_id

Type: STRING

resource_name

Type: STRING

tags

Type: UNORDERED_LIST_STRING

update_time

Type: TIMESTAMP
Provider name: updateTime
Description: Output only. The time template was last updated.

version

Type: INT32
Provider name: version
Description: Optional. Used to perform a consistent read-modify-write.This field should be left blank for a CreateWorkflowTemplate request. It is required for an UpdateWorkflowTemplate request, and must match the current server version. A typical update template flow would fetch the current template with a GetWorkflowTemplate request, which will return the current template with the version field filled in with the current server version. The user updates other fields in the template, then returns it as part of the UpdateWorkflowTemplate request.

zone_id

Type: STRING