S3Tables / Client / start_table_migration

start_table_migration¶

S3Tables.Client.start_table_migration(**kwargs)¶

Starts migrating an Amazon Web Services Glue Data Catalog table into a new table in a table bucket. The migration runs asynchronously: this operation returns the migration ID and its initial state, and you track progress with GetTableMigration.

The namespace must already exist, and a table with the specified name must not. The migration creates the table during planning and commits data to it as it runs. Until you call CompleteTableMigration, you can read the table but you can’t write to it. If the migration fails or is cancelled, the table and the data already committed to it remain.

The migration first checks the source and the destination, and writes the table compatibility and object compatibility reports to the bucket that you specify in migrationOutput. If you set reviewMigrationPlan to true, the migration waits in the pendingReview state so that you can review the reports, and continues only when you call UpdateTableMigration with requestedState set to queued. If you don’t continue or cancel the migration within 30 days, the migration fails. Otherwise it continues to migrate the data, and writes the object migration report when the migration ends. If planning fails, the migration writes no object migration report.

Permissions

You must have the s3tables:StartTableMigration permission to use this operation.
You must have the iam:PassRole permission on the role that you specify in roleARN, with a condition that allows passing it to migration.s3tables.amazonaws.com. The associated resource is the table bucket.
If destinationTable specifies SSE-KMS encryption, you must have the kms:DescribeKey and kms:GenerateDataKey permissions on the KMS key. You must also grant the S3 Tables maintenance principal access to the key. For more information, see Permissions requirements for S3 Tables SSE-KMS encryption.

Request Syntax

response = client.start_table_migration(
    roleARN='string',
    destinationTable={
        'tableBucketARN': 'string',
        'namespace': 'string',
        'name': 'string',
        'encryptionConfiguration': {
            'sseAlgorithm': 'AES256'|'aws:kms',
            'kmsKeyArn': 'string'
        },
        'storageClassConfiguration': {
            'storageClass': 'STANDARD'|'INTELLIGENT_TIERING'
        },
        'tags': {
            'string': 'string'
        },
        'properties': {
            'writeFormat': 'parquet',
            'icebergFormatVersion': 123,
            'compressionCodec': 'zstd'|'snappy'|'gzip'|'uncompressed',
            'dictionaryEncoding': True|False
        },
        'schema': {
            'type': 'struct',
            'fields': [
                {
                    'id': 123,
                    'name': 'string',
                    'type': {...}|[...]|123|123.4|'string'|True|None,
                    'required': True|False,
                    'doc': 'string'
                },
            ],
            'schemaId': 123,
            'identifierFieldIds': [
                123,
            ]
        },
        'icebergSortOrder': {
            'orderId': 123,
            'fields': [
                {
                    'sourceId': 123,
                    'transform': 'string',
                    'direction': 'asc'|'desc',
                    'nullOrder': 'nulls-first'|'nulls-last'
                },
            ]
        }
    },
    sourceType='glueTable',
    source={
        'glueTable': {
            'catalogId': 'string',
            'databaseName': 'string',
            'tableName': 'string'
        }
    },
    dataMigration={
        'maintenanceSettings': {
            'icebergCompaction': {
                'targetFileSizeMB': 123,
                'strategy': 'auto'|'binpack'|'sort'|'z-order'
            },
            'icebergSnapshotManagement': {
                'minSnapshotsToKeep': 123,
                'maxSnapshotAgeHours': 123
            }
        },
        'migrationPolicies': {
            'unmatchedSourceFieldsPolicy': 'fail'|'drop',
            'incompatibleValuesPolicy': 'fail'|'setNull'
        }
    },
    reviewMigrationPlan=True|False,
    migrationOutput={
        'bucket': 'string',
        'prefixes': {
            'tableCompatibilityReport': 'string',
            'objectCompatibilityReport': 'string',
            'objectMigrationReport': 'string'
        }
    }
)

Parameters:

roleARN (string) –
[REQUIRED]

The Amazon Resource Name (ARN) of the IAM role that the migration uses to read the source table and its data and to write the destination table and the reports. The role must be in your account, and its trust policy must allow migration.s3tables.amazonaws.com to assume it. You can limit the trust policy to the destination table bucket with the aws:SourceArn and aws:SourceAccount condition keys.
destinationTable (dict) –
[REQUIRED]

The table to create and migrate the source into.
- tableBucketARN (string) – [REQUIRED]
  
  The Amazon Resource Name (ARN) of the table bucket to create the table in.
- namespace (string) – [REQUIRED]
  
  The namespace to create the table in.
- name (string) – [REQUIRED]
  
  The name of the table to create.
- encryptionConfiguration (dict) –
  
  The encryption configuration for the table. If you don’t specify one, the table uses the default encryption configuration of its table bucket. If you specify an encryption configuration, the migration role must have the s3tables:PutTableEncryption permission. If you choose SSE-KMS, the migration role must have the kms:DescribeKey, kms:GenerateDataKey and kms:Decrypt permissions on the KMS key, and the S3 Tables maintenance principal must have access to it. For more information, see Permissions requirements for S3 Tables SSE-KMS encryption.
  - sseAlgorithm (string) – [REQUIRED]
    
    The server-side encryption algorithm to use. Valid values are AES256 for S3-managed encryption keys, or aws:kms for Amazon Web Services KMS-managed encryption keys. If you choose SSE-KMS encryption you must grant the S3 Tables maintenance principal access to your KMS key. For more information, see Permissions requirements for S3 Tables SSE-KMS encryption.
  - kmsKeyArn (string) –
    
    The Amazon Resource Name (ARN) of the KMS key to use for encryption. This field is required only when sseAlgorithm is set to aws:kms.
- storageClassConfiguration (dict) –
  
  The storage class configuration for the table. If you don’t specify one, the table inherits the storage class configuration of its table bucket. If you specify a storage class configuration, the migration role must have the s3tables:PutTableStorageClass permission.
  - storageClass (string) – [REQUIRED]
    
    The storage class for the table or table bucket. Valid values include storage classes optimized for different access patterns and cost profiles.
- tags (dict) –
  
  The tags to add to the table. A tag is a key-value pair that you apply to your resources. For more information, see Tagging for cost allocation or attribute-based access control (ABAC). If you specify tags, the migration role must have the s3tables:TagResource permission.
  - (string) –
    - (string) –
- properties (dict) –
  
  The properties of the destination table, such as its Iceberg format version and the format of its data files.
  - writeFormat (string) –
    
    The file format of the data files. The only supported format is parquet, the default.
  - icebergFormatVersion (integer) –
    
    The Iceberg format version of the destination table. Valid values are 2 and 3. The default is 2.
  - compressionCodec (string) –
    
    The compression codec of the data files. The default is zstd.
  - dictionaryEncoding (boolean) –
    
    Specifies whether the data files use dictionary encoding. The default is true.
- schema (dict) –
  
  The schema of the table to create. If you don’t specify a schema, the migration infers it from the source table, with every column optional. A schema that you specify must include every partition column of the source table. It can leave out other columns only if unmatchedSourceFieldsPolicy is drop. A struct column must have the same fields as the source column. Column default values aren’t supported. If you mark a column as required, every source object must have a value for it: an object with a null value in a required column fails to migrate. The destination table is partitioned in the same way as the source table. To specify fields in icebergSortOrder, you must specify a schema.
  - type (string) – [REQUIRED]
    
    The type of the top-level schema, which is always a struct type as defined in the Apache Iceberg specification. This value must be struct.
  - fields (list) – [REQUIRED]
    
    The schema fields for the table. Each field defines a column in the table, including its name, type, and whether it is required.
    - (dict) –
      
      Contains details about a schema field in the V2 format. This field format supports nested and complex data types such as struct, list, and map, in addition to primitive types.
      - id (integer) – [REQUIRED]
        
        The unique identifier for the schema field. Field IDs are used by Apache Iceberg to track schema evolution and maintain compatibility across schema changes.
      - name (string) – [REQUIRED]
        
        The name of the field.
      - type (document) – [REQUIRED]
        
        The data type of the field. This can be a primitive type string such as boolean, int, long, float, double, string, binary, date, timestamp, or timestamptz, or a complex type represented as a JSON object for nested types such as struct, list, or map. For more information, see the Apache Iceberg schemas and data types documentation.
      - required (boolean) – [REQUIRED]
        
        A Boolean value that specifies whether values are required for each row in this field. If this is true, the field does not allow null values.
      - doc (string) –
        
        An optional description of the field.
  - schemaId (integer) –
    
    An optional unique identifier for the schema. Schema IDs are used by Apache Iceberg to track schema evolution.
  - identifierFieldIds (list) –
    
    A list of field IDs that are used as the identifier fields for the table. Identifier fields uniquely identify a row in the table.
    - (integer) –
- icebergSortOrder (dict) –
  
  The sort order of the rows within each data file that the migration writes. A sort order can have up to five fields, each naming a column of the schema that you specify by its field ID, so to use a sort order you must specify a schema. StartTableMigration rejects a field that names no column of the schema, or whose transform doesn’t fit the column’s type.
  - orderId (integer) – [REQUIRED]
    
    The unique identifier for this sort order. If not specified, defaults to 1. The order ID is used by Apache Iceberg to track sort order evolution.
  - fields (list) – [REQUIRED]
    
    The list of sort fields that define how data is sorted within files. Each field specifies a source field, sort direction, and null ordering. This field is required if writeOrder is provided.
    - (dict) –
      
      Defines a single sort field in an Iceberg sort order specification.
      - sourceId (integer) – [REQUIRED]
        
        The ID of the source schema field to sort by. This must reference a valid field ID from the table schema.
      - transform (string) – [REQUIRED]
        
        The transform to apply to the source field before sorting. Use identity to sort by the field value directly, or specify other transforms as needed.
      - direction (string) – [REQUIRED]
        
        The sort direction. Valid values are asc for ascending order or desc for descending order.
      - nullOrder (string) – [REQUIRED]
        
        Specifies how null values are ordered. Valid values are nulls-first to place nulls before non-null values, or nulls-last to place nulls after non-null values.
sourceType (string) –
[REQUIRED]

The type of the source. It must match the member that you set in source.
source (dict) –
[REQUIRED]

The source table to migrate.

Note
This is a Tagged Union structure. Only one of the following top level keys can be set: glueTable.
- glueTable (dict) –
  
  An Amazon Web Services Glue Data Catalog table to migrate. The table must be an external Apache Hive table whose data is stored as Parquet in a general purpose bucket in the same account and Region as the table bucket. Views, transactional (ACID) tables, tables in an open table format such as Apache Iceberg, and tables registered with Amazon Web Services Lake Formation can’t be migrated. If the table can’t be migrated, the migration fails, and the table compatibility report lists the reason.
  - catalogId (string) – [REQUIRED]
    
    The ID of the Data Catalog that contains the table. It must be your account ID.
  - databaseName (string) – [REQUIRED]
    
    The name of the database that contains the table.
  - tableName (string) – [REQUIRED]
    
    The name of the table.
dataMigration (dict) –
The maintenance settings and migration policies for the destination table.
- maintenanceSettings (dict) –
  
  The maintenance settings for the destination table. The migration applies them when it creates the table: snapshot management starts then, and compaction starts after you complete the migration. If you don’t specify a setting, the migration uses its default: a target file size of 512 MB, the binpack compaction strategy, at least 3 snapshots, and a maximum snapshot age of 120 hours. Unreferenced file removal follows the table bucket’s configuration after you complete the migration.
  - icebergCompaction (dict) –
    
    The compaction settings.
    - targetFileSizeMB (integer) –
      
      The target size, from 64 to 512 MB, of the data files that the migration writes, and of the files that compaction writes after you complete the migration. The default is 512.
    - strategy (string) –
      
      The compaction strategy. The only valid value is binpack, which is the default.
  - icebergSnapshotManagement (dict) –
    
    The snapshot management settings.
    - minSnapshotsToKeep (integer) –
      
      The minimum number of snapshots to keep. The value must be 3 or more. The default is 3.
    - maxSnapshotAgeHours (integer) –
      
      The maximum age, in hours, of a snapshot before it can be expired. The default is 120.
- migrationPolicies (dict) –
  
  The policies for source data that doesn’t fit the destination schema.
  - unmatchedSourceFieldsPolicy (string) –
    
    The migration policy for a source field that has no matching column in the destination schema. Valid values:
    - fail - Fails the migration. This is the default.
    - drop - Leaves source columns that the destination schema doesn’t have out of the destination table, and doesn’t migrate source objects outside the Amazon Web Services Glue table’s partition locations.
    A destination schema that leaves out a partition column or a field of a struct column fails the migration under either value.
  - incompatibleValuesPolicy (string) –
    
    The migration policy for a source value that the destination column’s type can’t hold. Valid values:
    - fail - Fails the migration. This is the default.
    - setNull - Writes null for a value that the destination column’s type can’t hold. If a column’s type in a source object is incompatible with the destination column’s type, the column is null for that whole object.
    A required column, or a type that migration doesn’t support, fails the migration under either value.
reviewMigrationPlan (boolean) – Specifies whether the migration waits in the pendingReview state, so that you can review the compatibility reports before it migrates any data. If true, the migration continues when you call UpdateTableMigration with requestedState set to queued. If you don’t continue or cancel the migration within 30 days, the migration fails. The default is false.
migrationOutput (dict) –
[REQUIRED]

The bucket and prefixes that the migration writes its reports to. The bucket must be a general purpose bucket in your account and in the same Region as the table bucket, and the migration role must be able to write objects to it. Otherwise, the migration fails when it writes its first report.
- bucket (string) – [REQUIRED]
  
  The general purpose bucket that the migration writes every report to.
- prefixes (dict) –
  
  The key prefix for each report.
  - tableCompatibilityReport (string) –
    
    The key prefix for the table compatibility report, up to 512 bytes. If not specified, the report’s keys start at the root of the bucket.
  - objectCompatibilityReport (string) –
    
    The key prefix for the object compatibility report, up to 512 bytes. If not specified, the report’s keys start at the root of the bucket.
  - objectMigrationReport (string) –
    
    The key prefix for the object migration report, up to 512 bytes. If not specified, the report’s keys start at the root of the bucket.

Return type:

dict

Returns:

Response Syntax

{
    'migrationId': 'string',
    'state': 'new'|'planning'|'pendingReview'|'queued'|'migrating'|'succeeded'|'failed'|'cancelling'|'cancelled'
}

Response Structure

(dict) –
- migrationId (string) –
  
  The ID of the migration.
- state (string) –
  
  The state of the migration when the request is accepted, such as planning.

start_table_migration¶

Request Syntax

Note

Response Syntax

Response Structure

Exceptions