亚马逊 S3 表格构造库
本单元中更高层次的构件的API是实验性的并正在积极开发中. 在今后任何版本中,它们都须作非背向相容的修改或去除。 这些是 不适用 语义版本 模型和断开的更改 在发行笔记中宣布. 这意味着虽然你可能使用它们,但可能需要更新 当更新到此软件包的更新版本时,您的源代码.
亚马逊 S3 表
Amazon S3 Tables提供首个有内置Apache Iceberg支持的云对象商店,并精简了规模存储表格数据.
使用量
定义 S3 表格盒
// Build a Table bucket
const sampleTableBucket = new TableBucket(scope, 'ExampleTableBucket', {
// optional fields:
tableBucketName: 'example-bucket-1', // if omitted, CDK generates a unique name that satisfies the S3 Tables naming rules
unreferencedFileRemoval: {
status: UnreferencedFileRemovalStatus.ENABLED,
noncurrentDays: 20,
unreferencedDays: 20,
}
});
何时 tableBucketName 中省略,CDK从堆栈名称和
构建路径。 由于路径是输入的一部分,生成的名称是 没有
横跨堆栈或构造重命名 - 重命名或移动构造
生成不同的名称并替换桶。 如果你需要一个固定的,可预测的
名称( 例如, 引用来自 CDK 外部或隔开的桶)
部署),通过 tableBucketName 当然.
定义 S3 表格命名空间
// Build a namespace
const sampleNamespace = new Namespace(scope, 'ExampleNamespace', {
namespaceName: 'example-namespace-1',
tableBucket: tableBucket,
});
定义 S3 表格
// Build a table
const sampleTable = new Table(scope, 'ExampleTable', {
tableName: 'example_table',
namespace: namespace,
openTableFormat: OpenTableFormat.ICEBERG,
withoutMetadata: true,
});
// Build a table with an Iceberg Schema
const sampleTableWithSchema = new Table(scope, 'ExampleSchemaTable', {
tableName: 'example_table_with_schema',
namespace: namespace,
openTableFormat: OpenTableFormat.ICEBERG,
icebergMetadata: {
icebergSchema: {
schemaFieldList: [
{
name: 'id',
type: 'int',
required: true,
},
{
name: 'name',
type: 'string',
},
],
},
},
compaction: {
status: Status.ENABLED,
targetFileSizeMb: 128,
},
snapshotManagement: {
status: Status.ENABLED,
maxSnapshotAgeHours: 48,
minSnapshotsToKeep: 5,
},
});
更多关于表格桶维护操作和默认行为的信息 S3 表格用户指南
高级 Iceberg 表格配置
您可以配置分区规格、排序顺序和表格属性,以便优化查询性能.
将分区添加到表格的最简单方式 :
// Build a table with partition spec (minimal configuration)
const partitionedTable = new Table(scope, 'PartitionedTable', {
tableName: 'partitioned_table',
namespace: namespace,
openTableFormat: OpenTableFormat.ICEBERG,
icebergMetadata: {
icebergSchema: {
schemaFieldList: [
{ name: 'event_date', type: 'date', required: true },
{ name: 'event_name', type: 'string' },
],
},
icebergPartitionSpec: {
fields: [
{
sourceId: 1,
transform: IcebergTransform.IDENTITY,
name: 'date_partition',
},
],
},
},
});
对于全控,您也可以配置排序命令和表格属性 :
// Build a table with partition spec, sort order, and table properties
const advancedTable = new Table(scope, 'AdvancedTable', {
tableName: 'advanced_table',
namespace: namespace,
openTableFormat: OpenTableFormat.ICEBERG,
icebergMetadata: {
icebergSchema: {
schemaFieldList: [
{ id: 1, name: 'event_date', type: 'date', required: true },
{ id: 2, name: 'user_id', type: 'string', required: true },
],
},
icebergPartitionSpec: {
specId: 0,
fields: [
{
sourceId: 1,
transform: IcebergTransform.IDENTITY,
name: 'date_partition',
fieldId: 1000,
},
],
},
icebergSortOrder: {
orderId: 1,
fields: [
{
sourceId: 1,
transform: IcebergTransform.IDENTITY,
direction: SortDirection.ASC,
nullOrder: NullOrder.NULLS_LAST,
},
],
},
tableProperties: [
{ key: 'write.format.default', value: 'parquet' },
],
},
});
控制表盘权限
// Grant the principal read permissions to the bucket and all tables within
const accountId = '123456789012'
tableBucket.grantRead(new iam.AccountPrincipal(accountId), '*');
// Grant the role write permissions to the bucket and all tables within
const role = new iam.Role(stack, 'MyRole', { assumedBy: new iam.ServicePrincipal('sample') });
tableBucket.grantWrite(role, '*');
// Grant the user read and write permissions to the bucket and all tables within
tableBucket.grantReadWrite(new iam.User(stack, 'MyUser'), '*');
// Grant permissions to the bucket and a particular table within it
const tableId = '6ba046b2-26de-44cf-9144-0c7862593a7b'
tableBucket.grantReadWrite(new iam.AccountPrincipal(accountId), tableId);
// Add custom resource policy statements
const permissions = new iam.PolicyStatement({
effect: iam.Effect.ALLOW,
actions: ['s3tables:*'],
principals: [ new iam.ServicePrincipal('example.aws.internal') ],
resources: ['*']
});
tableBucket.addToResourcePolicy(permissions);
控制表格密钥加密设置
S3 TableBuckets 默认使用 S3 管理的密钥启用 SSE (服务器侧加密 AES-256). 您也可以为 KMS- SSE 带来自己的 KMS 密钥, 或者让 S3 为您创建 KMS 密钥 .
如果一个被 KMS 加密的桶, 桶上的赠款功能也会允许访问 到 TableBucket 的相关 KMS 密钥.
// Provide a user defined KMS Key:
const key = new kms.Key(scope, 'UserKey', {});
const encryptedBucket = new TableBucket(scope, 'EncryptedTableBucket', {
tableBucketName: 'table-bucket-1',
encryption: TableBucketEncryption.KMS,
encryptionKey: key,
});
// This account principal will also receive kms:Decrypt access to the KMS key
encryptedBucket.grantRead(new iam.AccountPrincipal('123456789012'), '*');
// Use S3 managed server side encryption (default)
const encryptedBucketDefault = new TableBucket(scope, 'EncryptedTableBucketDefault', {
tableBucketName: 'table-bucket-3',
encryption: TableBucketEncryption.S3_MANAGED, // Uses AES-256 encryption by default
});
当使用 KMS 加密时( W)TableBucketEncryption.KMS),如果不提供加密密钥,CDK会自动为表格桶创建新的KMS密钥,并获得必要的权限.
// If no key is provided, one will be created automatically
const encryptedBucketAuto = new TableBucket(scope, 'EncryptedTableBucketAuto', {
tableBucketName: 'table-bucket-2',
encryption: TableBucketEncryption.KMS,
});
启用云表请求量表
您可以启用 Cloud Watch 程序为您的餐桌桶请求参数 。 请求度量衡为亚马逊S3表格请求提供了洞察力,帮助您监控并优化了表格桶的使用.
有关S3表云观察测量的更多信息,请参见: S3 表云观察度量衡文档。 。 。 .
// Enable CloudWatch request metrics for the table bucket
const tableBucketWithMetrics = new TableBucket(scope, 'TableBucketWithMetrics', {
tableBucketName: 'metrics-enabled-bucket',
requestMetricsStatus: RequestMetricsStatus.ENABLED,
});
配置存储类
您可以为您的桌子桶和桌子配置存储类 。 存储类决定了数据的存储和计费方式,允许您优化不同的访问模式.
// Create a table bucket with INTELLIGENT_TIERING storage class
const tableBucketWithStorageClass = new TableBucket(scope, 'TableBucketWithStorageClass', {
tableBucketName: 'storage-class-bucket',
storageClass: StorageClass.INTELLIGENT_TIERING,
});
表默认从母桶中继承存储类. 您也可以在表格级别上覆盖存储类 :
// Create a table with explicit storage class (overrides bucket default)
const tableWithStorageClass = new Table(scope, 'TableWithStorageClass', {
tableName: 'storage_class_table',
namespace: namespace,
openTableFormat: OpenTableFormat.ICEBERG,
withoutMetadata: true,
storageClass: StorageClass.STANDARD,
});
可用的存储类 :
StorageClass.STANDARD- 经常获得的数据StorageClass.INTELLIGENT_TIERING- 根据使用模式,自动将数据移动到访问级别之间
注意:表存储类是只创建属性,在表创建后不能被更改.
控制表权限
// Grant the principal read permissions to the table
const accountId = '123456789012'
table.grantRead(new iam.AccountPrincipal(accountId));
// Grant the role write permissions to the table
const role = new iam.Role(stack, 'MyRole', { assumedBy: new iam.ServicePrincipal('sample') });
table.grantWrite(role);
// Grant the user read and write permissions to the table
table.grantReadWrite(new iam.User(stack, 'MyUser'));
// Grant an account permissions to the table
table.grantReadWrite(new iam.AccountPrincipal(accountId));
// Add custom resource policy statements
const permissions = new iam.PolicyStatement({
effect: iam.Effect.ALLOW,
actions: ['s3tables:*'],
principals: [ new iam.ServicePrincipal('example.aws.internal') ],
resources: ['*']
});
table.addToResourcePolicy(permissions);
缠绕
两者 TableBucket 财务报告和审定财务报表 Table 通过 CDK 标准标记机制支持标记 :
Tags.of(tableBucket).add('Environment', 'Production');
Tags.of(table).add('Team', 'DataEngineering');
// Stack-level tags propagate to all resources
Tags.of(stack).add('Project', 'DataLake');
快来了
L2 构造支持 :
- 表格的 KMS 加密支持