如何在转换数据类型时使"applymapping"正常工作

Question

我的问题是我的 ETL 作业中的特定列没有转换为受追捧的数据类型，这意味着每次运行作业并稍后爬网(每天)时，数据类型都是"字符串"，应该是"整数"。

在数据源中，数据类型是"bigint"，我希望它在最终表中是"bigint"或"int"。

但是，这不会发生，列"tester"的数据类型最终为字符串。

这是许多查询中使用的列，因此必须将其转换为整数(就像以前更改之前一样)。

我已经尝试了源数据类型和目标数据类型的多种组合，但是似乎没有一种有效，最终都是字符串。

我希望您能帮助我，因为我不是 AWS 的专家，并且无法获得任何进一步的支持。

在我的 ETL 作业代码下面被粘贴(审查)

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
import boto3
s3 = boto3.resource('s3')
bucket = s3.Bucket('XX-bi-team')
bucket.objects.filter(Prefix="Tables_after_jobs/XX/tester/").delete()

## @params: [JOB_NAME]
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
## @type: DataSource
## @args: [database = "mysql_XX", table_name = "XX_XX_XX", transformation_ctx = "datasource0"]
## @return: datasource0
## @inputs: []
datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "mysql_XX", table_name = "XX_XX_XX", transformation_ctx = "datasource0")
## @type: ApplyMapping
## @args: [mapping = [("v1", "string", "v1", "string"), ("XX_threshold_XX", "string", "XX_threshold_XX", "string"), ("limit_XX", "string", "limit_XX", "string"), ("XX_threshold", "decimal(19,2)", "XX_threshold", "decimal(19,2)"), ("created", "timestamp", "created", "timestamp"), ("gee_XX", "string", "gee_XX", "string"), ("type", "string", "type", "string"), ("uuid", "binary", "uuid", "binary"), ("v2", "int", "v2", "int"), ("geeXX", "decimal(19,2)", "geeXX", "decimal(19,2)"), ("limitXX", "decimal(29,2)", "limitXX", "decimal(29,2)"), ("XX", "int", "XX", "int"), ("id", "long", "id", "long"), ("XX", "decimal(29,2)", "XX", "decimal(29,2)"), ("XX_XX_id", "long", "XX_XX_id", "long"), ("replaces_id", "long", "tester", "int")], transformation_ctx = "applymapping1"]
## @return: applymapping1
## @inputs: [frame = datasource0]
applymapping1 = ApplyMapping.apply(frame = datasource0, mappings = [("v1", "string", "v1", "string"), ("XX_threshold_XX", "string", "XX_threshold_XX", "string"), ("limit_XX", "string", "limit_XX", "string"), ("XX_threshold", "decimal(19,2)", "XX_threshold", "decimal(19,2)"), ("created", "timestamp", "created", "timestamp"), ("gee_XX", "string", "gee_XX", "string"), ("type", "string", "type", "string"), ("uuid", "binary", "uuid", "binary"), ("v2", "int", "v2", "int"), ("geeXX", "decimal(19,2)", "geeXX", "decimal(19,2)"), ("limitXX", "decimal(29,2)", "limitXX", "decimal(29,2)"), ("XX", "int", "XX", "int"), ("id", "long", "id", "long"), ("XX", "decimal(29,2)", "XX", "decimal(29,2)"), ("XX_XX_id", "long", "XX_XX_id", "long"), ("tester", "bigint", "tester", "int")], transformation_ctx = "applymapping1")
## @type: DataSink
## @args: [connection_type = "s3", connection_options = {"path": "s3://XX-bi-team/Tables_after_jobs/XX/tester"}, format = "json", transformation_ctx = "datasink2"]
## @return: datasink2
## @inputs: [frame = applymapping1]
datasink2 = glueContext.write_dynamic_frame.from_options(frame = applymapping1, connection_type = "s3", connection_options = {"path": "s3://XX-bi-team/Tables_after_jobs/XX/tester"}, format = "json", transformation_ctx = "datasink2")
job.commit()

如上所述，我需要列"tester"成为整数而不是字符串，就像在当前状态下一样。

我希望你能帮助我。

提前感谢！

Answer 1

首先尝试打印datasource0(datasource0.printSchema( ))的模式以查看实际数据类型。如果是选择类型，则需要使用 ResolveChoice 解析它。

另请参阅此处。