使用地形的Athena Workgroup使用警报

Question

是否有任何方法可以使用terraform创建Athena工作组使用警报。我在地形资源块中看到，没有选项可以设置使用SNS主题的使用警报

我尝试过地形文档和其他在线资源，但从未找到任何有用的信息

resource "aws_athena_workgroup" "athena_team_workgroup" {
for_each      = var.athena_workgroups
name          = format("%s", each.key)
description   = "Athena workgroup"
force_destroy = false
tags = merge({
Name : format("%s-workgroup", each.key)
UseCase : format("%s-data-sharing", each.key)
UsedBy : format("%s", each.key)
Description : "workgroup of Athena allows to query glue catalog databases and tables"
}, local.tags)

state = each.value

configuration {
enforce_workgroup_configuration    = true
publish_cloudwatch_metrics_enabled = true
bytes_scanned_cutoff_per_query     = 52428800000
requester_pays_enabled             = false
result_configuration {
output_location = format("s3://%s/workgroups/%s/", module.s3_abi_athena_workgroups.bucket_name, each.key)
}
}
}

Answer 1

Terraform似乎没有工作组范围的数据使用控制限制功能(如AWS Athena文档中所述(。

然而，使用地形模板可以重新创建相同的功能。为了做到这一点，您需要使用lambda和/或SNS的以下关键组件：

• 设置Cloudwatch警报
• 设置Cloudwatch警报规则
  • 调用lambda的权限/或在SNS上写入的权限
• 设置读数和/或执行器
  • 设置一个Lambda函数，该函数将根据规则状态禁用/启用工作组(并可选择取消所有正在运行的查询(
    • 列出工作组(athena:ListWorkGroups(并更新它们(athena:UpdateWorkGroup(的权限
  • 设置SNS以扇出消息和/或发送警报

请注意，Athena WorkGroup度量发布和触发Lambda函数之间存在滞后，因此，在此期间启动的所有查询都将按扫描的数据量收费。

如果你试图使用AWS lambda函数禁用Athena Workgroup，这里是你的地形应该是什么样子的要点(不完全功能，也不包括权限部分(。这个例子是在了解了工作组范围内的数据使用控制限制通过仪表板完成的方式后放在一起的：

variable "total_datascan_threshold" {
description = "total amount of data scanned in bytes"
type        = number
}
variable "period" {
description = "period in seconds over which the specified statistic is applied"
type        = number
}
resource "aws_athena_workgroup" "a_workgroup" {
name          = "athena-wg"
description   = "an Athena workgroup"
force_destroy = false
configuration {
enforce_workgroup_configuration    = true
publish_cloudwatch_metrics_enabled = true
bytes_scanned_cutoff_per_query     = 52428800000
requester_pays_enabled             = false
result_configuration {
output_location = "s3://results_bucket/queries/"
}
}
}
# athena workgroup alarm
resource "aws_cloudwatch_metric_alarm" "athena_wg_datascan" {
alarm_name          = "athena-wg-datascan"
alarm_description   = "Monitors Athena WorkGroup total data scan"
comparison_operator = "GreaterThanOrEqualToThreshold"
evaluation_periods  = "1"
threshold           = format("%d", floor(var.total_datascan_treshold))
treat_missing_data = "notBreaching"
metric_query {
id          = "wg1"
return_data = "true"
metric {
metric_name = "ProcessedBytes"
namespace   = "AWS/Athena"
period      = format("%d", floor(var.period))
stat        = "Sum"
dimensions = {
WorkGroup = "${aws_athena_workgroup.a_workgroup.id}"
}
}
}
}
# event rule to switch athena workgroup
resource "aws_cloudwatch_event_rule" "switch_athena_wg" {
name        = "switch-athena-wg-state"
description = "Switch Athena WorkGroup based on event rule"
event_pattern = jsonencode(
{
"source" : [
"aws.cloudwatch"
],
"detail-type" : [
"CloudWatch Alarm State Change"
],
"resources" : [
"${aws_cloudwatch_metric_alarm.athena_wg_datascan.arn}"
]
}
)
}
# connect eventbridge to lambda function
resource "aws_cloudwatch_event_target" "switch_athena_wg" {
target_id = "switch-athena-wg"
arn       = aws_lambda_function.athena_wg_switch.arn
rule      = aws_cloudwatch_event_rule.switch_athena_wg.name
}
# master switch lambda function
resource "aws_lambda_function" "athena_wg_switch" {
function_name = "athena-wg-switch"
filename      = "${path.module}/athena_wg_switch.zip"
role = aws_iam_role.lambda_role.arn
handler = "main.lambda_handler"
runtime = var.runtime
}
data "archive_file" "athena_wg_switch" {
type        = "zip"
source_dir  = "${path.module}/athena-wg-switch/"
output_path = "${path.module}/athena_wg_switch.zip"
}
# grant permissions to invoke lambda function
resource "aws_lambda_permission" "eventbridge_trigger" {
statement_id  = "allow-eventbridge-trigger"
action        = "lambda:InvokeFunction"
function_name = aws_lambda_function.athena_wg_switch.function_name
principal     = "events.amazonaws.com"
source_arn    = aws_cloudwatch_event_rule.switch_athena_wg.arn
}

athena-wg-switch/main.py:中的Lambda函数代码

import os
from boto3 import client

class UnknownState(Exception):
"""Unable to handle workgroup state different than ENABLED or DISABLED"""
def lambda_handler(event, context):
"""
Updates the Athena WorkGroup state bases on the alarm state value
"""
athena = client("athena")
response = None
workgroup = [
m["metricStat"]["metric"]["dimensions"]["WorkGroup"]
for m in event["detail"]["configuration"]["metrics"]
][0]
state = event["detail"]["state"]["value"]
if state == "ALARM":
new_state = "DISABLED"
elif state == "OK":
new_state = "ENABLED"
else:
raise UnknownState(f'Unexpected {event["detail"]} state: {event}')
return athena.update_work_group(WorkGroup=workgroup, State=new_state)