我有下面的配置，我从我的blob存储从数据块读取文件。我的第一个配置工作，它使用SAS密钥从blob容器。下一个是我无法读取存储帐户密钥。我得到的错误是
account .blob.core.windows.net中的容器没有找到，我们无法使用anoynomous凭证,
有效的配置:
spark.conf.set("fs.azure.sas.containername.storageaccount.blob.core.windows.net","sas-key")
dbutils.fs.ls("wasbs://containername@storageaccount.blob.core.windows.net/")
table_name= "main.deltalake_db.Customers_Log_Table_test"
checkpoint_path = "wasbs://testaudit@azdevstoreforlogs.blob.core.windows.net/_checkpoint"
file_path = "wasbs://containername@storageaccount.blob.core.windows.net/topics/logs/"
schema = "xHeaderFields ARRAY<STRUCT<key: STRING, value: STRING>>"
(spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", checkpoint_path)
.schema(schema)
.load(file_path)
.writeStream
.option("checkpointLocation", checkpoint_path)
.trigger(availableNow=True)
.toTable(table_name))
不工作的是:(我得到上面的错误)
dbutils.fs.mount(
source = "wasbs://containername@storageaccount.blob.core.windows.net",
mount_point = "/mnt/topics/logs",
extra_configs = {"fs.azure.account.key.storageaccount.blob.core.windows.net":dbutils.secrets.get(scope="keyvaultscope",key="storage-account-azuredevauditlogs")})
table_name= "main.deltalake_db.Customers_Log_Table_test"
checkpoint_path = "wasbs://testaudit@azdevstoreforlogs.blob.core.windows.net/_checkpoint"
file_path = "wasbs://containername@storageaccount.blob.core.windows.net/topics/logs/"
schema = "xHeaderFields ARRAY<STRUCT<key: STRING, value: STRING>>"
(spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", checkpoint_path)
.schema(schema)
.load(file_path)
.writeStream
.option("checkpointLocation", checkpoint_path)
.trigger(availableNow=True)
.toTable(table_name))
算出答案了…您需要在文件路径中具有相同的挂载点。在上面的代码中将file_path替换为:
file_path ="/mnt/topics/logs"
与mount path
中提供的相同