批量上传文件到ADLS gen 1

Question

是否可以一次上传或转换多个文件到adls(批量上传)。如果这是不可能的任何并行处理功能可用?? ?

我正在寻找c#的例子。请告知. .

Answer 1

如果需要在一个目录下上传多个文件，可以使用BulkUpload方法实现批量上传。关于如何使用该方法，请参考此处

例如

1. 创建服务主体并配置权限

2. 代码

string TENANT = "";
string CLIENTID = "";
Uri ADL_TOKEN_AUDIENCE = new Uri(@"https://datalake.azure.net/");
string secret_key = "";
var adlServiceSettings = ActiveDirectoryServiceSettings.Azure;
adlServiceSettings.TokenAudience = ADL_TOKEN_AUDIENCE;
var adlCred = await ApplicationTokenProvider.LoginSilentAsync(TENANT, CLIENTID, secret_key, adlServiceSettings);
AdlsClient client = AdlsClient.CreateClient("<>.azuredatalakestore.net", adlCred);

但请注意，该方法有局限性。它只能用于一次上传同一目录下的文件，它将上传该目录下的整个文件。如果你想上传一个目录下的文件或者不同目录下的文件，请参考下面的代码

static async  Task  Main(string[] args)
{
string TENANT = "";
string CLIENTID = "";
Uri ADL_TOKEN_AUDIENCE = new Uri(@"https://datalake.azure.net/");
string secret_key = "";
var adlServiceSettings = ActiveDirectoryServiceSettings.Azure;
adlServiceSettings.TokenAudience = ADL_TOKEN_AUDIENCE;
var adlCred = await ApplicationTokenProvider.LoginSilentAsync(TENANT, CLIENTID, secret_key, adlServiceSettings);
AdlsClient client = AdlsClient.CreateClient("<>.azuredatalakestore.net", adlCred);

string[] filePaths = { };
int count = 0;
var tasks = new Queue<Task>();
foreach (string filePath in filePaths)
{

var fileName = "";
// Add the upload task to the queue
tasks.Enqueue(uplaodFile(client, filePath, fileName));
count++;
}
// Run all the tasks asynchronously.
await Task.WhenAll(tasks);

}
private static async  Task uplaodFile(AdlsClient client, string filePath, string fileName)
{
using (var createStream = client.CreateFile(fileName, IfExists.Overwrite))
{
using (var fileStrem = File.OpenRead(filePath))
{
await fileStrem.CopyToAsync(createStream);
}
}
}