我如何上传超过3gb的文件到azure datalake在.net核心api?

Question

我的应用程序有React UI .net core 6微服务，然后微服务从UI获取流数据，我们试图上传到DataLake，但它需要更长的时间或失败。

我已经尝试过用fetch和put调用块上传，但它也不能与文件一起工作我们是否可以使用azcopy与流或任何其他选项，如Azure数据移动，它可以与流数据并行上传，从而保证并提高从UI上传大文件的速度。

public async Task<bool> UploadFile(Guid id, Stream file, string name, string contentType)
{
bool ret = true;
int chunckSize = 2097152; //2MB
int totalChunks = (int)(file.Length / chunckSize);
if (file.Length % chunckSize != 0)
{
totalChunks++;
}
for (int i = 0; i < totalChunks; i++)
{
long position = (i * (long)chunckSize);
int toRead = (int)Math.Min(file.Length - position + 1, chunckSize);
byte[] buffer = new byte[toRead];
await file.ReadAsync(buffer, 0, toRead);
MultipartFormDataContent content = new MultipartFormDataContent();
content.Add(new StringContent(id.ToString()), "id");
var meta = JsonConvert.SerializeObject(new ChunkMetaData
{
UploadUid = id.ToString(),
FileName = name,
ChunkIndex = i,
TotalChunks = totalChunks,
TotalFileSize = file.Length,
ContentType = contentType
});
content.Add(new StringContent(meta), "metaData");
using (var ms = new MemoryStream(buffer))
{
content.Add(new StreamContent(ms),"files");
var response = await Client.PostAsync("/api/Upload", content).ConfigureAwait(false);
if (!response.IsSuccessStatusCode)
{
ret = false;
break;
}
}
}
return ret;
}

}

Answer 1

上传大文件有一个UploadAsync方法:

public async Task UploadFileBulk(DataLakeFileSystemClient fileSystemClient)
{
DataLakeDirectoryClient directoryClient =
fileSystemClient.GetDirectoryClient("my-directory");
DataLakeFileClient fileClient = directoryClient.GetFileClient("uploaded-file.txt");
FileStream fileStream =
File.OpenRead("C:\Users\contoso\file-to-upload.txt");
await fileClient.UploadAsync(fileStream);
}

见:https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-dotnet