在 ASP.NET 酷睿3.1中上传和下载大文件?

我正在使用干净的架构进行 ASP.NET Core 3.1 API项目，我有以下classlibs(层(：

基础设施(安全的东西和上传助手等...
持久性(DA 层(
(域模型(
应用程序(用例 - 业务逻辑(
API(API 项目作为我的启动项目(

我希望能够将大文件上传到服务器(例如 2Gb 的文件大小甚至更多(并在此之后下载它们，并希望这样做而不会在未来出现内存溢出和所有问题。

任何帮助将不胜感激。

如果您有这么大的文件，切勿在代码中使用byte[]或MemoryStream。仅在下载/上传文件时对流进行操作。

您有以下几种选择：

如果您同时控制客户端和服务器，请考虑使用 tus 之类的东西。.NET 有客户端和服务器实现。这可能是最简单，最强大的选择。
如果使用 HttpClient 上传大文件，只需使用StreamContent类发送它们即可。同样，不要使用MemoryStream作为源，而是使用其他类似FileStream的东西。
如果使用 HttpClient 下载大文件，请务必指定 HttpCompletionOptions，例如var response = await httpClient.SendAsync(httpRequest, HttpCompletionOption.ResponseHeadersRead)。否则，HttpClient 会将整个响应缓冲在内存中。然后，您可以通过var stream = response.Content.ReadAsStreamAsync()将响应文件作为流进行处理。

ASP.NET 核心具体建议：

如果要通过 HTTP POST 接收文件，则需要增加请求大小限制：[RequestSizeLimit(10L * 1024L * 1024L * 1024L)]和[RequestFormLimits(MultipartBodyLengthLimit = 10L * 1024L * 1024L * 1024L)]。另外，需要禁用表单值绑定，否则整个请求会被缓冲到内存中：

[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method)]
public class DisableFormValueModelBindingAttribute : Attribute, IResourceFilter
{
public void OnResourceExecuting(ResourceExecutingContext context)
{
var factories = context.ValueProviderFactories;
factories.RemoveType<FormValueProviderFactory>();
factories.RemoveType<FormFileValueProviderFactory>();
factories.RemoveType<JQueryFormValueProviderFactory>();
}
public void OnResourceExecuted(ResourceExecutedContext context)
{
}
}

要从控制器返回文件，只需通过File方法返回文件，该方法接受流：return File(stream, mimeType, fileName);

示例控制器如下所示(有关缺少的帮助程序类，请参阅 https://learn.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-3.1(：

private const MaxFileSize = 10L * 1024L * 1024L * 1024L; // 10GB, adjust to your need
[DisableFormValueModelBinding]
[RequestSizeLimit(MaxFileSize)]
[RequestFormLimits(MultipartBodyLengthLimit = MaxFileSize)]
public async Task ReceiveFile()
{
if (!MultipartRequestHelper.IsMultipartContentType(Request.ContentType))
throw new BadRequestException("Not a multipart request");
var boundary = MultipartRequestHelper.GetBoundary(MediaTypeHeaderValue.Parse(Request.ContentType));
var reader = new MultipartReader(boundary, Request.Body);
// note: this is for a single file, you could also process multiple files
var section = await reader.ReadNextSectionAsync();
if (section == null)
throw new BadRequestException("No sections in multipart defined");
if (!ContentDispositionHeaderValue.TryParse(section.ContentDisposition, out var contentDisposition))
throw new BadRequestException("No content disposition in multipart defined");
var fileName = contentDisposition.FileNameStar.ToString();
if (string.IsNullOrEmpty(fileName))
{
fileName = contentDisposition.FileName.ToString();
}
if (string.IsNullOrEmpty(fileName))
throw new BadRequestException("No filename defined.");
using var fileStream = section.Body;
await SendFileSomewhere(fileStream);
}
// This should probably not be inside the controller class
private async Task SendFileSomewhere(Stream stream)
{
using var request = new HttpRequestMessage()
{
Method = HttpMethod.Post,
RequestUri = new Uri("YOUR_DESTINATION_URI"),
Content = new StreamContent(stream),
};
using var response = await _httpClient.SendAsync(request);
// TODO check response status etc.
}

在此示例中，我们将整个文件流式传输到另一个服务。在某些情况下，最好将文件临时保存到磁盘。

有时问题是我们使用 Nginx 作为部署在 ubuntu/Linux 环境中 docker 中的 asp.net Core 应用程序的前端代理。这正是我尝试在 docker 或 .net 核心端调试的情况，但实际解决方案是通过将 Nginx 配置配置为

client_max_body_size 50米;

此行可以添加到您遇到问题的站点的 Nginx 配置的位置或服务器设置部分。

可能对某人有帮助。

我建议阅读MicrosoftDocs Upload ASP.NET Core中的文件，@ManuelAllenspach也链接到该文件。

https://learn.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-6.0

您应该查看的部分是根据您的要求上传带有流式传输的大文件：(例如 2Gb 的文件大小甚至更多(

https://learn.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-6.0#upload-large-files-with-streaming

public class GenerateAntiforgeryTokenCookieAttribute : ResultFilterAttribute
{
public override void OnResultExecuting(ResultExecutingContext context)
{
var antiforgery = context.HttpContext.RequestServices.GetService<IAntiforgery>();
// Send the request token as a JavaScript-readable cookie
var tokens = antiforgery.GetAndStoreTokens(context.HttpContext);
context.HttpContext.Response.Cookies.Append(
"RequestVerificationToken",
tokens.RequestToken,
new CookieOptions() { HttpOnly = false });
}
public override void OnResultExecuted(ResultExecutedContext context)
{
}
}
[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method)]
public class DisableFormValueModelBindingAttribute : Attribute, IResourceFilter
{
public void OnResourceExecuting(ResourceExecutingContext context)
{
var factories = context.ValueProviderFactories;
factories.RemoveType<FormValueProviderFactory>();
factories.RemoveType<FormFileValueProviderFactory>();
factories.RemoveType<JQueryFormValueProviderFactory>();
}
public void OnResourceExecuted(ResourceExecutedContext context)
{
}
}
services.AddRazorPages(options =>
{
options.Conventions
.AddPageApplicationModelConvention("/StreamedSingleFileUploadDb",
model =>
{
model.Filters.Add(
new GenerateAntiforgeryTokenCookieAttribute());
model.Filters.Add(
new DisableFormValueModelBindingAttribute());
});
options.Conventions
.AddPageApplicationModelConvention("/StreamedSingleFileUploadPhysical",
model =>
{
model.Filters.Add(
new GenerateAntiforgeryTokenCookieAttribute());
model.Filters.Add(
new DisableFormValueModelBindingAttribute());
});
});
[HttpPost]
[DisableFormValueModelBinding]
[ValidateAntiForgeryToken]
public async Task<IActionResult> UploadDatabase()
{
if (!MultipartRequestHelper.IsMultipartContentType(Request.ContentType))
{
ModelState.AddModelError("File", 
$"The request couldn't be processed (Error 1).");
// Log error
return BadRequest(ModelState);
}
// Accumulate the form data key-value pairs in the request (formAccumulator).
var formAccumulator = new KeyValueAccumulator();
var trustedFileNameForDisplay = string.Empty;
var untrustedFileNameForStorage = string.Empty;
var streamedFileContent = Array.Empty<byte>();
var boundary = MultipartRequestHelper.GetBoundary(
MediaTypeHeaderValue.Parse(Request.ContentType),
_defaultFormOptions.MultipartBoundaryLengthLimit);
var reader = new MultipartReader(boundary, HttpContext.Request.Body);
var section = await reader.ReadNextSectionAsync();
while (section != null)
{
var hasContentDispositionHeader = 
ContentDispositionHeaderValue.TryParse(
section.ContentDisposition, out var contentDisposition);
if (hasContentDispositionHeader)
{
if (MultipartRequestHelper
.HasFileContentDisposition(contentDisposition))
{
untrustedFileNameForStorage = contentDisposition.FileName.Value;
// Don't trust the file name sent by the client. To display
// the file name, HTML-encode the value.
trustedFileNameForDisplay = WebUtility.HtmlEncode(
contentDisposition.FileName.Value);
streamedFileContent = 
await FileHelpers.ProcessStreamedFile(section, contentDisposition, 
ModelState, _permittedExtensions, _fileSizeLimit);
if (!ModelState.IsValid)
{
return BadRequest(ModelState);
}
}
else if (MultipartRequestHelper
.HasFormDataContentDisposition(contentDisposition))
{
// Don't limit the key name length because the 
// multipart headers length limit is already in effect.
var key = HeaderUtilities
.RemoveQuotes(contentDisposition.Name).Value;
var encoding = GetEncoding(section);
if (encoding == null)
{
ModelState.AddModelError("File", 
$"The request couldn't be processed (Error 2).");
// Log error
return BadRequest(ModelState);
}
using (var streamReader = new StreamReader(
section.Body,
encoding,
detectEncodingFromByteOrderMarks: true,
bufferSize: 1024,
leaveOpen: true))
{
// The value length limit is enforced by 
// MultipartBodyLengthLimit
var value = await streamReader.ReadToEndAsync();
if (string.Equals(value, "undefined", 
StringComparison.OrdinalIgnoreCase))
{
value = string.Empty;
}
formAccumulator.Append(key, value);
if (formAccumulator.ValueCount > 
_defaultFormOptions.ValueCountLimit)
{
// Form key count limit of 
// _defaultFormOptions.ValueCountLimit 
// is exceeded.
ModelState.AddModelError("File", 
$"The request couldn't be processed (Error 3).");
// Log error
return BadRequest(ModelState);
}
}
}
}
// Drain any remaining section body that hasn't been consumed and
// read the headers for the next section.
section = await reader.ReadNextSectionAsync();
}
// Bind form data to the model
var formData = new FormData();
var formValueProvider = new FormValueProvider(
BindingSource.Form,
new FormCollection(formAccumulator.GetResults()),
CultureInfo.CurrentCulture);
var bindingSuccessful = await TryUpdateModelAsync(formData, prefix: "",
valueProvider: formValueProvider);
if (!bindingSuccessful)
{
ModelState.AddModelError("File", 
"The request couldn't be processed (Error 5).");
// Log error
return BadRequest(ModelState);
}
// **WARNING!**
// In the following example, the file is saved without
// scanning the file's contents. In most production
// scenarios, an anti-virus/anti-malware scanner API
// is used on the file before making the file available
// for download or for use by other systems. 
// For more information, see the topic that accompanies 
// this sample app.
var file = new AppFile()
{
Content = streamedFileContent,
UntrustedName = untrustedFileNameForStorage,
Note = formData.Note,
Size = streamedFileContent.Length, 
UploadDT = DateTime.UtcNow
};
_context.File.Add(file);
await _context.SaveChangesAsync();
return Created(nameof(StreamingController), null);
}
using System;
using System.IO;
using Microsoft.Net.Http.Headers;
namespace SampleApp.Utilities
{
public static class MultipartRequestHelper
{
// Content-Type: multipart/form-data; boundary="----WebKitFormBoundarymx2fSWqWSd0OxQqq"
// The spec at https://tools.ietf.org/html/rfc2046#section-5.1 states that 70 characters is a reasonable limit.
public static string GetBoundary(MediaTypeHeaderValue contentType, int lengthLimit)
{
var boundary = HeaderUtilities.RemoveQuotes(contentType.Boundary).Value;
if (string.IsNullOrWhiteSpace(boundary))
{
throw new InvalidDataException("Missing content-type boundary.");
}
if (boundary.Length > lengthLimit)
{
throw new InvalidDataException(
$"Multipart boundary length limit {lengthLimit} exceeded.");
}
return boundary;
}
public static bool IsMultipartContentType(string contentType)
{
return !string.IsNullOrEmpty(contentType)
&& contentType.IndexOf("multipart/", StringComparison.OrdinalIgnoreCase) >= 0;
}
public static bool HasFormDataContentDisposition(ContentDispositionHeaderValue contentDisposition)
{
// Content-Disposition: form-data; name="key";
return contentDisposition != null
&& contentDisposition.DispositionType.Equals("form-data")
&& string.IsNullOrEmpty(contentDisposition.FileName.Value)
&& string.IsNullOrEmpty(contentDisposition.FileNameStar.Value);
}
public static bool HasFileContentDisposition(ContentDispositionHeaderValue contentDisposition)
{
// Content-Disposition: form-data; name="myfile1"; filename="Misc 002.jpg"
return contentDisposition != null
&& contentDisposition.DispositionType.Equals("form-data")
&& (!string.IsNullOrEmpty(contentDisposition.FileName.Value)
|| !string.IsNullOrEmpty(contentDisposition.FileNameStar.Value));
}
}
}
[HttpPost]
[DisableFormValueModelBinding]
[ValidateAntiForgeryToken]
public async Task<IActionResult> UploadPhysical()
{
if (!MultipartRequestHelper.IsMultipartContentType(Request.ContentType))
{
ModelState.AddModelError("File", 
$"The request couldn't be processed (Error 1).");
// Log error
return BadRequest(ModelState);
}
var boundary = MultipartRequestHelper.GetBoundary(
MediaTypeHeaderValue.Parse(Request.ContentType),
_defaultFormOptions.MultipartBoundaryLengthLimit);
var reader = new MultipartReader(boundary, HttpContext.Request.Body);
var section = await reader.ReadNextSectionAsync();
while (section != null)
{
var hasContentDispositionHeader = 
ContentDispositionHeaderValue.TryParse(
section.ContentDisposition, out var contentDisposition);
if (hasContentDispositionHeader)
{
// This check assumes that there's a file
// present without form data. If form data
// is present, this method immediately fails
// and returns the model error.
if (!MultipartRequestHelper
.HasFileContentDisposition(contentDisposition))
{
ModelState.AddModelError("File", 
$"The request couldn't be processed (Error 2).");
// Log error
return BadRequest(ModelState);
}
else
{
// Don't trust the file name sent by the client. To display
// the file name, HTML-encode the value.
var trustedFileNameForDisplay = WebUtility.HtmlEncode(
contentDisposition.FileName.Value);
var trustedFileNameForFileStorage = Path.GetRandomFileName();
// **WARNING!**
// In the following example, the file is saved without
// scanning the file's contents. In most production
// scenarios, an anti-virus/anti-malware scanner API
// is used on the file before making the file available
// for download or for use by other systems. 
// For more information, see the topic that accompanies 
// this sample.
var streamedFileContent = await FileHelpers.ProcessStreamedFile(
section, contentDisposition, ModelState, 
_permittedExtensions, _fileSizeLimit);
if (!ModelState.IsValid)
{
return BadRequest(ModelState);
}
using (var targetStream = System.IO.File.Create(
Path.Combine(_targetFilePath, trustedFileNameForFileStorage)))
{
await targetStream.WriteAsync(streamedFileContent);
_logger.LogInformation(
"Uploaded file '{TrustedFileNameForDisplay}' saved to " +
"'{TargetFilePath}' as {TrustedFileNameForFileStorage}", 
trustedFileNameForDisplay, _targetFilePath, 
trustedFileNameForFileStorage);
}
}
}
// Drain any remaining section body that hasn't been consumed and
// read the headers for the next section.
section = await reader.ReadNextSectionAsync();
}
return Created(nameof(StreamingController), null);
}

我发现这篇文章很有用 - https://www.tugberkugurlu.com/archive/efficiently-streaming-large-http-responses-with-httpclient

以下是下载大文件的代码版本：

static public async Task HttpDownloadFileAsync(HttpClient httpClient, string url, string fileToWriteTo) {
using HttpResponseMessage response = await httpClient.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
using Stream streamToReadFrom = await response.Content.ReadAsStreamAsync(); 
using Stream streamToWriteTo = File.Open(fileToWriteTo, FileMode.Create); 
await streamToReadFrom.CopyToAsync(streamToWriteTo);
}

问题是你必须处理长文件，无论你在哪里使用它们，你需要大量的资源才能读取。一种可能的解决方案是根据信息将文件划分为不同的块，或者在单独的作业或线程中处理它，或者在 .net 中使用并行性来处理它。您可以指定文件的大小，也请阅读以下对您非常有用的博客。

上传大文件

相关内容

最新更新

热门标签：