在线程中运行时 Python 记录器消息冲突

Question

使用模块线程和记录器进行并发文件下载。 似乎按预期工作。 唯一的问题是我每次下载记录的状态行相互冲突

代码片段

import sys, os, re
import threading
import requests
import queue
import logging
import time
import argparse
def download(session, url, filename):
connection_pool.acquire()
url = re.sub(r'/+$', '', url)
filename = re.sub(r'^/+', '', filename)
localpath = args.basepath + "/" + filename
os.makedirs(os.path.dirname(localpath), exist_ok=True)
full_url = url + "/" + filename + "?B"
starttime = time.time()
logger.info("Downloading {}".format(full_url))
response = session.get(full_url, allow_redirects=True, stream=True)
filesize = response.headers.get('Content-Length')
with open(localpath, 'wb') as f:
for chunk in response.iter_content(1024):
f.write(chunk)
local_filesize = os.path.getsize(localpath)
if (int(filesize) != int(local_filesize)):
logger.error("File sizes do not match remote={} local={}".format(filesize, local_filesize))
exit(1)
endtime = time.time()
elapsedtime = (endtime - starttime)
bitrate = int(filesize) / elapsedtime
logger.info("STATUS - file={} size={} time={} rate={} region={}".format(filename, filesize, elapsedtime, int(bitrate), args.region))
connection_pool.release()
def setup_logger():
hdlr = logging.StreamHandler()
hdlr.flush = sys.stdout.flush
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logger.setLevel(logging.DEBUG)
... bunch of other functions ...

输出：

895 2020-05-30 14:29:32,992 - get_files.py - INFO - Downloading https://10.12.134.17/3010A120200528.z
p?B 20-05-30 14:29:28,564 - get_files.py - INFO - STATUS - file=3066A120200528.zip size=9427150 time=8.5233154296875 rate=110604
2020-05-30 14:29:28,565 - get_files.py - INFO - Downloading https://10.12.134.17/3033A120200528.zip?
2020-05-30 14:29:29,610 - get_files.py - INFO - STATUS - file=3038A120200528.zip size=2472569 time=2.0577292442321777 rate=1201
00 2020-05-30 14:29:29,610 - get_files.py - INFO - Downloading https://10.12.134.17/3031A120200528.zi
?B 2020-05-30 14:29:29,836 - get_files.py - INFO - STATUS - file=3034A120200528.zip size=1441683 time=1.5025453567504883 rate=95
493 2020-05-30 14:29:29,836 - get_files.py - INFO - Downloading https://10.12.134.17/3030A120200528.z
p?B 2020-05-30 14:29:30,065 - get_files.py - INFO - STATUS - file=3037A120200528.zip size=3431121 time=2.3450839519500732 rate=14
3112 2020-05-30 14:29:30,065 - get_files.py - INFO - Downloading https://10.12.134.17/3029A120200528.
p?B 20-05-30 14:29:28,564 - get_files.py - INFO - STATUS - file=3066A120200528.zip size=9427150 time=8.5233154296875 rate=110604
2020-05-30 14:29:28,565 - get_files.py - INFO - Downloading https://10.12.134.17/3033A120200528.zip?

如上所示，来自某个线程的记录器消息与其他线程冲突。

• 旁注，这是在运行 amazonlinux：2 映像并使用 python 3.6 记录到 AWS Cloudwatch 日志流的 AWS ECS Fargate v1.4.0 容器中运行的
。

Answer 1

虽然它起作用的原因对我来说没有意义，但解决方案来自这篇文章。 完全删除处理程序并仅使用默认处理程序后，不会再发生行冲突。