共享计数器显示负值

Question

我正试图在脚本中实现多处理计数器，但不确定我做得是否正确，因为结果没有任何意义。

# Global shared variable 
total_bytes_written = multiprocessing.Value('i', 0)
# Worker method
def s3_put_worker(**kwargs):
global total_bytes_written
# local var that stores a data chunk length
data_chunk_len = len(data_chunk)
while not stop_event.is_set():
# do some work
# ...
# The counter that sums shared_total_written variable with local var
with total_bytes_written.get_lock():
total_bytes_written.value += data_chunk_len
# I'm using ProcessPoolExecutor to start several Gevent thread pools that runs my worker method

现在，在停止脚本后，我得到了那些奇怪的结果：

2018-11-06 04:42:55,412; [s3_stress.s3_stress_runner] (MainProcess : MainThread) INFO - Total bytes written to storage: -1946157056

我怀疑我无法将多处理共享的var与本地方法的var相加，但我找不到关于这个主题的任何信息，也找不到如何解决这个问题。

感谢

Answer 1

您的问题似乎来自溢出：Value('i', 0)是一个有符号的32位整数，其计数可达2147483647(字节计数大约为2GiB(。

>>> total_bytes_written = multiprocessing.Value('i', 0)
>>> total_bytes_written.value += 2147483647  # 2GiB - 1B
>>> total_bytes_written
<Synchronized wrapper for c_int(2147483647)>
>>> total_bytes_written.value += 1
>>> total_bytes_written
<Synchronized wrapper for c_int(-2147483648)>

Value的最大数据类型是无符号长-长(至少64位(的'Q'。这可以处理范围[0, 18_446_744_073_709_551_615]或高达16EiB的字节计数。如果您关心正确的符号处理，则带符号的长-长类型'q'仍然可以计数到8EiB。

>>> total_bytes_written = multiprocessing.Value('q', 0)
>>> total_bytes_written.value += 9223372036854775807  # 8EiB - 1B
>>> total_bytes_written
<Synchronized wrapper for c_long(9223372036854775807)>