Python文本帮助大型文件

Question

这段代码运行得很好，唯一的问题是它不适用于大型txt文件。1GB文本文件。我能做些什么来修复？

import os
file_1 = open('file1.txt', 'r', encoding='utf8').read().splitlines()
file_2 = open('file2.txt', 'r', encoding='utf8').read().splitlines()
[file_2.remove(l) for l in file_1 if l in file_2]
with open('file2.txt', 'w') as new_file:
[new_file.write(l + 'n') for l in file_2]

Answer 1

您需要在不将内容保存在内存中的情况下读取文件。您可以在输入文件上使用with

with open(r'C:UsersGuy.SADesktopfileB.txt', 'r') as file_2, open(r'C:UsersGuy.SADesktopfileC.txt', 'w') as new_file:
for line_2 in file_2:
with open(r'C:UsersGuy.SADesktopfileA.txt', 'r') as file_1:
for line_1 in file_1:
if line_1 == line_2:
break
else:
new_file.write(line_2)

Answer 2

您应该为此使用文件对象：

with open('file1.txt', 'r', encoding='utf8') as file_1,
open('file2.txt', 'r', encoding='utf8') as file_2:
for line in file1: # or file 2
# Do what you need to do with reading it line by line

还要注意：

with将在之后自动关闭文件。