有没有一种方法可以读取和更改PyCharm中一个巨大csv文件的内容

Question

我目前正试图创建一个程序，该程序可以读取csv，确定每行的某一列中是否包含子字符串，如果不存在，则将某些列重写为新的csv。我已经记下了这么多代码，但我需要使用该程序的csv有300多万行。我使用PyCharm，目前无法处理这么多数据。它只能以只读格式查看csv，这不允许我使用它。我知道pandas有一个块大小的功能，但我不知道如何用我的其他代码实现它。

def reading(csv_input):
originalLength = 0
rowCount = 0
with open(f'Web Report {csv_input}', 'w') as file:
writer = csv.writer(file)
writer.writerow(['Index', 'URL Category', 'User IP', 'URL'])
dropCount = 0
data = pd.read_csv(csv_input, chunksize=100000)
df = pd.DataFrame(data,
columns=['Line', 'Date', 'Hour', 'User Name', 'User IP', 'Site Name',
'URL Category', 'Action', 'Action Description'])
originalLength = len(df.index)
for line in range(originalLength):
dataLine = df.loc[line]
x = dataLine.get(key='Action')
if x == 0:
siteName = dataLine.get(key='Site Name')
if 'dbk' in siteName:
dropCount = dropCount + 1
elif 'ptc' in siteName:
dropCount = dropCount + 1
elif 'wcf' in siteName:
dropCount = dropCount + 1
elif 'google' in siteName:
dropCount = dropCount + 1
else:
writer.writerow([line,  # Original Index
df.loc[line].get(key='URL Category'),  # Original URL Category
df.loc[line].get(key='User IP'),  # Original User IP
df.loc[line].get(key='Site Name')])  # Original Site Name
rowCount = rowCount + 1
else:
dropCount = dropCount + 1
file.close()
print("Input: " + str(csv_input))
print("Output: " + str(file.name))
print("Original Length: " + str(originalLength))
print("Current Length: " + str(rowCount))
print("Drop Count: " + str(dropCount) + "n")
return df

Answer 1

如果使用csv写入文件，那么也可以使用它逐行读取。

import csv
with open('input.csv') as infile, open('output.csv', 'w') as outfile:
csv_reader = csv.reader(infile)
csv_writer = csv.writer(outfile)

# copy headers
headers = next(csv_reader)
csv_writer.writerow(headers)

# process rows
for row in csv_reader:  # read row by row
# keep only rows with even index
if int(row[0]) % 2 == 0:
print('--- row ---')
print(row)
csv_writer.writerow(row)

---

如果要将pandas与chunk一起使用，则应使用for-循环
当您使用panda编写时，您需要不带标题的append模式。

import pandas as pd
first = True
for df in pd.read_csv('input.csv', chunksize=1):  # read row by row
# keep only rows with even index
if df.index % 2 == 0:
print('--- row ---')
print(df)
if first:
# create new file with headers
df.to_csv('output.csv', mode='w')
first = False
else:
# append to existing file without headers
df.to_csv('output.csv', mode='a', header=False)

---

最小工作代码

import pandas as pd
import csv
# --- create some data ---
data = {
'A': range(0,10), 
'B': range(10,20),
'C': range(20,30),
} # columns
df = pd.DataFrame(data)
df.to_csv('input.csv', index=False)
# --- read and write with `pandas` ---
first = True
for df in pd.read_csv('input.csv', chunksize=1):  # read row by row
# keep only rows with even index
if df.index % 2 == 0:
print('--- row ---')
print(df)
if first:
# create empty with headers
df.to_csv('output_pandas.csv', mode='w')
first = False
else:
# append to existing file without headers
df.to_csv('output_pandas.csv', mode='a', header=False)

# --- read and write with `csv` ---

with open('input.csv') as infile, open('output.csv', 'w') as outfile:
csv_reader = csv.reader(infile)
csv_writer = csv.writer(outfile)

# copy headers
headers = next(csv_reader)
csv_writer.writerow(headers)

# process rows
for row in csv_reader:
# keep only rows with even index
if int(row[0]) % 2 == 0:
print('--- row ---')
print(row)
csv_writer.writerow(row)

---

Doc:read_csv((，to_csv