在python并行处理中，如何找到第一个进程调用的函数?

Question

我有以下代码片段，它读取CSV文件列表并将它们合并到一个csv中。

import multiprocessing
def do():
pool = multiprocessing.Pool(max_threads)
list_of_csvs=[]
outputdir = 'output/'
for csvFile in glob(outputdir + '*.csv'):
list_of_csvs.append(csvFile)
pool.map(writeToSingleCSV, list_of_csvs)
pool.close()
def writeToSingleCSV(csvFile):
with open('singleDataFile.csv', 'a') as singleFile:
inFile = open(csvFile, 'r')
for line in inFile:
singleFile.write(line)

上面的代码有效，但我想跳过以下 CSV 文件的标题。(因为所有 CSV 文件都包含相同的标头(如何从第二个文件中跳过标头？

Answer 1

另一种方法：使用 Pandas 可能会有所帮助ignore_index=True可以解决标头问题

import pandas as pd
import numpy as np
import glob
all_data = pd.DataFrame()
for f in glob.glob("*.xlsx"): #read all xlsx file from a folder
df = pd.read_excel(f)
all_data = all_data.append(df,ignore_index=True)
print (all_data.describe())
all_data.to_excel('SingleFile.xlsx')

Answer 2

你为什么不单独编写标题？像这样的东西

import multiprocessing
def do():
pool = multiprocessing.Pool(max_threads)
list_of_csvs=[]
outputdir = 'output/'
for csvFile in glob(outputdir + '*.csv'):
list_of_csvs.append(csvFile)
writeToHEADERCSV(list_of_csvs[0])
pool.map(writeToSingleCSV, list_of_csvs)
pool.close()
def writeToHEADERCSV(csvFile):
with open('singleDataFile.csv', 'a') as singleFile:
inFile = open(csvFile, 'r')
# Get the first line and write it on the file 
def writeToSingleCSV(csvFile):
with open('singleDataFile.csv', 'a') as singleFile:
inFile = open(csvFile, 'r')
for line in inFile:
# skip the first line which is header

Answer 3

我只会在执行映射到writeToSingleCSV之前附加标题，并使writeToSingleCSV默认忽略标头。

import multiprocessing
def do():
pool = multiprocessing.Pool(max_threads)
list_of_csvs=[]
outputdir = 'output/'
for csvFile in glob(outputdir + '*.csv'):
list_of_csvs.append(csvFile)
#Write a CSV file with the header
csv_with_header = list_of_csvs.pop()
writeToSingleCSV(csv_with_header, ignoreHeader=False)
#Write the following CSV files without the header
pool.map(writeToSingleCSV, list_of_csvs)
pool.close()
def writeToSingleCSV(csvFile, ignoreHeader=True):
with open('singleDataFile.csv', 'a') as singleFile:
inFile = open(csvFile, 'r')
if ignoreHeader:
#Ignore/Remove header from inFile - I would ignore len(header) characters
for line in inFile:
singleFile.write(line)

这使它保持简单，明确，并且应该易于实现。