如何导入多个csv文件并使用panda连接到一个DataFrame中

Question

我有问题No objects to concatenate。我无法从主目录及其子目录导入.csv文件以将它们连接到一个DataFrame中。我在用熊猫。旧的答案对我没有帮助，所以请不要标记为重复。

文件夹结构类似

main/*.csv
main/name1/name1/*.csv
main/name1/name2/*.csv
main/name2/name1/*.csv
main/name3/*.csv

import pandas as pd
import os
import glob
folder_selected = 'C:/Users/jacob/Documents/csv_files'

1. 不起作用

frame = pd.concat(map(pd.read_csv, glob.iglob(os.path.join(folder_selected, "/*.csv"))))

2. 不起作用

csv_paths = glob.glob('*.csv')
dfs = [pd.read_csv(folder_selected) for folder_selected in csv_paths]
df = pd.concat(dfs)

3. 不起作用

all_files = []

all_files = glob.glob (folder_selected + "/*.csv")

file_path = []
for file in all_files:
df = pd.read_csv(file, index_col=None, header=0)
file_path.append(df)

frame = pd.concat(file_path, axis=0, ignore_index=False)

Answer 1

您需要递归地搜索子目录。

folder = 'C:/Users/jacob/Documents/csv_files'
path = folder+"/**/*.csv"

1. 使用glob.iglob

df = pd.concat(map(pd.read_csv, glob.iglob(path, recursive=True)))

2. 使用glob.glob

csv_paths = glob.glob(path, recursive=True)
dfs = [pd.read_csv(csv_path) for csv_path in csv_paths]
df = pd.concat(dfs)

3. 使用os.walk

file_paths = []
for base, dirs, files in os.walk(folder):
for file in fnmatch.filter(files, '*.csv'):
file_paths.append(os.path.join(base, file))
df = pd.concat([pd.read_csv(file) for file in file_paths])

4. 使用pathlib

from pathlib import Path
files = Path(folder).rglob('*.csv')
df = pd.concat(map(pd.read_csv, files))

Answer 2

如下检查Dask库，该库将多个文件读取到一个df

>>> import dask.dataframe as dd
>>> df = dd.read_csv('data*.csv')

阅读他们的文档https://examples.dask.org/dataframes/01-data-access.html#Read-CSV文件

Answer 3

Python的pathlib是用于此类任务的工具

from pathlib import Path
FOLDER_SELECTED = 'C:/Users/jacob/Documents/csv_files'
path = Path(FOLDER_SELECTED) / Path("main")
# grab all csvs in main and subfolders
df = pd.concat(pd.read_csv(f.name) for f in path.rglob("*.csv"))

注：

如果CSV需要预关联，您可以创建一个read_CSV函数来处理问题，并将其放置在pd.read_CSV 的位置