在python中使用panda加载.csv文件时出错



我有一个大尺寸的csv文件,大约6gb,加载到python上需要很多时间。我得到以下错误:

import pandas as pd
df = pd.read_csv('nyc311.csv', low_memory=False)

Python(1284,0x7fffa37773c0) malloc: *** mach_vm_map(size=18446744071562067968) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/io/parsers.py", line 646, in parser_f
return _read(filepath_or_buffer, kwds)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/io/parsers.py", line 401, in _read
data = parser.read()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/io/parsers.py", line 939, in read
ret = self._engine.read(nrows)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/io/parsers.py", line 1508, in read
data = self._reader.read(nrows)
File "pandas/parser.pyx", line 851, in pandas.parser.TextReader.read (pandas/parser.c:10438)
File "pandas/parser.pyx", line 939, in pandas.parser.TextReader._read_rows (pandas/parser.c:11607)
File "pandas/parser.pyx", line 2024, in pandas.parser.raise_parser_error (pandas/parser.c:27037)
pandas.io.common.CParserError: Error tokenizing data. C error: out of memory

我不认为我理解错误代码,最后一行似乎表明文件太大,无法加载?我也尝试了low_memory=FALSE选项,但这也不起作用。

我不知道"不能分配区域"是什么意思,是否可能标题中包含"区域",Panda无法定位下面的列?

RAM导致内存不足。对此没有其他解释。

内存中对象的所有数据内存开销的总和<RAM

malloc: *** mach_vm_map(size=18446744071562067968) failed您可以从这个错误语句中清楚地理解。

尝试使用。

df = pd.read_csv('nyc311.csv',chunksize =5000,lineterminator='r')

或者,如果读取此csv只是程序的一部分,并且之前创建了任何其他数据帧,请尝试在不使用时清理它们。

import gc
del old_df              #clear dataframes not in use
gc.collect()        # collect Garbage 
del gc.garbage[:]   # Clears RAM

`

最新更新