单独的数据文件负号和空白分隔符

Question

我正在尝试将空白"one_answers"-"分隔为列分隔符。我的文件有一个错误，即不一致地被一个空格分隔，例如：

8.55500000  42.93079187 -99.98428964  -0.59917942  20.86164814   8.37369433   0.56431509
8.55600000  42.94500503-100.05470144  -0.55062999  20.86380446   8.38865674   0.56429834
8.55700000  42.99565203-100.11651750  -0.54444340  20.87003752   8.39975047   0.55109542
8.55800000  42.99873154-100.07383720  -0.54648262  20.85777962   8.41246904   0.55645774

Answer 1

这是sep的一个更复杂的用法，所以这就是解释。不能仅在某些情况下将分隔符作为列的一部分，因此这一次代码实际上将分隔符保留为列。这被定义为可选的-符号，后面跟着连续的数字。这种方法将解决问题，但是它将创建多个nan列(这些列已删除(。如果文件的列和行很大，这可能会导致内存问题。

from io import StringIO
S = '''
8.500000  42.93079187 -99.98428964  -0.59917942  20.86164814   8.37369433   0.56431509
8.55600000  42.94500503-100.05470144  -0.55062999  20.86380446   8.38865674   0.56429834
8.55700000  42.99565203-100.11651750  -0.54444340  20.87003752   8.39975047   0.55109542
8.55800000  42.99873154-100.07383720  -0.54648262  20.85777962   8.41246904   0.55645774'''
df = pd.read_csv(StringIO(S),
sep='s*(-?[0-9.]+)',
engine='python', header=None).dropna(axis=1)
df.head()
#   1       3           5           7           9           11          13
# 0 8.500   42.930792   -99.984290  -0.599179   20.861648   8.373694    0.564315
# 1 8.556   42.945005   -100.054701 -0.550630   20.863804   8.388657    0.564298
# 2 8.557   42.995652   -100.116518 -0.544443   20.870038   8.399750    0.551095
# 3 8.558   42.998732   -100.073837 -0.546483   20.857780   8.412469    0.556458

Answer 2

如果所有文件数据都是这种简单的格式，那么这种方法可以有效地生成panda可以用来构建数据帧的行数据。

>>> import re
>>> 
>>> float_expr = re.compile(r"-?d*.?d+")
>>> 
>>> def gen_file_data(f):
...     for line in f:
...         line_data = float_expr.findall(line)
...         yield (float(v) for v in line_data)
...         
>>> df = pd.DataFrame.from_records(gen_file_data(open('filedata.txt', 'r')))
>>> 
>>> df
0          1           2         3          4         5         6
0  8.555  42.930792  -99.984290 -0.599179  20.861648  8.373694  0.564315
1  8.556  42.945005 -100.054701 -0.550630  20.863804  8.388657  0.564298
2  8.557  42.995652 -100.116518 -0.544443  20.870038  8.399750  0.551095
3  8.558  42.998732 -100.073837 -0.546483  20.857780  8.412469  0.556458
>>> 

标题行？

>>> def gen_file_data(f):
...     yield next(f).split()  # Header row?
...     for line in f:
...         line_data = float_expr.findall(line)
...         yield (float(v) for v in line_data)
...         
>>> g = gen_file_data(open("filedata.txt", 'r'))
>>> 
>>> df = pd.DataFrame.from_records(g, columns=next(g))
>>> df
foo        bar         baz       qux       quux      quuz     corge
0  8.555  42.930792  -99.984290 -0.599179  20.861648  8.373694  0.564315
1  8.556  42.945005 -100.054701 -0.550630  20.863804  8.388657  0.564298
2  8.557  42.995652 -100.116518 -0.544443  20.870038  8.399750  0.551095
3  8.558  42.998732 -100.073837 -0.546483  20.857780  8.412469  0.556458
>>>

生成器假设标题行由标题组成，这些标题是由空格分隔的连续字符。如果是其他模式，则可以更新生成器的第一行来处理它