使用python读取一个大的文本文件比使用Matlab读取相同文本的相同代码慢得多，知道为什么吗

Question

我在Matlab中有以下代码用于读取文本文件文本文件具有XML格式，但我将其作为文本文件读取：

function [jointAngleData,PositionData, AccelerationData,OrientationData, 
AngularVelocityData,AngularAccelerationData,TimeStamps] = getDatafromMVNX 
(file,eliminate_samples)
fid=fopen (file);
currentline=fgetl(fid);
jointAngleData =[];
PositionData = [];
AccelerationData = [];
OrientationData = [];
AngularVelocityData = [];
AngularAccelerationData = [];
while ischar(currentline)
if (contains(currentline,'<jointAngle>'))  
[data,~]=strsplit(currentline,'<D*>','DelimiterType', 'RegularExpression');
currentlinedata = str2num(data{2}); %#ok<*ST2NM>
jointAngleData = [jointAngleData ; currentlinedata];  %#ok<*AGROW>
end
if (contains(currentline,'<position>'))
[data,~]=strsplit(currentline,'<D*>','DelimiterType', 'RegularExpression');
currentlinedata = str2num(data{2});
PositionData = [PositionData ; currentlinedata]; 
end
if (contains(currentline,'<acceleration>'))
[data,~]=strsplit(currentline,'<D*>','DelimiterType', 'RegularExpression');
currentlinedata = str2num(data{2});
AccelerationData = [AccelerationData ; currentlinedata]; 
end
if (contains(currentline,'<orientation>'))
[data,~]=strsplit(currentline,'<D*>','DelimiterType', 'RegularExpression');
currentlinedata = str2num(data{2});
OrientationData = [OrientationData ; currentlinedata]; 
end
if (contains(currentline,'<angularVelocity>'))
[data,~]=strsplit(currentline,'<D*>','DelimiterType', 'RegularExpression');
currentlinedata = str2num(data{2});
AngularVelocityData = [AngularVelocityData ; currentlinedata]; 
end
if (contains(currentline,'<angularAcceleration>'))
[data,~]=strsplit(currentline,'<D*>','DelimiterType', 'RegularExpression');
currentlinedata = str2num(data{2});
AngularAccelerationData = [AngularAccelerationData ; currentlinedata]; 
end
currentline=fgetl(fid);
end
Data_ends = size(jointAngleData,1)-eliminate_samples;
jointAngleData = jointAngleData(1:Data_ends,:);
AccelerationData = AccelerationData(1:Data_ends,:);
OrientationData = OrientationData(4:Data_ends+3,:);
PositionData = PositionData(4:Data_ends+3,:);
AngularVelocityData = AngularVelocityData(1:Data_ends,:);
AngularAccelerationData = AngularAccelerationData(1:Data_ends,:);
TimeStamps = size(OrientationData,1);
end

对于同样的任务，我用python写了一段代码：

def _read_feature_text(line):

start = line.find('>')+1
lend = line.find('</') 
workingportion = line[start:lend]
return pd.DataFrame([np.fromstring(workingportion,sep= ' ')])

def read_mvnx(mvnxfile):
from bs4 import BeautifulSoup
myfile  = open (mvnxfile,"r")
contents = myfile.read()
orientation = pd.DataFrame()
positions = pd.DataFrame()
velocities = pd.DataFrame()
accelerations = pd.DataFrame()
angularVelocities = pd.DataFrame()
angularAccelerations = pd.DataFrame()
jointAngles = pd.DataFrame()
with myfile:
wholefilecontent = myfile.readlines()
#line = myfile.readline()
start_time = timeit.default_timer()
for line in wholefilecontent:
if ('orientation' in line):
orientation = orientation.append(_read_feature_text(line),ignore_index = True)
elif ('position' in line):
positions = positions.append(_read_feature_text(line),ignore_index = True)
elif ('velocity' in line):
velocities = velocities.append(_read_feature_text(line),ignore_index = True)
elif ('acceleration' in line):
accelerations = accelerations.append(_read_feature_text(line),ignore_index = True)
elif ('angularVelocity' in line):
angularVelocities = angularVelocities.append(_read_feature_text(line),ignore_index = True)
elif ('angularAcceleration' in line):
angularAccelerations = angularAccelerations.append(_read_feature_text(line),ignore_index = True)
elif ('joinAngle' in line):
jointAngles = jointAngles.append(_read_feature_text(line),ignore_index = True)
elapsed = timeit.default_timer() -start_time
print(elapsed)

我甚至尝试使用正则表达式和BeautifulSoup包。两者都没有给我更好的时机。有什么建议吗？还有其他方法可以让它更快吗。我所说的更快是指比这个快得多。

Answer 1

对于我的代码，我发现它太慢的原因是在每行中找到数据后，我将其转换为数据帧，并将其附加到全局数据帧的末尾。这种转换使它变得非常慢。我只是把数据放在一个numpy数组中，然后在最后把整个numpy数组转换成一个数据帧。

我还使用xmltodic包来解析整个文件，而不是逐行解析。