读取 ASCII 文件，在 Python 中具有不常见的格式

Question

我从未使用过从 ASCII 文件导入数据，我注意到不同的 ASCII 文件有不同的格式，因此试图找到适用于任何格式的通用解决方案对我来说具有挑战性。

我有一个.dat(ASCII( 文件，我需要读取并提取变量(请参阅问题底部的 txt 片段(。下面是我试图弄清楚如何读取数据的不同尝试(用###分隔(的代码。

f_41 = open(fileRS41, 'r')
data_41 = f_41.read()
for line in data_41:
print(repr(line))
data_41.close()
############################
f = open(fileRS41, 'r')
# Read and ignore header lines
header1 = f.readline()
header2 = f.readline()
header3 = f.readline()
# Loop over lines and extract variables of interest
for line in f:
line = line.strip()
columns = line.split()
name = columns[1] # Not sure what the different numbers do but this was code from another solution
j = float(columns[1]) # ERROR: string can't be converted to float
print(name, j)
f.close()
############################
from astropy.io import ascii
data = ascii.read(f_41, guess=False)  
print(data) 
############################
x = np.genfromtxt(f_41, dtype=None)

另一种选择是先将其转换为CSV文件，然后使用Pandas使用它。但是，当我进行转换时，变量名称将作为堆叠在一起的列导入，而不是每个各自的列一个变量名称。

# convert ASCII to CSV
f = open(file, 'r')
lines = f.readlines()
with open("FILEOUT.csv", 'w') as csvfile:
writer = csv.writer(csvfile)
for l in lines:
asdf = l.split()
writer.writerow(asdf)
print("out?")

.dat文件相关示例：

Generated by Rfunction:  Get.mw41.edt.func2 
============> Radisonde_info:
RS_type:        RS41-SGP
RS_config:      -32768
RS_serialnum:   R3340183
RS_freq:        403
RS__windtype:   ccGPS
=============> Station_info:
Station:        HUBV_RS41SGP
Latitude:       39.0563
Longitude:      -76.8755
Altitude:       52.3
SW version:     MW41 2.15.0
Start time:     2020-01-23 06:46:41
=============> Variables & units - Vaisala EDT
NA_numeric value:  -9999
NA_string:  xx or NA
-----------------------------
Variable       Unit
time        sec
xx         NA
Ta          K
RH          %
v(S->N)        m/s
u(E->W)        m/s
Height          m
press        hPa
Td          K
MR       g/Kg
DD        dgr
FF        m/s
Ascend_FLG  (0-N,1-Y)
xx         NA
xx         NA
Lon        dgr
Lat        dgr
xx         NA
xx         NA
xx         NA
=============> Data:
0.00  -9999.    268.37    85.00      0.00      0.00         52.3   1023.19    266.24     2.22       0.00      0.00 1  -9999.  -9999.   -76.8755    39.0563  -9999.  -9999.  -9999.
0.81  -9999.    268.46    83.38      0.46      0.86         54.5   1022.90    266.08     2.19     241.86      0.98 1  -9999.  -9999.   -76.8757    39.0564  -9999.  -9999.  -9999.

Answer 1

我找到了一个有效的解决方案，但我想避免硬编码列标题并能够直接从 ASCII 文件中读取它们：

# set the directory for data files
fileRS41 = 'filename.dat'
# load text from .dat files
f41 = np.loadtxt(fileRS41, skiprows = 40)
# create column names for variables
c = ['time' , 'xx0', 'temp', 'RH', 'v(S_N)', 'u(E_W)', 'height', 'pressure', 'Td', 'mixingratio', 'DD', 'FF', 'Ascend_FLG', 'xx1', 'xx2', 'lon', 'lat', 'xx3', 'xx4', 'xx5']
# skip columns when reading in file and converting to dataFrame
skip = ['xx0', 'DD', 'FF', 'Ascend_FLG', 'xx1', 'xx2', 'xx3', 'xx4', 'xx5']
# convert to Pandas Dataframe
df_f41 = pd.DataFrame.from_records(f41, exclude=skip, columns = c)