如何在不爆炸内存的情况下加载TMX文件？
在ParaCrawl数据集中加载时，我希望使用TMX格式。为了加载TMX文件，我一直在使用这个[SO post](python中的TMX(Translation Memory eXchange(文件(
然而，对于瑞典语到英语之类的东西，简单地加载数据集就需要66GB的内存，而根据数据集的大小，我预计从西班牙语到英语加载几乎需要500GB的内存。最大的AWS实例是128GB内存，看起来我做错了什么？
将XML解析器与流一起使用似乎效果很好。TMX文件<tmx><body><tu>中有三个级别。
import xmltodict
from gzip import GzipFile
language_pairs = []
def get_lang_pair(_, tree):
lang_pair = {}
for elem in tree['tuv']:
language = elem['@xml:lang']
text = elem['seg']
lang_pair[language] = text
language_pairs.append(lang_pair)
return True
# <tmx>
# <body>
# <tu>
xmltodict.parse(
GzipFile('en-da.tmx.gz'),
item_depth=3, item_callback=get_lang_pair,
)
输出：
[{'en': 'Galaxy S9 price: $799.99 full price or $33.33/month for 24 months with 0 percent APR Galaxy S9 Plus price: $929.99 full price or $38.74/month for 24 months with 0 percent APR Pre-order launch date: 1 March, 9:01 p.m Pacific Launch date: 16 March Trade-in promo: Up to $350 credit (iPhone X, 8, 8 Plus; Samsung GS8, GS8+, Note8) Other promos: $150 prepaid MasterCard for porting in a number, $10 off accessories for Verizon Up customers, 50 percent off Samsung tablet with 2-year agreement, $100 off Gear S3, and Samsung fast charge power bundle for $99.99',
'da': 'Galaxy S9 pris: $799.99 fuld pris eller $33.33/måned i 24 måneder med 0% p.a Galaxy S9 Plus pris: $929.99 fuld pris eller $38.74/måned i 24 måneder med 0% p.a Forud for lanceringen dato: 1 Marts, 9:01 p.m Pacific Lanceringen dato: 16 Marts Handel-promo: Op til $350 kredit (iPhone X, 8, 8, Plus, Samsung GS8, GS8+, Note8) Andre promotions: $150 prepaid MasterCard for portering af et nummer, $10 off tilbehør til Verizon kunder Op, 50% off Samsung tablet med 2-årig aftale, $100 off Gear S3, og Samsung hurtig opladning power bundle for $99.99'},
{'en': 'Room facilities: Air conditioning, Bath or Shower, Cable Channels, Desk, Dining table, Entire unit located on ground floor, Fan, Flat-screen TV, Free toiletries, Heating, Linen, Microwave, Private bathroom, Private entrance, Refrigerator, Seating Area, Shower, TV, Tile/Marble floor, Toilet, Toilet paper, Towels Max persons: 3 Size of the room: 13.94 m² Beds: 1x Large bed(s) (King size) bed Rate: US$ 55.71 ...',
'da': 'Værelses faciliteter: Aircondition, Badekar eller bruser, Bruser, Egen indgang, Eget badeværelse, Fjernsyn, Fladskærms-tv, Flise-/marmorgulv, Gratis toiletartikler, Hele boligen er beliggende i stueetagen, Håndklæder, Kabelkanaler, Køleskab, Loftvifte, Mikrobølgeovn, Sengelinned, Siddekrog, Skrivebord, Spisebord, Toilet, Toiletpapir , Varme Max antal personer: 3 Størrelsen af u200bu200brummet: 13.94 m² Senge: 1x Large bed(s) (King size) bed Pris: US$ 55.71 ...'},