Nokogiri::XML::Reader语言 - 处理大型XML文件并跳过不感兴趣的节点

Question

我有一些格式如下的xml，我试图使用Nokogiri：：XML：：Reader解析，因为文件大小非常大(~1GB(。该文件具有以下格式的许多packets。

从每个packet我需要收集frame.time_epoch，s1ap.procedureCode。

我目前正在执行以下操作。

data = []
file = `some_file.xml`
reader = Nokogiri::XML::Reader(File.open(file))
reader.each do |node|
    if (node.name == 'packet' && node.node_type == Nokogiri::XML::Reader::TYPE_ELEMENT)
      doc = Nokogiri::XML(node.outer_xml)
      next if !doc.css("field[name='s1ap.procedureCode']") ## do nothing if the <packet> is not of s1ap type
      epochTime = doc.css("field[name='frame.time_epoch']").first["show"].to_i
      procedureCode = procedureCode_node = doc.css("field[name='s1ap.procedureCode']").first["show"].to_i
      data << { epochTime: epochTime, procedureCode: procedureCode }
    end
end

问题

我面临的挑战是解析非常慢。我注意到的一件事是，阅读器扫描<packet> </packet>中的所有后续行 - 有没有办法让读者移动到下一个节点，名称为packet，而不是进一步浏览packet内的每一行。

XML格式

<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet type="text/xsl" href="pdml2html.xsl"?>
<packet>
  <proto name="geninfo" pos="0" showname="General information" size="126">
    <field name="num" pos="0" show="6" showname="Number" value="6" size="126"/>
  </proto>
  <proto name="frame" showname="Frame 6: 126 bytes on wire (1008 bits), 126 bytes captured (1008 bits) on interface 0" size="126" pos="0">
    <field name="frame.encap_type" showname="Encapsulation type: Ethernet (1)" size="0" pos="0" show="1"/>
    <field name="frame.time_epoch" showname="Epoch Time: 1474267259.184197000 seconds" size="0" pos="0" show="1474267259.184197000"/>
  </proto>
  <proto name="eth" showname="Ethernet II, Src: JuniperN_e6:a6:cc (40:b4:f0:e6:a6:cc), Dst: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="14" pos="0">
    <field name="eth.dst" showname="Destination: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="6" pos="0" show="ac:16:2d:89:a5:91" value="ac162d89a591">
      <field name="eth.dst_resolved" showname="Destination (resolved): HewlettP_89:a5:91" hide="yes" size="6" pos="0" show="HewlettP_89:a5:91" value="ac162d89a591"/>
    </field>
    <field name="eth.type" showname="Type: IPv4 (0x0800)" size="2" pos="12" show="0x00000800" value="0800"/>
  </proto>  
  <proto name="s1ap" showname="S1 Application Protocol" size="45" pos="78">
    <field name="per.choice_index" showname="Choice Index: 0" hide="yes" size="1" pos="78" show="0" value="00"/>
    <field name="s1ap.S1AP_PDU" showname="S1AP-PDU: initiatingMessage (0)" size="45" pos="78" show="0" value="000b402900000300000005c007c03ae900080003403b53001a0012113743f99f9500075d010605f070c04070c1">
      <field name="s1ap.initiatingMessage_element" showname="initiatingMessage" size="45" pos="78" show="" value="">
        <field name="s1ap.procedureCode" showname="procedureCode: id-downlinkNASTransport (11)" size="1" pos="79" show="11" value="0b"/>
       </field>
    </field>
  </proto>
</packet>
<packet>
  <proto name="geninfo" pos="0" showname="General information" size="126">
    <field name="num" pos="0" show="6" showname="Number" value="6" size="126"/>
  </proto>
  <proto name="frame" showname="Frame 6: 126 bytes on wire (1008 bits), 126 bytes captured (1008 bits) on interface 0" size="126" pos="0">
    <field name="frame.encap_type" showname="Encapsulation type: Ethernet (1)" size="0" pos="0" show="1"/>
    <field name="frame.time_epoch" showname="Epoch Time: 1474267260.184197000 seconds" size="0" pos="0" show="1474267259.184197000"/>
  </proto>
  <proto name="eth" showname="Ethernet II, Src: JuniperN_e6:a6:cc (40:b4:f0:e6:a6:cc), Dst: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="14" pos="0">
    <field name="eth.dst" showname="Destination: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="6" pos="0" show="ac:16:2d:89:a5:91" value="ac162d89a591">
      <field name="eth.dst_resolved" showname="Destination (resolved): HewlettP_89:a5:91" hide="yes" size="6" pos="0" show="HewlettP_89:a5:91" value="ac162d89a591"/>
    </field>
    <field name="eth.type" showname="Type: IPv4 (0x0800)" size="2" pos="12" show="0x00000800" value="0800"/>
  </proto>  
  <proto name="s1ap" showname="Some other protocol" size="45" pos="78">
    <field name="per.choice_index" showname="Choice Index: 0" hide="yes" size="1" pos="78" show="0" value="00"/>
    <field name="other.OTH_PDU" showname="S1AP-PDU: initiatingMessage (0)" size="45" pos="78" show="0" value="000b402900000300000005c007c03ae900080003403b53001a0012113743f99f9500075d010605f070c04070c1">
      <field name="other.initiatingMessage_element" showname="initiatingMessage" size="45" pos="78" show="" value="">
        <field name="other.procedureCode" showname="procedureCode: id-someTransport (99)" size="1" pos="79" show="11" value="0b"/>
       </field>
    </field>
  </proto>
</packet>
<packet>
  <proto name="geninfo" pos="0" showname="General information" size="126">
    <field name="num" pos="0" show="6" showname="Number" value="6" size="126"/>
  </proto>
  <proto name="frame" showname="Frame 6: 126 bytes on wire (1008 bits), 126 bytes captured (1008 bits) on interface 0" size="126" pos="0">
    <field name="frame.encap_type" showname="Encapsulation type: Ethernet (1)" size="0" pos="0" show="1"/>
    <field name="frame.time_epoch" showname="Epoch Time: 1474267261.184197000 seconds" size="0" pos="0" show="1474267259.184197000"/>
  </proto>
  <proto name="eth" showname="Ethernet II, Src: JuniperN_e6:a6:cc (40:b4:f0:e6:a6:cc), Dst: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="14" pos="0">
    <field name="eth.dst" showname="Destination: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="6" pos="0" show="ac:16:2d:89:a5:91" value="ac162d89a591">
      <field name="eth.dst_resolved" showname="Destination (resolved): HewlettP_89:a5:91" hide="yes" size="6" pos="0" show="HewlettP_89:a5:91" value="ac162d89a591"/>
    </field>
    <field name="eth.type" showname="Type: IPv4 (0x0800)" size="2" pos="12" show="0x00000800" value="0800"/>
  </proto>  
  <proto name="s1ap" showname="S1 Application Protocol" size="45" pos="78">
    <field name="per.choice_index" showname="Choice Index: 0" hide="yes" size="1" pos="78" show="0" value="00"/>
    <field name="s1ap.S1AP_PDU" showname="S1AP-PDU: initiatingMessage (0)" size="45" pos="78" show="0" value="000b402900000300000005c007c03ae900080003403b53001a0012113743f99f9500075d010605f070c04070c1">
      <field name="s1ap.initiatingMessage_element" showname="initiatingMessage" size="45" pos="78" show="" value="">
        <field name="s1ap.procedureCode" showname="procedureCode: id-uplinkTransport (13)" size="1" pos="79" show="13" value="0b"/>
       </field>
    </field>
  </proto>
</packet>
<!-- more <packet>s here -->

Answer 1

对于如此庞大的文档，您应该使用 SAX 解析器

http://www.rubydoc.info/github/sparklemotion/nokogiri/Nokogiri/XML/SAX

最好是流处理大型文档，而无需将整个文档拉入内存并解析为 DOM。特别是考虑到问题只需要一次通过。

下面是通过使用 SAX 流式传输 XML 来完成任务的代码：

require 'nokogiri'
class PacketFilter < Nokogiri::XML::SAX::Document
  def initialize
    reset
  end
  def end_document
    puts 'the document has ended'
  end
  def start_element(name, attributes = [])
    case name
    when 'packet'
      @in_packet = true
    when 'proto'
      @have_s1ap = @in_packet && attribute_value(attributes, 'name') == 's1ap'
    when 'field'
      case attribute_value(attributes, 'name')
      when 's1ap.procedureCode'
        @procedure_code = attribute_value(attributes, 'showname')
      when 'frame.time_epoch'
        @epoch_time = attribute_value(attributes, 'showname')
      end
    end
  end
  def end_element(name)
    if name == 'packet'
      puts "#{@procedure_code}, #{@epoch_time}" if @have_s1ap
      reset
    end
  end
  private
  def attribute_value(attributes, name)
    attributes.reduce(nil) do |value, assoc|
      assoc[0] == name ? assoc[1] : value
    end
  end
  def reset
    @in_packet = false
    @have_s1ap = false
    @procedure_code = nil
    @epoch_time = nil
  end
end
parser = Nokogiri::XML::SAX::Parser.new(PacketFilter.new)
parser.parse($stdin)

如果将数据示例粘贴到 data.xml 中，并将上述 ruby 粘贴到slap.rb中：

$ cat data.xml | ruby poke.rb
procedureCode: id-downlinkNASTransport (11), Epoch Time: 1474267259.184197000 seconds
the document has ended

Answer 2

您可以只循环遍历packet元素，然后跳过任何不符合您条件的元素，而不是遍历每个节点。这将只执行packet元素而不是所有元素，这应该会快得多。

data = []
file = 'some_file.xml'
doc = Nokogiri::XML.fragment(File.read(file)) # use `read` instead of `open`
doc.xpath('packet').each do |packet|
    next if !packet.css("field[name='s1ap.procedureCode']") ## do nothing if the <packet> is not of s1ap type
    epochTime = packet.css("field[name='frame.time_epoch']").first["show"].to_i
    procedureCode = procedureCode_node = packet.css("field[name='s1ap.procedureCode']").first["show"].to_i
    data << { epochTime: epochTime, procedureCode: procedureCode }
end 
» data
=> [{:epochTime=>1474267259, :procedureCode=>11}