如何从warc文件中读取记录的子集

Question

我正在尝试解析。warc文件从Common Crawl在Python.

由于文件很大，我想从前几条记录的样本/子集开始。

我如何截断文件的文件，只包括前X行，同时保留换行/回车的地方?

我已经试过了:

1. head -n 250 oldfile > newfile这删除了解析文件所需的一些返回。这是我得到的错误，如果我试图在我的Hadoop工作中使用这个文件(阅读它与warc包):

     Traceback (most recent call last):
         File "test.py", line 46, in <module>
           TagGrabber.run()
         File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/job.py", line 461, in run
           mr_job.execute()
         File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/job.py", line 479, in execute
           super(MRJob, self).execute()
         File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/launch.py", line 151, in execute
           self.run_job()
         File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/launch.py", line 214, in run_job
           runner.run()
         File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/runner.py", line 464, in run
           self._run()
         File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/sim.py", line 173, in _run
           self._invoke_step(step_num, 'mapper')
         File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/sim.py", line 264, in _invoke_step
           self.per_step_runner_finish(step_num)
         File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/local.py", line 152, in per_step_runner_finish
           self._wait_for_process(proc_dict, step_num)
         File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/mrjob/local.py", line 268, in _wait_for_process
           (proc_dict['args'], returncode, ''.join(tb_lines)))
       Exception: Command ['sh', '-ex', 'setup-wrapper.sh', '/var/cc-mrjob/venv/bin/python', 'test.py', '--step-num=0', '--mapper', '/tmp/test.root.20150520.071726.549519/input_part-00000'] returned non-zero exit status 1:
       Traceback (most recent call last):
         File "test.py", line 46, in <module>
           TagGrabber.run()
         File "/tmp/test.root.20150520.071726.549519/job_local_dir/0/mapper/0/mrjob.tar.gz/mrjob/job.py", line 461, in run
           mr_job.execute()
         File "/tmp/test.root.20150520.071726.549519/job_local_dir/0/mapper/0/mrjob.tar.gz/mrjob/job.py", line 470, in execute
           self.run_mapper(self.options.step_num)
         File "/tmp/test.root.20150520.071726.549519/job_local_dir/0/mapper/0/mrjob.tar.gz/mrjob/job.py", line 535, in run_mapper
           for out_key, out_value in mapper(key, value) or ():
         File "/var/cc-mrjob/mrcc.py", line 33, in mapper
           for i, record in enumerate(f):
         File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/warc/warc.py", line 390, in __iter__
           record = self.read_record()
         File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/warc/warc.py", line 373, in read_record
           header = self.read_header(fileobj)
         File "/var/cc-mrjob/venv/local/lib/python2.7/site-packages/warc/warc.py", line 331, in read_header
           raise IOError("Bad version line: %r" % version_line)
       IOError: Bad version line: 'WARC/1.0n'
   

2. 与#1相同，但使用tail命令

3. 与#1相同，但使用tr或sed后替换任何丢失的换行符或^M(回车)字符。这导致warc包仍然抱怨预期的回车或换行符没有到位。
4. unix2dos oldfile

Answer 1

正确处理换行是很困难的，因为.warc文件也可能包含二进制数据。截断还可能产生破碎的.warc文件，因为python库相信Content-Length头文件是有效的。

warc python库每次只从.warc文件中读取一条记录(避免一次将整个文件读取到内存中)，因此可以仅使用python处理子集。例如:

import warc
from itertools import islice
N = 10
warc_file = warc.open('/path/to/file.warc')
for record in islice(warc_file, N):
    do_stuff_with(record)