<p> 用美丽汤提取嵌套标签的内容

Question

我很难将漂亮汤的优点用于我的用例。有许多相似但并不总是相等的嵌套p标记，我想从中获取内容。示例如下：

<p><span class="example" data-location="1:20">20</span>normal string</p>
<p><span class="example" data-location="1:21">21</span>this text <strong>belongs together</strong></p>
<p><span class="example" data-location="1:22">22</span>some text (<span class="referencequote">a reference text</span>)that might continue</p>
<p><span class="example" data-location="1:23">23</span>more text</p><div class="linebreak"></div>
<p><span class="example" data-location="1:22">24</span>text with (<span class="referencequote">first</span>)two references <span class="referencequote">first</span>.</p>

我需要保存span标记的字符串以及p标记内的字符串，无论其样式如何，如果适用，还需要保存referencequote。因此，从上面的例子中，我想提取：

example = 20, text = 'normal string', reference = []
example = 21, text = 'this text belongs together', reference = []
example = 22, text = 'some text that might continue', reference = ['a reference text']
example = 23, text = 'more text', reference = []
example = 24, text = 'text with two references', reference = ['first', 'second']

我尝试的是用"example"类收集所有项目，然后循环遍历它的父级内容。

for span in bs.find_all("span", {"class": "example"}):
references = []
for item in span.parent.contents:
if (type(item) == NavigableString):
text= item
elif (item['class'][0]) == 'verse':
number= int(item.string)
elif (item['class']) == 'referencequote':
references.append(item.string)
else:
#how to handle <strong> tags?
verses.append(MyClassObject(n=number, t=text, r=references))

我的方法很容易出错，可能还有更多像<strong>、<em>这样的标签我现在忽略了。不幸的是，get_text((方法返回了类似"22sometext一个可能继续的引用文本"的内容。

必须有一种优雅的方法来提取这些信息。你能给我一些其他方法的想法吗？提前感谢！

Answer 1

试试这个。

from simplified_scrapy.core.regex_helper import replaceReg
from simplified_scrapy import SimplifiedDoc,utils
html = '''
<p><span class="example" data-location="1:20">20</span>normal string</p>
<p><span class="example" data-location="1:21">21</span>this text <strong>belongs together</strong></p>
<p><span class="example" data-location="1:22">22</span>some text (<span class="referencequote">a reference text</span>)that might continue</p>
<p><span class="example" data-location="1:23">23</span>more text</p><div class="linebreak"></div>
<p><span class="example" data-location="1:22">24</span>text with (<span class="referencequote">first</span>)two references <span class="referencequote">second</span>.</p>
'''
html = replaceReg(html,"<[/]*strong>","") # Pretreatment
doc = SimplifiedDoc(html)
ps = doc.ps
for p in ps:
text = ''.join(p.spans.nextText())
text = replaceReg(text,"[()]+","") # Remove ()
span = p.span # Get first span
spans = span.getNexts(tag="span").text # Get references
print (span["class"], span.text, text, spans)

结果：

example 20 normal string []
example 21 this text belongs together []
example 22 some text that might continue ['a reference text']
example 23 more text []
example 24 text with two references. ['first', 'second']

下面是更多的例子。https://github.com/yiyedata/simplified-scrapy-demo/tree/master/doc_examples

Answer 2

我发现了一种不同的方法-没有正则表达式，可能对可能出现的不同跨度更稳健

for s in bsItem.select('span'):
if s['class'][0] == 'example' :
# do whatever needed with the content of this span 
s.extract()
elif s['class'][0] == 'referencequote':
# do whatever needed with the content of this span 
s.extract()
# check for all spans with a class where you want the text excluded
# finally get all the text
text = span.parent.text.replace(' ()', '')

读到这篇文章的人可能会对这种方法感兴趣：(