Python用美丽的套件嵌套了HTML标签

我正在尝试从嵌套的HTML代码中获取所有HREF URL：

...
<li class="dropdown">
<a href="#" class="dropdown-toggle wide-nav-link" data-toggle="dropdown">TEXT_1 <b class="caret"></b></a>
<ul class="dropdown-menu">
<li class="class_A"><a title="Title_1" href="http://www.customurl_1.com">Title_1</a></li>
<li class="class_B"><a title="Title_2" href="http://www.customurl_2.com">Title_2</a></li>
...
<li class="class_A"><a title="Title_X" href="http://www.customurl_X.com">Title_X</a></li>
</ul>
</li>
...
<li class="dropdown">
<a href="#" class="dropdown-toggle wide-nav-link" data-toggle="dropdown">TEXT_2 <b class="caret"></b></a>
<ul class="dropdown-menu">
<li class="class_A"><a title="Title_1" href="http://www.customurl_1.com">Title_1</a></li>
<li class="class_B"><a title="Title_2" href="http://www.customurl_2.com">Title_2</a></li>
...
<li class="class_A"><a title="Title_X" href="http://www.customurl_X.com">Title_X</a></li>
</ul>
</li>
...

在原始的HTML代码中，大约有15个" LI"块，具有"下拉"，但是我只想从text = text_1的块中获取URL。可以用美丽的小组抓住所有这些嵌套的URL？

感谢您的帮助

lxml和xpath的示例：

from lxml import etree
from io import StringIO
parser = etree.HTMLParser()
tree   = etree.parse(StringIO(html), parser)
hrefs = tree.xpath('//li[@class="dropdown" and a[starts-with(.,"TEXT_1")]]/ul[@class="dropdown-menu"]/li/a/@href')
print hrefs

其中html是带有HTML内容的Unicode字符串。结果：

['http://www.customurl_1.com', 'http://www.customurl_2.com', 'http://www.customurl_X.com']

注意：我在XPath查询中使用starts-with函数更精确，但是如果TEXT_1并不总是在文本节点的开始。

，您可以以相同的方式使用contains。

查询详细信息：

//              # anywhere in the domtree
li              # a li tag with the following conditions:
[                               # (opening condition bracket for li)
    @class="dropdown"           # li has a class attribute equal to "dropdown" 
  and                           # and
    a                           # a child tag "a"
    [                           # (open a condition for "a")
        starts-with(.,"TEXT_1") # that the text starts with "TEXT_1"
    ]                           # (close a condition for "a")
]                               # (close the condition for li)
/                            # li's child (/ stands for immediate descendant)
ul[@class="dropdown-menu"]   # "ul" with class equal to "dropdown-menu"
/li                          # "li" children of "ul"
/a                           # "a" children of "li"
/@href                       # href attributes children of "a"

虽然不如xpath那样优雅，但您始终可以使用日常的python迭代来编写逻辑。当您有这样的情况时，Beautifutsoup允许在情况下将函数作为过滤器传递到find_all。

from bs4 import BeautifulSoup
html_doc = """<html>..."""
soup = BeautifulSoup(html_doc)
def matches_block(tag):
    return matches_dropdown(tag) and tag.find(matches_text) != None
def matches_dropdown(tag):
    return tag.name == 'li' and tag.has_attr('class') and 'dropdown' in tag['class']
def matches_text(tag):
    return tag.name == 'a' and tag.get_text().startswith('TEXT_1')
for li in soup.find_all(matches_block):
    for ul in li.find_all('ul', class_='dropdown-menu'):
        for a in ul.find_all('a'):
            if a.has_attr('href'):
                print (a['href'])

相关内容

最新更新

热门标签：