BeautifulSoup未找到所有tr标签

Question

我完全不知所措。我正试图从以下url中提取一个表https://www.baseball-reference.com/leagues/MLB/2019-appearances-fielding.shtml.

在这一页上，有两张表。第一个是团队外观，第二个是球员外观。

下面的代码片段让我可以毫无问题地获得页面的html。

url = 'https://www.baseball-reference.com/leagues/MLB/2019-appearances-fielding.shtml'
html_content = requests.get(url).text
soup = BeautifulSoup(html_content, 'lxml')
print(soup.prettify())

一切都很顺利。当我尝试解析以获取标头信息时，我没有遇到任何问题。要做到这一点，我使用：

header = soup.find_all('tr', attrs ={"class": "thead"})
header

这里发生的情况是，它只是第一次打印它，它来自团队外观表，而不是两者。

[<tr class="thead">
<th aria-label="Tm" class="sort_default_asc show_partial_when_sorting center" data-stat="team_ID">Tm</th>
<th aria-label="G_team" class="center" data-stat="G_team"></th>
<th aria-label="Games Played" class="center" data-stat="G_all" data-tip="All games played">G</th>
<th aria-label="Games Started" class="center" data-stat="GS" data-tip="Games Started">GS</th>
<th aria-label="Games Played" class="center" data-stat="G_batting" data-tip="Games appeared in the batting order,&lt;br&gt;but may not have batted.">Batting</th>
<th aria-label="Games Played" class="center" data-stat="G_defense" data-tip="Games in lineup at a defensive position.">Defense</th>
<th aria-label="Games Played" class="center" data-stat="G_p_app" data-tip="Games in lineup or announced as a pitcher">P</th>
<th aria-label="Games Played" class="center" data-stat="G_c" data-tip="Games in lineup as a catcher">C</th>
<th aria-label="Games Played" class="center" data-stat="G_1b" data-tip="Games in lineup as a first baseman">1B</th>
<th aria-label="Games Played" class="center" data-stat="G_2b" data-tip="Games in lineup as a second baseman">2B</th>
<th aria-label="Games Played" class="center" data-stat="G_3b" data-tip="Games in lineup as a third baseman">3B</th>
<th aria-label="Games Played" class="center" data-stat="G_ss" data-tip="Games in lineup as a shortstop">SS</th>
<th aria-label="Games Played" class="center" data-stat="G_lf_app" data-tip="Games in lineup as a left fielder">LF</th>
<th aria-label="Games Played" class="center" data-stat="G_cf_app" data-tip="Games in lineup as a center fielder">CF</th>
<th aria-label="Games Played" class="center" data-stat="G_rf_app" data-tip="Games in lineup as a right fielder">RF</th>
<th aria-label="Games Played" class="center" data-stat="G_of_app" data-tip="Games in lineup as an outfielder">OF</th>
<th aria-label="Games Played" class="center" data-stat="G_dh" data-tip="Games in lineup as a designated hitter">DH</th>
<th aria-label="Games Played" class="center" data-stat="G_ph" data-tip="Games in lineup as a pinch hitter&lt;br&gt;May have played another position as well.">PH</th>
<th aria-label="Games Played" class="center" data-stat="G_pr" data-tip="Games in lineup as a pinch runner&lt;br&gt;May have played another position as well.">PR</th>
</tr>]

当我检查页面的html时，播放器外观表也有这个类，所以我不确定为什么两者都没有填充。此外，如果我将代码更改为类似的代码

table = soup.find_all('tr', attrs ={"class": "full_table"})
table

这符合实际数据的类别，我没有得到任何回报。

我已经做了一段时间了，现在很失落。我只是在这里忽略了一些简单的东西，还是什么？我们非常感谢任何帮助和见解。

谢谢！

Answer 1

您可以使用comments = soup.find_all(string=lambda text: isinstance(text, Comment))提取注释，然后解析其中的表。

import requests
from bs4 import BeautifulSoup, Comment
import pandas as pd
headers = {'User-Agent': 
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36'}
# Get list of tables with pandas    
url = "https://www.baseball-reference.com/leagues/MLB/2019-appearances-fielding.shtml"
tables = pd.read_html(url)

# then pull out comments and get the tables within those
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.content, 'html.parser')
comments = soup.find_all(string=lambda text: isinstance(text, Comment))
for each in comments:
if '<table' in each:
tables.append(pd.read_html(each)[0])