我想通过分页来提取所有信息。
分页部分的源代码为:
<tr class="pagination" valign="middle" align="center">
<td colspan="9">
<table>
<tbody>
<tr>
<td><span>1</span></td>
<td><a href="javascript:__doPostBack('ctl00$cph1$grdRfqSearch','Page$2')">2</a></td>
<td><a href="javascript:__doPostBack('ctl00$cph1$grdRfqSearch','Page$3')">3</a></td>
<td><a href="javascript:__doPostBack('ctl00$cph1$grdRfqSearch','Page$4')">4</a></td>
<td><a href="javascript:__doPostBack('ctl00$cph1$grdRfqSearch','Page$5')">5</a></td>
<td><a href="javascript:__doPostBack('ctl00$cph1$grdRfqSearch','Page$6')">6</a></td>
<td><a href="javascript:__doPostBack('ctl00$cph1$grdRfqSearch','Page$7')">7</a></td>
<td><a href="javascript:__doPostBack('ctl00$cph1$grdRfqSearch','Page$8')">8</a></td>
<td><a href="javascript:__doPostBack('ctl00$cph1$grdRfqSearch','Page$9')">9</a></td>
<td><a href="javascript:__doPostBack('ctl00$cph1$grdRfqSearch','Page$10')">10</a></td>
<td><a href="javascript:__doPostBack('ctl00$cph1$grdRfqSearch','Page$11')">...</a></td>
<td><a href="javascript:__doPostBack('ctl00$cph1$grdRfqSearch','Page$Last')">Last</a></td>
</tr>
</tbody>
</table>
</td>
</tr>
和我的硒脚本是:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from datetime import date,timedelta
import sqlite3
import os
today = date.today()
yesterday = today - timedelta(days=2)
d3 = yesterday.strftime("%m-%d-%Y")
URL = 'https://www.dibbs.bsm.dla.mil//rfq/rfqdates.aspx?category=recent'
options = webdriver.ChromeOptions()
driver = webdriver.Chrome(ChromeDriverManager().install(),options=options)
driver.get(URL)
driver.find_element_by_id("butAgree").click()
driver.find_element_by_partial_link_text(d3).click()
col1 = driver.find_elements_by_xpath('//table[@id="ctl00_cph1_grdRfqSearch"]/tbody/tr/td[1]')
col2 = driver.find_elements_by_xpath('//table[@id="ctl00_cph1_grdRfqSearch"]/tbody/tr/td[2]')
col3 = driver.find_elements_by_xpath('//table[@id="ctl00_cph1_grdRfqSearch"]/tbody/tr/td[3]')
col4 = driver.find_elements_by_xpath('//table[@id="ctl00_cph1_grdRfqSearch"]/tbody/tr/td[4]')
col5 = driver.find_elements_by_xpath('//table[@id="ctl00_cph1_grdRfqSearch"]/tbody/tr/td[5]')
col6 = driver.find_elements_by_xpath('//table[@id="ctl00_cph1_grdRfqSearch"]/tbody/tr/td[6]')
col7 = driver.find_elements_by_xpath('//table[@id="ctl00_cph1_grdRfqSearch"]/tbody/tr/td[7]')
col8 = driver.find_elements_by_xpath('//table[@id="ctl00_cph1_grdRfqSearch"]/tbody/tr/td[8]')
col9 = driver.find_elements_by_xpath('//table[@id="ctl00_cph1_grdRfqSearch"]/tbody/tr/td[9]')
col1_data = [s.text for s in col1]
col2_data = [s.text for s in col2]
col3_data = [s.text for s in col3]
col4_data = [s.text for s in col4]
col5_data = [s.text for s in col5]
col6_data = [s.text for s in col6]
col7_data = [s.text for s in col7]
col8_data = [s.text for s in col8]
col9_data = [s.text for s in col9]
这段代码从一个页面中提取数据，我想从分页中列出的所有页面中提取数据。
您可以将代码包装在while循环中，使用变量pagination_starting_point并将初始值设置为2，并且对于每次迭代，我们将增加计数器。
代码:
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
driver.implicitly_wait(30)
wait = WebDriverWait(driver, 30)
while True:
try:
col1 = driver.find_elements_by_xpath('//table[@id="ctl00_cph1_grdRfqSearch"]/tbody/tr/td[1]')
col2 = driver.find_elements_by_xpath('//table[@id="ctl00_cph1_grdRfqSearch"]/tbody/tr/td[2]')
col3 = driver.find_elements_by_xpath('//table[@id="ctl00_cph1_grdRfqSearch"]/tbody/tr/td[3]')
col4 = driver.find_elements_by_xpath('//table[@id="ctl00_cph1_grdRfqSearch"]/tbody/tr/td[4]')
col5 = driver.find_elements_by_xpath('//table[@id="ctl00_cph1_grdRfqSearch"]/tbody/tr/td[5]')
col6 = driver.find_elements_by_xpath('//table[@id="ctl00_cph1_grdRfqSearch"]/tbody/tr/td[6]')
col7 = driver.find_elements_by_xpath('//table[@id="ctl00_cph1_grdRfqSearch"]/tbody/tr/td[7]')
col8 = driver.find_elements_by_xpath('//table[@id="ctl00_cph1_grdRfqSearch"]/tbody/tr/td[8]')
col9 = driver.find_elements_by_xpath('//table[@id="ctl00_cph1_grdRfqSearch"]/tbody/tr/td[9]')
col1_data = [s.text for s in col1]
col2_data = [s.text for s in col2]
col3_data = [s.text for s in col3]
col4_data = [s.text for s in col4]
col5_data = [s.text for s in col5]
col6_data = [s.text for s in col6]
col7_data = [s.text for s in col7]
col8_data = [s.text for s in col8]
col9_data = [s.text for s in col9]
wait.until(EC.element_to_be_clickable((By.XPATH, f"//a[contains(@href,'Page${pagination_starting_point}')]"))).click()
print("Click on page " + pagination_starting_point)
pagination_starting_point = pagination_starting_point + 1
if pagination_starting_point == 45:
break
except:
print("Looks like job done !")
break
进口:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
更新为使用xpath//a[contains(@href,'Page${pagination_starting_point}')]代替页码。
不要这样组织代码，避免使用函数或列表推导式重复:
all_cols = [driver.find_elements_by_xpath(f'//table[@id="ctl00_cph1_grdRfqSearch"]/tbody/tr/td[{i}]') for i in range(1,10)]
all_cols_data = [[s.text for s in col] for col in all_cols]