Parce data, Python

Question

我对Python很感兴趣。我需要解析来自Fred的数据(https://fred.stlouisfed.org/series/A191RI1A225NBEA)。所以，我需要让Python访问这个网站，然后单击"下载"按钮，然后单击"CSV(数据("按钮，并将这些数据保存在Python中。事实上，这个任务有一个很大的问题，我尝试使用硒：

selenium.webdriver.common.keys import Keys
browser = webdriver.Chrome('/Users/davidabramyan/Downloads/chromedriver 3')
browser = browser.get('https://fred.stlouisfed.org/series/A191RI1A225NBEA')
elm = browser.find_element_by_link_text('Download')
browser.implicitly_wait(5)
elm.click()

使用BeautifulSoup：

response = urllib.request.urlopen(url)
return response.read()
def parse(html):
soup = BeautifulSoup(html, 'lxml')
table = soup.find('span', class_='pull-right col-xs-1')
rows = table.find('fg-download-menu')
print(table.prettify())
def main():
parse(get_html('https://fred.stlouisfed.org/series/A191RI1A225NBEA#0'))
if __name__ == '__main__':
main()

但它不起作用。我认为Beautifulsoup更好，但在那里我真的不知道该怎么做。你能帮我吗？提前谢谢！

Answer 1

您不能使用BeautifulSoup，因为"CSV(数据("的下载url是由ajax 生成的

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
browser = webdriver.Chrome('/Users/davidabramyan/Downloads/chromedriver 3')
# make sure above is chromedriver executable path not directory
wait = WebDriverWait(browser, 10)
browser.get('https://fred.stlouisfed.org/series/A191RI1A225NBEA')
dlButton = browser.find_element_by_id("download-button")
dlButton.click()
wait.until(lambda driver: browser.execute_script("return $('#download-data-csv').attr('href') != '#'"))
dlButton = browser.find_element_by_id("download-data-csv")
dlButton.click()
# get url to download with python
# csv_download_url = dlButton.get_attribute('href')
# csv_file = urllib2.urlopen(download)
# ....