我正在尝试下载谷歌趋势数据并使用PhantomJS加载加载页面并提取所需的数据。当我在url中只使用一个关键字运行代码时(例如url: https://www.google.com/trends/explore?date=today%203-m&geo=US&q=Blue),它工作得很好。只要我添加第二个关键字(示例url: https://www.google.com/trends/explore?date=today%203-m&geo=US&q=Blue,Red) PhantomJS不再正确加载页面,我无法找到我需要的数据。我已经尝试增加浏览器等待的时间,并尝试了许多不同的关键字,但没有任何成功。我的想法,根本不明白为什么我的程序不再工作后改变网址如此轻微(标签和页面结构几乎是相同的两个网址,所以它的问题不是标签不再有相同的名称像以前)下面是有问题的代码:
# Reading google trends data
google_trend_array = []
url = 'https://www.google.com/trends/explore?date=today%203-m&geo=US&q=Blue,Red'
browser = webdriver.PhantomJS('...\phantomjs-2.1.1-windows\bin\phantomjs.exe')
ran_smooth = False
time_to_sleep = 3
# ran_smooth makes sure that page has loaded and necessary code was extracted, if not it will try to load the page again
while ran_smooth is False:
browser.get(url)
time.sleep(time_to_sleep)
soup = BeautifulSoup(browser.page_source, "html.parser") # BS object to use bs4
table = soup.find('div', {'aria-label': 'A tabular representation of the data in the chart.'})
# If page didn't load, this try will throw an exception
try:
# Copies all the data out of google trends table
for col in table.findAll('td'):
# google has both dates and trend values, the following function ensures that we only read the trend values
if col.string.isdigit() is True:
trend_number = int(col.string)
google_trend_array.append(trend_number)
# program ran through, leave while loop
ran_smooth = True
except AttributeError:
print 'page not loading for term ' + str(term_to_trend) + ', trying again...'
time_to_sleep += 1 # increase time to sleep so that page can load
print google_trend_array
你应该看看pytrends,而不是重新发明轮子。
下面是一个小示例:如何从Google Trends中提取数据帧:import pytrends.request
google_username = "<your_login>@gmail.com"
google_password = "<your_password>"
# connect to Google
pytrend = pytrends.request.TrendReq(google_username, google_password, custom_useragent='My Pytrends Script')
trend_payload = {'q': 'Pizza, Italian, Spaghetti, Breadsticks, Sausage', 'cat': '0-71'}
# trend = pytrend.trend(trend_payload)
df = pytrend.trend(trend_payload, return_type='dataframe')
你会得到:
breadsticks italian pizza sausage spaghetti
Date
2004-01-01 0.0 9.0 34.0 3.0 3.0
2004-02-01 0.0 10.0 32.0 2.0 3.0
2004-03-01 0.0 10.0 32.0 2.0 3.0
2004-04-01 0.0 9.0 31.0 2.0 2.0
2004-05-01 0.0 9.0 32.0 2.0 2.0
2004-06-01 0.0 8.0 29.0 2.0 3.0
2004-07-01 0.0 8.0 34.0 2.0 3.0
[...]