如何使用BeautifulSoup抓取YouTube评论

Question

我是新手。 我想知道如何使用BeautifulSoup抓取YouTube评论。我在这里被震撼了。任何人都可以帮我写代码吗？

这是我写的：

import requests    
from bs4 import BeautifulSoup
r = requests.get("https://www.youtube.com/watch?v=kffacxfA7G4"    
req =r.conten    
soup = BeautifulSoup(req,'html.parser')    
print(soup.prettify())    
all = soup.find_all('div',{'id' : 'contents'})

我被困在这里没有得到任何输出，检查它显示评论的 wb 页面有 id = 内容

Answer 1

该站点的评论是动态生成的。您无法使用主链接获取它们，利用requests和BeautifulSoup库。要获得上述链接的内容跟踪，您需要使用任何浏览器模拟器，例如 selenium .作为初学者，您可以尝试如下。以下脚本将获取解开的注释。顺便说一句，该网站还激活了延迟加载方法，因此您需要抽搐for loop以获取更多内容。

import time
from selenium.webdriver import Chrome
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
with Chrome() as driver:
    wait = WebDriverWait(driver,10)
    driver.get("https://www.youtube.com/watch?v=kffacxfA7G4")
    for item in range(3): #by increasing the highest range you can get more content
        wait.until(EC.visibility_of_element_located((By.TAG_NAME, "body"))).send_keys(Keys.END)
        time.sleep(3)
    for comment in wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "#comment #content-text"))):
        print(comment.text)

部分输出：

15 April 2018 ?¿?
April 2018??
8 years people 👌
Nice songs Justin Bieber https://youtu.be/OvfAc7JGoc4
2018 hit like...♥️♥️♥️♥️😁👌🏻
8 years complete 🙏
Can likes beat dislikes??
View 1, 8 billion great song