我想从Python的URL用户中获取输入

Question

我想从用户输入中获取网站名称，最大值。他想爬网站的页面上的页面...但是无法获得任何解决方案。

import requests
from bs4 import *
from urllib import request

url1 = input("Enter url you want to crawl:")
max_pages1 = int(input("Enter no. of pages you want to crawl:"))

def web_crawler(max_pages,url):
   page = 1
   while page <= max_pages:
      url4 = str(url) + str(page)
      url_get = requests.get(url4)
      plain_text = url_get.text
      soup = BeautifulSoup(plain_text,"html.parser")
      for a in soup.findAll('a',{'rel':'bookmark'}):
          href = a.get('href')
          title = a.string
          #print(title)
          print(href)
          #info_about_web_pages(href)
      page +=1
def info_about_web_pages(url):
   url_get = requests.get(url)
   plain_text = url_get.text
   soup = BeautifulSoup(plain_text,"html.parser" )
   links = set()
   for about in soup.findAll('a'):
       href = about.get('href')
       links.update([href])
   print(links)
web_crawler(max_pages1,url1)

它在输出中没有显示我什么都没显示

Answer 1

如果您试图在html源代码中找到的属性没有锚点，那么这总是什么都不打印。尝试打印汤。prettify（），看看您是否正在寻找的标签是否存在。当我不打印我期望的值时，通常是因为值没有我要寻找的属性。