我正在使用请求库进行练习，并且在首次使用会话登录后尝试从网页中获取表时遇到了问题。
我试图从这样的页面中获取"交易历史"表，但为了查看表格中的实际交易，您必须先登录到该页面。我有登录凭据，并且我尝试使用以下代码，它似乎可以使用 200 状态代码登录，但是当我尝试在此之后从该 URL 中提取表时，我的请求的输出仍然显示我需要登录才能查看表。
我使用的代码如下所示：
import requests
import requests
from bs4 import BeautifulSoup
login_url = "https://www.mql5.com/en/auth_login"
payload = {
'Login': "MY_USERNAME_HERE",
'Password': "MY_PASSWORD_HERE"
}
table_url = "https://www.mql5.com/en/signals/811921?source=Site+Signals+MT5+Table#!tab=history"
# Start the session
session = requests.Session()
# Post the payload to the site to log in
s = session.post(login_url, data=payload)
print(s.status_code)
# Navigate to the next page and scrape the data
html = session.get(table_url)
print(html.status_code)
# Get the soup of the page
soup = BeautifulSoup(html.text, 'html.parser')
# Find the table
table = soup.find("div", {"id": "tab_content_history"})
table = table.find("div", {"class": "tab__content__item"})
print(table)
我的打印输出如下所示：
200
200
<div class="tab__content__item">
<div class="need-auth">To see trades in realtime, please <strong><a href="https://www.mql5.com/en/auth_login">log in</a></strong> or <strong><a href="https://www.mql5.com/en/auth_register">register</a></strong></div>
</div>
Login和Password是登录元素的name。在使会话正确登录时，我还会缺少什么？我没有看到任何需要添加到有效负载中的令牌，但我也可能错了。我还能尝试什么？如果您没有登录此站点，那很好，如果您有使用请求会话的经验，只需寻找有关尝试的其他想法。您可以从此页面查看登录表单的页面检查。我承认不允许网络抓取，我只是将此站点用作公共示例，因为我需要在工作项目中使用类似的逻辑。谢谢！
试试这个：
import random
import time
import pandas as pd
from bs4 import BeautifulSoup
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import StaleElementReferenceException
user_agent_list = [
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:90.0) Gecko/20100101 Firefox/90.0',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 11.5; rv:90.0) Gecko/20100101 Firefox/90.0',
'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_5_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36',
'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0',
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36'
]
data = []
# set url
url = 'https://www.mql5.com/en/signals/811921?source=Site+Signals+MT5+Table#!tab=history'
# call open browser function
chrome_options = Options()
chrome_options.add_argument("--window-size=1920,1080")
chrome_options.headless = True
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument("--no-sandbox")
prefs = {"profile.managed_default_content_settings.images": 2}
chrome_options.add_experimental_option("prefs", prefs)
userAgent = random.choice(user_agent_list)
chrome_options.add_argument(f'user-agent={userAgent}')
driver = webdriver.Chrome(ChromeDriverManager().install(), options=chrome_options)
driver.get(url)
# login to website
sign_in_link = driver.find_element_by_link_text('Log in')
sign_in_link.click()
username = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, 'Login')))
password = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, 'Password')))
username.send_keys('your_username')
password.send_keys('your_password')
driver.find_element_by_id("loginSubmit").click()
time.sleep(2)
innerHTML = driver.page_source
soup = BeautifulSoup(innerHTML, "html.parser")
# Find the table
table = soup.find("div", {"id": "tab_content_history"})
table = table.find("div", {"class": "tab__content__item"})
table_body = table.find('tbody')
rows = table_body.find_all('tr')
for row in rows:
cols = row.find_all('td')
cols = [ele.text.strip() for ele in cols]
data.append([ele for ele in cols])
df = pd.DataFrame(data[1:])
df.columns = ['Time', 'Type', 'Volume', 'Symbol', 'Price', 'Time', 'Price', 'Commission', 'Swap', 'Profit']
print(df)
driver.quit()
输出：
====== WebDriver manager ======
Current google-chrome version is 93.0.4577
Get LATEST driver version for 93.0.4577
Driver [/home/user/.wdm/drivers/chromedriver/linux64/93.0.4577.63/chromedriver] found in cache
Time Type Volume Symbol Price Time Price Commission Swap Profit
0 2021.09.17 09:56 Buy 0.03 NZDCAD 0.89502 2021.09.17 14:54 0.89623 -0.12 2.85
1 2021.09.17 02:58 Buy 0.03 NZDCAD 0.89570 2021.09.17 06:36 0.89625 -0.12 1.30
2 2021.09.16 14:17 Buy 0.03 NZDCAD 0.89626 2021.09.16 19:03 0.89749 -0.12 2.91
3 2021.09.16 12:48 Buy 0.03 NZDCAD 0.89714 2021.09.16 19:03 0.89749 -0.12 0.83
4 2021.09.16 04:56 Sell 0.03 NZDCAD 0.89857 2021.09.16 12:38 0.89777 -0.12 1.90
5 2021.09.16 01:31 Sell 0.03 AUDCAD 0.92665 2021.09.16 02:11 0.92650 -0.12 0.36
6 2021.09.15 14:32 Buy 0.03 NZDCAD 0.89842 2021.09.15 22:45 0.89896 -0.12 -0.51 1.28
7 2021.09.15 12:49 Buy 0.03 NZDCAD 0.89995 2021.09.15 22:45 0.89896 -0.12 -0.51 -2.35
8 2021.09.15 04:16 Sell 0.03 NZDCAD 0.89992 2021.09.15 12:35 0.89977 -0.12 0.35
9 2021.09.14 13:07 Sell 0.03 NZDCAD 0.90053 2021.09.14 14:12 0.90043 -0.12 0.24
10 2021.09.14 08:08 Sell 0.03 NZDCAD 0.90040 2021.09.14 10:16 0.89976 -0.12 1.52
11 2021.09.13 07:44 Sell 0.02 NZDCAD 0.90165 2021.09.13 09:42 0.90094 -0.08 1.12
12 2021.09.13 01:10 Sell 0.03 NZDCAD 0.90096 2021.09.13 06:30 0.89982 -0.12 2.70
13 2021.09.10 12:32 Sell 0.05 NZDCAD 0.90067 2021.09.10 12:58 0.90042 -0.20 0.99
14 2021.09.09 07:50 Sell 0.03 NZDCAD 0.90334 2021.09.09 16:59 0.89919 -0.12 9.86
15 2021.09.08 11:16 Sell 0.03 NZDCAD 0.90021 2021.09.09 16:59 0.89920 -0.12 -0.30 2.40
16 2021.09.07 16:20 Sell 0.03 NZDCAD 0.89673 2021.09.09 16:59 0.89914 -0.12 -0.40 -5.72
17 2021.09.07 11:07 Sell 0.03 NZDCAD 0.89637 2021.09.07 13:54 0.89522 -0.12 2.74
18 2021.09.03 16:54 Sell 0.03 NZDCAD 0.89541 2021.09.06 14:26 0.89438 -0.12 -0.10 2.47
19 2021.09.03 14:03 Sell 0.03 NZDCAD 0.89447 2021.09.03 14:49 0.89342 -0.12 2.52
20 2021.09.02 12:52 Sell 0.03 NZDCAD 0.89466 2021.09.02 16:43 0.89307 -0.12 3.80
21 2021.09.02 09:45 Sell 0.03 NZDCAD 0.89297 2021.09.02 16:43 0.89307 -0.12 -0.24
22 2021.09.02 06:21 Sell 0.03 NZDCAD 0.89244 2021.09.02 08:00 0.89140 -0.12 2.48
23 2021.08.31 11:29 Sell 0.03 NZDCAD 0.88831 2021.09.01 12:40 0.88704 -0.12 -0.10 3.03
24 2021.08.31 06:32 Sell 0.03 NZDCAD 0.88773 2021.09.01 12:40 0.88707 -0.12 -0.10 1.57
25 2021.08.31 03:02 Sell 0.03 NZDCAD 0.88654 2021.09.01 12:40 0.88707 -0.12 -0.10 -1.26
26 2021.08.30 02:31 Sell 0.03 NZDCAD 0.88391 2021.08.30 06:12 0.88335 -0.12 1.33
27 2021.08.27 16:47 Sell 0.03 NZDCAD 0.88475 2021.08.29 22:04 0.88437 -0.12 -0.10 0.90
28 2021.08.27 12:31 Sell 0.03 NZDCAD 0.88189 2021.08.27 13:22 0.88120 -0.12 1.63
29 2021.08.27 01:10 Sell 0.03 NZDCAD 0.88193 2021.08.27 03:31 0.88013 -0.12 4.26
30 2021.08.26 15:55 Sell 0.03 NZDCAD 0.88039 2021.08.27 03:31 0.88013 -0.12 -0.10 0.61
31 2021.08.26 14:42 Sell 0.03 NZDCAD 0.87928 2021.08.27 03:31 0.88013 -0.12 -0.10 -2.01
32 2021.08.25 15:13 Sell 0.02 NZDCAD 0.87859 2021.08.26 09:05 0.87712 -0.08 -0.20 2.33
33 2021.08.25 10:09 Sell 0.03 NZDCAD 0.87745 2021.08.26 09:05 0.87710 -0.12 -0.30 0.83
34 2021.08.25 07:58 Sell 0.03 NZDCAD 0.87632 2021.08.26 09:05 0.87712 -0.12 -0.30 -1.90
35 2021.08.24 08:45 Sell 0.03 NZDCAD 0.87599 2021.08.24 17:02 0.87533 -0.12 1.57
36 2021.08.23 04:20 Buy 0.03 NZDCAD 0.87396 2021.08.23 12:44 0.87461 -0.12 1.53
37 2021.08.20 13:20 Sell 0.03 NZDCAD 0.87990 2021.08.20 13:47 0.87896 -0.12 2.19
38 2021.08.20 12:08 Sell 0.03 NZDCAD 0.88049 2021.08.20 13:16 0.88043 -0.12 0.14
39 2021.08.20 04:32 Sell 0.03 NZDCAD 0.87886 2021.08.20 06:40 0.87781 -0.12 2.45
40 2021.08.18 13:30 Buy 0.03 NZDCAD 0.86860 2021.08.18 15:31 0.87046 -0.12 4.42
41 2021.08.18 12:35 Buy 0.03 NZDCAD 0.87085 2021.08.18 15:31 0.87049 -0.12 -0.85
42 2021.08.18 11:37 Sell 0.03 AUDNZD 1.05070 2021.08.18 12:00 1.05023 -0.12 0.97
43 2021.08.18 10:14 Buy 0.03 NZDCAD 0.87009 2021.08.18 11:28 0.87071 -0.12 1.47
44 2021.08.18 07:11 Buy 0.03 EURUSD 1.17214 2021.08.18 07:39 1.17277 -0.18 1.89
45 2021.08.18 05:13 Buy 0.03 EURUSD 1.17161 2021.08.18 06:59 1.17172 -0.18 0.33
46 2021.08.18 05:01 Sell 0.03 NZDCAD 0.87438 2021.08.18 05:55 0.87325 -0.12 2.69
47 2021.08.17 12:59 Buy 0.03 NZDCAD 0.87254 2021.08.18 01:13 0.87490 -0.12 -0.17 5.61
确保系统中安装了硒和美丽汤