我正在尝试获取此url 中的所有存储位置
https://www.walmart.com/store/finder?location=Pennsylvani&距离=100
我目前使用这种方法
def getAllStoreLocation():
session = requests.Session()
url = "https://www.walmart.com/store/finder?location=Pennsylvani&distance=100"
try:
stateStoreSoup = BeautifulSoup(session.get(url).text, 'html.parser')
stateTagId = find_tag(stateStoreSoup)
stateScript = stateStoreSoup.find("script", {"id":stateTagId})
stateData = json.loads(stateScript.get_text(strip=True))
except Exception as e:
print(e)
tagId函数
def find_tag(soup):
script = soup.find('script', {'type': 'application/json', 'id':re.compile(r'^((?!tb-djs).)*$')})
return script['id']
这是有效的，但提供了太多不需要的数据。我想在这个答案中使用的方法
如何使用beautifulsoup 从javascript加载的站点获取数据
获取所有存储数据的列表
如何使用答案中的Ajax方法获取商店位置数据。请注意，我只熟悉beautifulsoup，而不熟悉Ajax，所以我非常感谢对
代码的良好解释
import requests
import re
def main(url):
r = requests.get(url)
match = [x.group(1) for x in re.finditer(r"address":({.*?})", r.text)]
for item in match:
item = eval(item)
print("City: {:<30}, Address: {}".format(
item['city'], item['address']))
main("https://www.walmart.com/store/finder?location=Pennsylvani&distance=100")
输出：
City: Montoursville , Address: 1015 N Loyalsock Ave
City: Mill Hall , Address: 167 Hogan Blvd
City: Lewisburg , Address: 120 Ajk Blvd
City: Selinsgrove , Address: 980 N Susquehanna Trl
City: Bloomsburg , Address: 100 Lunger Dr
City: Mansfield , Address: 1169 S Main St
City: State College , Address: 373 Benner Pike
City: State College , Address: 1665 N Atherton St
City: Coal Township , Address: 9300 State Route 61
City: Lewistown , Address: 10180 Us Highway 522 S
City: Elizabethville , Address: 200 Kocher Ln
City: Saint Clair , Address: 500 Terry Rich Blvd
City: Sayre , Address: 1887 Elmira St
City: Hazle Township , Address: 761 Airport Rd
City: Clearfield , Address: 100 Supercenter Dr
City: Painted Post , Address: 3217 Silverback Ln
City: Huntingdon , Address: 6716 Towne Center Blvd
City: Mechanicsburg , Address: 6520 Carlisle Pike Ste 550
City: Tamaqua , Address: 35 Plaza Dr
City: Horseheads , Address: 1400 County Rd 64
City: Tunkhannock , Address: 808 Hunter Hwy
City: Harrisburg , Address: 6535 Grayson Rd
City: Camp Hill , Address: 3400 Hartzdale Drive
City: Palmyra , Address: 100 No Londonderry Square
City: Wilkes Barre , Address: 2150 Wilkes Barre Twnsp Mktpl
City: Carlisle , Address: 60 Noble Blvd
City: Saint Marys , Address: 1102 Million Dollar Hwy
City: Lebanon , Address: 1355 E Lehman St
City: Pittston , Address: 390 Route 315 Hwy
City: Etters , Address: 50 Newberry Pkwy
City: Hamburg , Address: 1800 Tilden Ridge
City: Taylor , Address: 1325 Main
City: Du Bois , Address: 20 Industrial Dr
City: Shippensburg , Address: 100 S Conestoga Dr
City: Lehighton , Address: 1731 Blakeslee Boulevard Dr E
City: Altoona , Address: 2600 Plank Road Commons
City: Dickson City , Address: 900 Commerce Blvd
City: Hornell , Address: 1000 State Route 36
City: Watkins Glen , Address: 515 E 4th St
City: Temple , Address: 5370 Allentown Pike
City: Duncansville , Address: 200 Commerce Dr
City: Vestal , Address: 2405 Vestal Pkwy E
City: Wyomissing , Address: 1135 Berkshire Blvd
City: York , Address: 1000 Town Center Drive
City: York , Address: 2801 E Market St Bldg B
City: Olean , Address: 1869 Plaza Dr
City: Ephrata , Address: 890 E Main St
City: Johnson City , Address: 2 Gannett Dr
City: Lancaster , Address: 2030 Fruitville Pike
City: Bradford , Address: 50 Foster Brook Blvd
更新代码：
import requests
import re
def main(url):
r = requests.get(url)
match = [x.group(1, 2) for x in re.finditer(
r',"id":(.*?),.*?address":({.*?})', r.text)]
for a, b in match:
item = eval(b)
idf = a
print("ID: {}, City: {:<30}, Address: {}".format(idf,
item['city'], item['address']))
main("https://www.walmart.com/store/finder?location=Pennsylvani&distance=100")
输出：
ID: 1887, City: Montoursville , Address: 1015 N Loyalsock Ave
ID: 2528, City: Mill Hall , Address: 167 Hogan Blvd
ID: 1644, City: Lewisburg , Address: 120 Ajk Blvd
ID: 2185, City: Selinsgrove , Address: 980 N Susquehanna Trl
ID: 1794, City: Bloomsburg , Address: 100 Lunger Dr
ID: 1945, City: Mansfield , Address: 1169 S Main St
ID: 2230, City: State College , Address: 373 Benner Pike
ID: 1640, City: State College , Address: 1665 N Atherton St
ID: 2481, City: Coal Township , Address: 9300 State Route 61
ID: 1607, City: Lewistown , Address: 10180 Us Highway 522 S
ID: 3412, City: Elizabethville , Address: 200 Kocher Ln
ID: 2535, City: Saint Clair , Address: 500 Terry Rich Blvd
ID: 2208, City: Sayre , Address: 1887 Elmira St
ID: 2255, City: Hazle Township , Address: 761 Airport Rd
ID: 2129, City: Clearfield , Address: 100 Supercenter Dr
ID: 2992, City: Painted Post , Address: 3217 Silverback Ln
ID: 5470, City: Huntingdon , Address: 6716 Towne Center Blvd
ID: 1886, City: Mechanicsburg , Address: 6520 Carlisle Pike Ste 550
ID: 3634, City: Tamaqua , Address: 35 Plaza Dr
ID: 1976, City: Horseheads , Address: 1400 County Rd 64
ID: 2024, City: Tunkhannock , Address: 808 Hunter Hwy
ID: 1591, City: Harrisburg , Address: 6535 Grayson Rd
ID: 5888, City: Camp Hill , Address: 3400 Hartzdale Drive
ID: 2888, City: Palmyra , Address: 100 No Londonderry Square
ID: 1623, City: Wilkes Barre , Address: 2150 Wilkes Barre Twnsp Mktpl
ID: 2574, City: Carlisle , Address: 60 Noble Blvd
ID: 1792, City: Saint Marys , Address: 1102 Million Dollar Hwy
ID: 2023, City: Lebanon , Address: 1355 E Lehman St
ID: 2543, City: Pittston , Address: 390 Route 315 Hwy
ID: 4404, City: Etters , Address: 50 Newberry Pkwy
ID: 4612, City: Hamburg , Address: 1800 Tilden Ridge
ID: 4276, City: Taylor , Address: 1325 Main
ID: 1769, City: Du Bois , Address: 20 Industrial Dr
ID: 5358, City: Shippensburg , Address: 100 S Conestoga Dr
ID: 2169, City: Lehighton , Address: 1731 Blakeslee Boulevard Dr E
ID: 2049, City: Altoona , Address: 2600 Plank Road Commons
ID: 1884, City: Dickson City , Address: 900 Commerce Blvd
ID: 2326, City: Hornell , Address: 1000 State Route 36
ID: 3221, City: Watkins Glen , Address: 515 E 4th St
ID: 2614, City: Temple , Address: 5370 Allentown Pike
ID: 4348, City: Duncansville , Address: 200 Commerce Dr
ID: 1835, City: Vestal , Address: 2405 Vestal Pkwy E
ID: 1670, City: Wyomissing , Address: 1135 Berkshire Blvd
ID: 2205, City: York , Address: 1000 Town Center Drive
ID: 1529, City: York , Address: 2801 E Market St Bldg B
ID: 2159, City: Olean , Address: 1869 Plaza Dr
ID: 2340, City: Ephrata , Address: 890 E Main St
ID: 5732, City: Johnson City , Address: 2 Gannett Dr
ID: 5200, City: Lancaster , Address: 2030 Fruitville Pike
ID: 3514, City: Bradford , Address: 50 Foster Brook Blvd
我已经注意到您使用另一个python lib请求给出的答案。
如果尚未安装库，请尝试pip install requests。
并尝试使用这样的URL来获取数据
https://www.walmart.com/store/finder/electrode/api/stores?singleLineAddr=Pennsylvani&距离=25