爬取网页成功但是没有数据的问题

原创已于 2025-02-10 13:12:38 修改 · 416 阅读

0 ·

本内容遵循CC 4.0 BY-SA版权协议

GEO检测

标签

#学习

于 2025-02-05 19:31:56 首次发布

爬取百度图片数据提示需要验证

尝试在请求头headers加入以下字段

accept:
text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7
accept-encoding:
gzip, deflate, br, zstd
accept-language:
zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6

例如：

import requests
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 "
                  "Safari/537.36 Edg/131.0.0.0",
    'accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
    'accept-encoding':'gzip, deflate, br, zstd',
    'accept-language':'zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6',
}
ori_url = 'https://image.baidu.com/search/index?tn=baiduimage&ipn=r&ct=201326592&cl=2&lm=&st=-1&fm=result&fr=&sf=1&fmq=1737206496134_R&pv=&ic=0&nc=1&z=&hd=&latest=&copyright=&se=1&showtab=0&fb=0&width=&height=&face=0&istype=2&dyTabStr=&ie=utf-8&sid=&word=%E7%94%B5%E5%BD%B1%E6%B5%B7%E6%8A%A5'
req = requests.get(ori_url, headers=headers)

爬取知网没有数据

尝试在post请求请求头headers加入以下字段
‘referer’:‘https://kns.cnki.net/kns8s/defaultresult/index’,

post请求加入以下数据
‘QueryJson’: ‘{“Platform”: “”, “Resource”: “CROSSDB”, “Classid”: “WD0FTY92”, “Products”: “”, “QNode”: {“QGroup”: [{“Key”: “Subject”, “Title”: “”, “Logic”: 0,“Items”: [{“Field”: “SU”, “Value”: “C语言”, “Operator”: “TOPRANK”, “Logic”: 0, “Title”: “主题”}],“ChildItems”: []}]}, “ExScope”: 1, “SearchType”: 2, “Rlang”: “CHINESE”,“KuaKuCode”: “YSTT4HG0,LSTPFY1C,JUP3MUPD,MPMFIG1A,EMRPGLPA,WQ0UVIAA,BLZOG7CK,PWFIRAGL,NN3FJMUV,NLBO1Z6R”,“SearchFrom”: 1}’,

例如


headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36',
    'referer':'https://kns.cnki.net/kns8s/defaultresult/index',
}

url = 'https://kns.cnki.net/kns8s/brief/grid'

data = {
    'QueryJson': '{"Platform": "", "Resource": "CROSSDB", "Classid": "WD0FTY92", "Products": "", "QNode": {"QGroup": [{"Key": "Subject", "Title": "", "Logic": 0,"Items": [{"Field": "SU", "Value": "C语言", "Operator": "TOPRANK", "Logic": 0, "Title": "主题"}],"ChildItems": []}]}, "ExScope": 1, "SearchType": 2, "Rlang": "CHINESE","KuaKuCode": "YSTT4HG0,LSTPFY1C,JUP3MUPD,MPMFIG1A,EMRPGLPA,WQ0UVIAA,BLZOG7CK,PWFIRAGL,NN3FJMUV,NLBO1Z6R","SearchFrom": 1}',
    'pageNum': 1,
    'pageSize': 20,
    'aside': '主题：C语言',
    'searchFrom': '资源范围：总库',
    'CurPage': 1,
}

response = requests.post(url, headers=headers, data=data)
print(response.text)

出些类似问题可以参考下