Java模拟浏览器访问并获取网页代码

最新推荐文章于 2023-10-13 10:37:21 发布

原创最新推荐文章于 2023-10-13 10:37:21 发布 · 2.5k 阅读

1 ·

本内容遵循CC 4.0 BY-SA版权协议

收录于

本文介绍如何使用Java模拟浏览器访问那些不能直接通过URL获取HTML代码的网站，探讨相关技术和方法。

一般的网页可以直接通过URL直接获取HTML的代码

public static void main(String[] args) throws Exception {
    //获取url
    URL url = new URL("https://www.baidu.com");
    //下载资源
    InputStream is = url.openStream();
    BufferedReader br = new BufferedReader(new InputStreamReader(is,"utf-8"));
    String msg = null;
    while (null != (msg = br.readLine())){
        System.out.println(msg);
    }
    br.close();
}

但是有一些网站是不能直接访问的，此时我们需要模拟浏览器再去获取代码

public static void main(String[] args) throws Exception {
    //获取url
    URL url = new URL("https://www.dianping.com");
    //下载资源
    HttpURLConnection conn = (HttpURLConnection) url.openConnection();
    conn.setRequestMethod("GET");
    conn.setRequestProperty("User-Agent","Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36");
    BufferedReader br = new BufferedReader(new InputStreamReader(conn.getInputStream(),"utf-8"));
    String msg = null;
    while (null != (msg = br.readLine())){
        System.out.println(msg);
    }
    br.close();
}