记一次java后台实现数据爬虫爬取数据DEMO

古城微笑少年丶 2021-07-09 04:26 300阅读 0赞

1、最近需要用到一个功能，需要功能：前端调用我后台的一个接口，并传一个url参数，我后台需要根据这个url去获取该网页的里面视频以及音频的地址、格式、分辨率等信息，并全部返回。

解决方案：这时候我就需要一个能够访问的前端传来的参数一个网页地址，然后该网页返回一个页面，我再后台用java解析该页面，并切割前端需要的视频，音频信息。

在网页操作过程

1、进入一个能下载视频的网址

![20210407100256606.png][]

2、点击“下载”后会进行一个网页跳转去访问我们输入的url地址，最后返回的页面

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTc4OTk4_size_16_color_FFFFFF_t_70][]

有我们需要的视频，以及音频文件，这时候我们就要获取视频，音频的下载地址，以及文件格式等信息。

以上是我们通过网页来请求的，这时候我们需要通过后台代码来实现。具体实现方式如下

1、搭建一个简单的SpringBoot工程

引入相应的jar,该jar是为了解析html页面

<dependency>
    			<groupId>org.jsoup</groupId>
    			<artifactId>jsoup</artifactId>
    			<version>1.12.2</version>
    		</dependency>

编写http连接工具类

import org.apache.http.conn.ClientConnectionManager;
    import org.apache.http.conn.scheme.Scheme;
    import org.apache.http.conn.scheme.SchemeRegistry;
    import org.apache.http.conn.ssl.SSLSocketFactory;
    import org.apache.http.impl.client.DefaultHttpClient;
    
    import javax.net.ssl.SSLContext;
    import javax.net.ssl.TrustManager;
    import javax.net.ssl.X509TrustManager;
    import java.security.cert.CertificateException;
    import java.security.cert.X509Certificate;
    
    
    public class SSLClient extends DefaultHttpClient {
    
        public SSLClient() throws Exception {
            super();
            //传输协议需要根据自己的判断
            SSLContext ctx = SSLContext.getInstance("TLSv1.2");
            X509TrustManager tm = new X509TrustManager() {
                @Override
                public void checkClientTrusted(X509Certificate[] chain,
                                               String authType) throws CertificateException {
                }
    
                @Override
                public void checkServerTrusted(X509Certificate[] chain,
                                               String authType) throws CertificateException {
                }
    
                @Override
                public X509Certificate[] getAcceptedIssuers() {
                    return null;
                }
            };
            ctx.init(null, new TrustManager[]{tm}, null);
            SSLSocketFactory ssf = new SSLSocketFactory(ctx, SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
            ClientConnectionManager ccm = this.getConnectionManager();
            SchemeRegistry sr = ccm.getSchemeRegistry();
            sr.register(new Scheme("https", 443, ssf));
        }
    }

通过在控制层编写调用的方法

@Api(description = "获取视频或者音频接口")
    @RestController
    public class SysLoginController extends AbstractController {
    
    	/**
    	 * 网址请求
    	 */
    	@ApiOperation(value="网址请求")
    	@PostMapping("/paqu")
    	public Object login(@RequestParam String url)throws IOException {
    		Map resultmap = new HashMap();
    		List<Map> urlList = new ArrayList<>();
    		String charset = "utf-8";
    		Map map = new HashMap();
    		map.put("URL",url);
    
    		String httpOrgCreateTestRtn = doPost("https://www.需要下载的网页地址/download.php", map, charset);
    		//语言转换
    		Document doc = Jsoup.parse(httpOrgCreateTestRtn);
    		Elements  liElements = doc.getElementsByClass("table").get(0)
    				//.getElementsByAttribute("href");
    		.getElementsByTag("tr");
    
    		for (int i = 1; i < liElements.size(); i++) {
    			//String strurl = liElements.get(i).getElementsByAttribute("href").attr("href");
    
    			String geshi = liElements.get(i).getElementsByTag("td").get(2).text();
    			if ("MP4".equals(geshi)){
    				String strurl = liElements.get(i).getElementsByTag("td").get(3).getElementsByAttribute("href").attr("href");
    				if (strurl.length()>5){
    					//文件格式
    					String form = liElements.get(i).getElementsByTag("td").get(2).text();
    					//文件分辨率
    					String resolution = liElements.get(i).getElementsByTag("td").get(1).text();
    					Map limap = new HashMap();
    					limap.put("url",strurl);
    					limap.put("form",form);
    					limap.put("resolution",resolution);
    					urlList.add(limap);
    				}
    			}else {
    				String strurl = liElements.get(i).getElementsByTag("td").get(2).getElementsByAttribute("href").attr("href");
    				if (strurl.length()>5){
    					//文件格式
    					String form = liElements.get(i).getElementsByTag("td").get(1).text();
    					Map limap = new HashMap();
    					limap.put("url",strurl);
    					limap.put("form",form);
    					limap.put("resolution",null);
    					urlList.add(limap);
    				}
    			}
    
    
    
    		}
    		resultmap.put("list",urlList);
    		return resultmap;
    	}
    
    
    
    
    	public String doPost(String url, Map<String,String> map, String charset) {
    		org.apache.http.client.HttpClient httpClient = null;
    		HttpPost httpPost = null;
    		String result = null;
    		try {
    			httpClient = new SSLClient();
    			httpPost = new HttpPost(url);
    			//设置参数
    			List<NameValuePair> list = new ArrayList<NameValuePair>();
    			Iterator iterator = map.entrySet().iterator();
    			while (iterator.hasNext()) {
    				Map.Entry<String, String> elem = (Map.Entry<String, String>) iterator.next();
    				list.add(new BasicNameValuePair(elem.getKey(), elem.getValue()));
    			}
    			if (list.size() > 0) {
    				UrlEncodedFormEntity entity = new UrlEncodedFormEntity(list, charset);
    				entity.setContentType("application/x-www-form-urlencoded");
    				httpPost.setHeader("Accept", "*/*");
    				httpPost.setHeader("Content-type", "application/x-www-form-urlencoded;charset=utf-8");
    				httpPost.setEntity(entity);
    			}
    			HttpResponse response = httpClient.execute(httpPost);
    			if (response != null) {
    				HttpEntity resEntity = response.getEntity();
    				if (resEntity != null) {
    					result = EntityUtils.toString(resEntity, charset);
    				}
    			}
    		} catch (Exception ex) {
    			ex.printStackTrace();
    		}
    		return result;
    	}
    
    }

解释下代码:

在我分析下载也的请求时候，发现当我点击“下载”的时候，该页面发出的也是一个post请求

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTc4OTk4_size_16_color_FFFFFF_t_70 1][]

那么我就可以，用java代码模拟这个post请求了

String httpOrgCreateTestRtn = doPost("https://www.需要下载的网页地址/download.php", map, charset);
    		//语言转换
    		Document doc = Jsoup.parse(httpOrgCreateTestRtn);
    		Elements  liElements = doc.getElementsByClass("table").get(0)
    				//.getElementsByAttribute("href");
    		.getElementsByTag("tr");

这段就是用java代码，进行请求，返回的数据，我们用Jsoup转成Document格式，然后采用Elements对返回的html页面进行标签提取数据

最后我们处理好返回的数据，用map接受，存入list再进行返回

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTc4OTk4_size_16_color_FFFFFF_t_70 2][]

最后启动项目在swagger-ui界面，进行测试接口

请求后，![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTc4OTk4_size_16_color_FFFFFF_t_70 3][]

与我们在页面请求的结果相同

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTc4OTk4_size_16_color_FFFFFF_t_70 4][]

到此记录结束，欢迎评论与交流

[20210407100256606.png]: /images/20210708/7a91c329ed354a27af211e22e243ae33.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTc4OTk4_size_16_color_FFFFFF_t_70]: /images/20210708/5521575ab49942a3876c71f3daef64e6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTc4OTk4_size_16_color_FFFFFF_t_70 1]: /images/20210708/e64557f8b9714e449ecef777ad54e73a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTc4OTk4_size_16_color_FFFFFF_t_70 2]: /images/20210708/93c8f839ad67459c8766702119dd6977.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTc4OTk4_size_16_color_FFFFFF_t_70 3]: /images/20210708/9ad5fa63bcee4f7fa0b8fb9baa22bf77.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0MTc4OTk4_size_16_color_FFFFFF_t_70 4]: /images/20210708/1d5922f2792c49f5af18ee0226bf6d92.png