Node.js 实现简易爬虫

￡神魔★判官ぃ 2022-09-30 12:52 188阅读 0赞

为什么选择利用node来写爬虫呢？就是因为cheerio这个库，全兼容jQuery语法，熟悉的话用起来真真是爽

## 依赖选择 ##

*  [cheerio][]： Node.js 版的jQuery
 *  http：封装了一个HTPP服务器和一个简易的HTTP客户端
 *  [iconv-lite][]：解决爬取gb2312网页出现乱码

## 初步实现 ##

既然是要爬取网站内容，那我们就应该先去看看网站的基本构成  
选取的是[电影天堂][Link 1]作为目标网站，想要去爬取所有最新电影的下载链接

#### 分析页面 ####

页面结构如下：  
![ygdy.tiff][]

我们可以看到每个电影的标题都在一个`class`为`ulink`的`a`标签下，再往上定位，我们可以看到最外部的盒子`class`为`co_content8`

ok，可以开工了

#### 获取一页电影标题 ####

首先引入依赖，并设定需要爬取的url

核心代码 `index.js`

<table style="margin:15px 0px; width:755.594px; overflow:auto; display:block; border-spacing:0px; border-collapse:collapse"> 
 <tbody style=""> 
  <tr style="display:flex"> 
   <td style="display:block; overflow:auto; background:rgb(43,48,59); color:rgb(192,197,206); padding:16px; border:none; font-size:13px; line-height:19px"> <pre style="margin-top:0px; margin-bottom:0px">
     
      
     <div style="height:19px">
      
       
      http.get(url, 
      
       
      <span style="color:rgb(180,142,173)">function</span>(sres) {
     
      
     </div>
     
      
     <div style="height:19px">
      
       
        var chunks = [];
     
      
     </div>
     
      
     <div style="height:19px">
      
       
        sres.on(
      
       
      <span style="color:rgb(163,190,140)">'data'</span>, 
      
       
      <span style="color:rgb(180,142,173)">function</span>(chunk) {
     
      
     </div>
     
      
     <div style="height:19px">
      
       
          chunks.push(chunk);
     
      
     </div>
     
      
     <div style="height:19px">
      
       
        });
     
      
     </div>
     
      
     <div style="height:19px">
      
       
        // chunks里面存储着网页的 html 内容，将它zhuan ma传给 cheerio.load 之后
     
      
     </div>
     
      
     <div style="height:19px">
      
       
        // 就可以得到一个实现了 jQuery 接口的变量，将它命名为 `$`
     
      
     </div>
     
      
     <div style="height:19px">
      
       
        // 剩下就都是 jQuery 的内容了
     
      
     </div>
     
      
     <div style="height:19px">
      
       
        sres.on(
      
       
      <span style="color:rgb(163,190,140)">'end'</span>, 
      
       
      <span style=""><span style="color:rgb(143,161,179)">function</span></span>() {
     
      
     </div>
     
      
     <div style="height:19px">
      
       
          var titles = [];
     
      
     </div>
     
      
     <div style="height:19px">
      
       
          //由于咱们发现此网页的编码格式为gb2312，所以需要对其进行转码，否则乱码
     
      
     </div>
     
      
     <div style="height:19px">
      
       
          //依据：“&lt;meta http-equiv=
      
       
      <span style="color:rgb(163,190,140)">"Content-Type"</span> content=
      
       
      <span style="color:rgb(163,190,140)">"text/html; charset=gb2312"</span>&gt;”
     
      
     </div>
     
      
     <div style="height:19px">
      
       
          var html = iconv.decode(Buffer.concat(chunks), 
      
       
      <span style="color:rgb(163,190,140)">'gb2312'</span>);
     
      
     </div>
     
      
     <div style="height:19px">
      
       
          var $ = cheerio.load(html, {decodeEntities: 
      
       
      <span style="color:rgb(208,135,112)">false</span>});
     
      
     </div>
     
      
     <div style="height:19px">
      
       
          $(
      
       
      <span style="color:rgb(163,190,140)">'.co_content8 .ulink'</span>).each(
      
       
      <span style="color:rgb(180,142,173)">function</span> (idx, element) {
     
      
     </div>
     
      
     <div style="height:19px">
      
       
            var 
      
       
      <span style="color:rgb(191,97,106)">$element</span> = $(element);
     
      
     </div>
     
      
     <div style="height:19px">
      
       
            titles.push({
     
      
     </div>
     
      
     <div style="height:19px">
      
       
              title: 
      
       
      <span style="color:rgb(191,97,106)">$element</span>.text()
     
      
     </div>
     
      
     <div style="height:19px">
      
       
            })
     
      
     </div>
     
      
     <div style="height:19px">
      
       
          })    
     
      
     </div>
     
      
     <div style="height:19px">
      
       
          console.log(titles);     
     
      
     </div>
     
      
     <div style="height:19px">
      
       
        });
     
      
     </div>
     
      
     <div style="height:19px">
      
       
      });
     
      
     </div></pre> </td> 
  </tr> 
 </tbody> 
</table>

运行`node index`

结果如下  
![node-repfile-title.png][]

成功获取电影title，那如果我想获取多个页面的title呢，总不可能一个一个url去改吧。这当然有办法，请往下看！

#### 获取多页电影标题 ####

我们只要将之前的代码封装成一个函数并递归执行就完成了

核心代码 `index.js`

<table style="margin:15px 0px; width:755.594px; overflow:auto; display:block; border-spacing:0px; border-collapse:collapse"> 
 <tbody style=""> 
  <tr style="display:flex"> 
   <td style="display:block; overflow:auto; background:rgb(43,48,59); color:rgb(192,197,206); padding:16px; border:none; font-size:13px; line-height:19px"> <pre style="margin-top:0px; margin-bottom:0px">
     
      
     <div style="height:19px">
      
       
      var index = 
      
       
      <span style="color:rgb(208,135,112)">1</span>; 
      
       
      <span style="color:rgb(191,97,106)">//</span>页面数控制
     
      
     </div>
     
      
     <div style="height:19px">
      
       
      var url = 
      
       
      <span style="color:rgb(163,190,140)">'http://www.ygdy8.net/html/gndy/dyzz/list_23_'</span>;
     
      
     </div>
     
      
     <div style="height:19px">
      
       
      var titles = []; 
      
       
      <span style="color:rgb(191,97,106)">//</span>用于保存title
     
      
     </div>
     
      
     <div style="height:19px"></div>
     
      
     <div style="height:19px">
      
       
      function getTitle(url, i) {
     
      
     </div>
     
      
     <div style="height:19px">
      
       
        console.log(
      
       
      <span style="color:rgb(163,190,140)">"正在获取第"</span> + i + 
      
       
      <span style="color:rgb(163,190,140)">"页的内容"</span>); 
     
      
     </div>
     
      
     <div style="height:19px">
      
       
        http.get(url + i + 
      
       
      <span style="color:rgb(163,190,140)">'.html'</span>, function(sres) {
     
      
     </div>
     
      
     <div style="height:19px">
      
       
          var chunks = [];
     
      
     </div>
     
      
     <div style="height:19px">
      
       
          sres.on(
      
       
      <span style="color:rgb(163,190,140)">'data'</span>, function(chunk) {
     
      
     </div>
     
      
     <div style="height:19px">
      
       
            chunks.push(chunk);
     
      
     </div>
     
      
     <div style="height:19px">
      
       
          });
     
      
     </div>
     
      
     <div style="height:19px">
      
       
          sres.on(
      
       
      <span style="color:rgb(163,190,140)">'end'</span>, function() {
     
      
     </div>
     
      
     <div style="height:19px">
      
       
            var html = iconv.decode(Buffer.concat(chunks), 
      
       
      <span style="color:rgb(163,190,140)">'gb2312'</span>);
     
      
     </div>
     
      
     <div style="height:19px">
      
       
            var $ = cheerio.load(html, {
      
       
      <span style="color:rgb(163,190,140)">decodeEntities:</span> 
      
       
      <span style="color:rgb(208,135,112)">false</span>});
     
      
     </div>
     
      
     <div style="height:19px">
      
       
            $(
      
       
      <span style="color:rgb(163,190,140)">'.co_content8 .ulink'</span>).each(function (idx, element) {
     
      
     </div>
     
      
     <div style="height:19px">
      
       
              var $element = $(element);
     
      
     </div>
     
      
     <div style="height:19px">
      
       
              titles.push({
     
      
     </div>
     
      
     <div style="height:19px">          
      
       
      <span style="color:rgb(163,190,140)">title:</span> $element.text()
     
      
     </div>
     
      
     <div style="height:19px">
      
       
              })
     
      
     </div>
     
      
     <div style="height:19px">
      
       
            })  
     
      
     </div>
     
      
     <div style="height:19px">      
      
       
      <span style="color:rgb(180,142,173)">if</span>(i &lt; 
      
       
      <span style="color:rgb(208,135,112)">2</span>) { 
      
       
      <span style="color:rgb(191,97,106)">//</span>为了方便只爬了两页
     
      
     </div>
     
      
     <div style="height:19px">
      
       
              getTitle(url, ++index); 
      
       
      <span style="color:rgb(191,97,106)">//</span>递归执行，页数+
      
       
      <span style="color:rgb(208,135,112)">1</span>
     
      
     </div>
     
      
     <div style="height:19px">
      
       
            } 
      
       
      <span style="color:rgb(180,142,173)">else</span> {
     
      
     </div>
     
      
     <div style="height:19px">
      
       
              console.log(titles); 
     
      
     </div>
     
      
     <div style="height:19px">
      
       
              console.log(
      
       
      <span style="color:rgb(163,190,140)">"Title获取完毕！"</span>);              
     
      
     </div>
     
      
     <div style="height:19px">
      
       
            }
     
      
     </div>
     
      
     <div style="height:19px">
      
       
          });
     
      
     </div>
     
      
     <div style="height:19px">
      
       
        });
     
      
     </div>
     
      
     <div style="height:19px">
      
       
      }
     
      
     </div>
     
      
     <div style="height:19px"></div>
     
      
     <div style="height:19px">
      
       
      function main() {
     
      
     </div>
     
      
     <div style="height:19px">
      
       
        console.log(
      
       
      <span style="color:rgb(163,190,140)">"开始爬取"</span>);
     
      
     </div>
     
      
     <div style="height:19px">
      
       
        getTitle(url, index);
     
      
     </div>
     
      
     <div style="height:19px">
      
       
      }
     
      
     </div>
     
      
     <div style="height:19px"></div>
     
      
     <div style="height:19px">
      
       
      main(); 
      
       
      <span style="color:rgb(191,97,106)">//</span>运行主函数
     
      
     </div></pre> </td> 
  </tr> 
 </tbody> 
</table>

结果如下  
![node-reptitle-many-title.png][]

#### 获取电影下载连接 ####

如果是人工操作，我们需要一次操作，通过点击进入电影详情页才能找到下载地址  
那我们通过node如何来实现呢

常规先来分析页面布局  
![ygdy-des.png][]

我们如果想要准确定位到下载链接，需要先找到`id`为`Zoom`的div，下载链接就在这个`div`下的`tr`下的`a`标签内。

那我们就再定义一个函数，用于获取下载链接

getBtLink()

<table style="margin:15px 0px; width:755.594px; overflow:auto; display:block; border-spacing:0px; border-collapse:collapse"> 
 <tbody style=""> 
  <tr style="display:flex"> 
   <td style="display:block; overflow:auto; background:rgb(43,48,59); color:rgb(192,197,206); padding:16px; border:none; font-size:13px; line-height:19px"> <pre style="margin-top:0px; margin-bottom:0px">
     
      
     <div style="height:19px">
      
       
      <span style="color:rgb(180,142,173)">function</span> getBtLink(urls, n) { //urls里面包含着所有详情页的地址
     
      
     </div>
     
      
     <div style="height:19px">
      
       
        console.log(
      
       
      <span style="color:rgb(163,190,140)">"正在获取第"</span> + n + 
      
       
      <span style="color:rgb(163,190,140)">"个url的内容"</span>);
     
      
     </div>
     
      
     <div style="height:19px">
      
       
        http.get(
      
       
      <span style="color:rgb(163,190,140)">'http://www.ygdy8.net'</span> + urls[n].title, 
      
       
      <span style="color:rgb(180,142,173)">function</span>(sres) {
     
      
     </div>
     
      
     <div style="height:19px">
      
       
          var chunks = [];
     
      
     </div>
     
      
     <div style="height:19px">
      
       
          sres.on(
      
       
      <span style="color:rgb(163,190,140)">'data'</span>, 
      
       
      <span style="color:rgb(180,142,173)">function</span>(chunk) {
     
      
     </div>
     
      
     <div style="height:19px">
      
       
            chunks.push(chunk);
     
      
     </div>
     
      
     <div style="height:19px">
      
       
          });
     
      
     </div>
     
      
     <div style="height:19px">
      
       
          sres.on(
      
       
      <span style="color:rgb(163,190,140)">'end'</span>, 
      
       
      <span style=""><span style="color:rgb(143,161,179)">function</span></span>() {
     
      
     </div>
     
      
     <div style="height:19px">
      
       
            var html = iconv.decode(Buffer.concat(chunks), 
      
       
      <span style="color:rgb(163,190,140)">'gb2312'</span>); //进行转码
     
      
     </div>
     
      
     <div style="height:19px">
      
       
            var $ = cheerio.load(html, {decodeEntities: 
      
       
      <span style="color:rgb(208,135,112)">false</span>});
     
      
     </div>
     
      
     <div style="height:19px">
      
       
            $(
      
       
      <span style="color:rgb(163,190,140)">'#Zoom td'</span>).children(
      
       
      <span style="color:rgb(163,190,140)">'a'</span>).each(
      
       
      <span style="color:rgb(180,142,173)">function</span> (idx, element) {
     
      
     </div>
     
      
     <div style="height:19px">
      
       
              var 
      
       
      <span style="color:rgb(191,97,106)">$element</span> = $(element);
     
      
     </div>
     
      
     <div style="height:19px">
      
       
              btLink.push({
     
      
     </div>
     
      
     <div style="height:19px">
      
       
                bt: 
      
       
      <span style="color:rgb(191,97,106)">$element</span>.attr(
      
       
      <span style="color:rgb(163,190,140)">'href'</span>)
     
      
     </div>
     
      
     <div style="height:19px">
      
       
              })
     
      
     </div>
     
      
     <div style="height:19px">
      
       
            })
     
      
     </div>
     
      
     <div style="height:19px">      
      
       
      <span style="color:rgb(180,142,173)">if</span>(n &lt; urls.length - 1) {
     
      
     </div>
     
      
     <div style="height:19px">
      
       
              getBtLink(urls, ++count); /／递归
     
      
     </div>
     
      
     <div style="height:19px">
      
       
            } 
      
       
      <span style="color:rgb(180,142,173)">else</span> {
     
      
     </div>
     
      
     <div style="height:19px">
      
       
              console.log(
      
       
      <span style="color:rgb(163,190,140)">"btlink获取完毕！"</span>);
     
      
     </div>
     
      
     <div style="height:19px">
      
       
              console.log(btLink);   
     
      
     </div>
     
      
     <div style="height:19px">
      
       
            }
     
      
     </div>
     
      
     <div style="height:19px">
      
       
          });
     
      
     </div>
     
      
     <div style="height:19px">
      
       
        });
     
      
     </div>
     
      
     <div style="height:19px">
      
       
      }
     
      
     </div></pre> </td> 
  </tr> 
 </tbody> 
</table>

再次运行 `node index`  
![WX20170318-190537@2x.png][WX20170318-190537_2x.png]  
![WX20170318-190549@2x.png][WX20170318-190549_2x.png]

就这样我们将3个页面内所有电影的下载链接获取完毕，是不是很简单？

## 保存数据 ##

我们讲这些数据爬取出来当然是要进行保存的啊，在这里我选用了MongoDB来对其进行保存处理

数据保存函数 `save()`

这里的操作很简单，就没必要上mongoose啦  
再次运行 `node index`  
![mongodb-node-reptitle.png][]

这个Node.js实现的爬虫就是这样了，祝大家能爬到自己想要的数据；）

最后附上源码地址：[https://github.com/HuangXiZhou/node-reptitle][https_github.com_HuangXiZhou_node-reptitle]

个人建了前端学习群，旨在一起学习前端。纯净、纯粹技术讨论，非前端人员勿扰！入群加我[微信][Link 2]：iamaixiaoxiao。

![1460000008545203_w_276_h_261][]

[cheerio]: https://github.com/cheeriojs/cheerio
[iconv-lite]: https://www.npmjs.com/package/iconv-lite
[Link 1]: http://www.dytt8.net/
[ygdy.tiff]: http://ojiq40lzd.bkt.clouddn.com/WX20170318-202304@2x.png
[node-repfile-title.png]: http://ojiq40lzd.bkt.clouddn.com/node-repfile-title.png
[node-reptitle-many-title.png]: http://ojiq40lzd.bkt.clouddn.com/node-reptitle-many-title.png
[ygdy-des.png]: http://ojiq40lzd.bkt.clouddn.com/ygdy-des.png
[WX20170318-190537_2x.png]: http://ojiq40lzd.bkt.clouddn.com/WX20170318-190537@2x.png
[WX20170318-190549_2x.png]: http://ojiq40lzd.bkt.clouddn.com/WX20170318-190549@2x.png
[mongodb-node-reptitle.png]: http://ojiq40lzd.bkt.clouddn.com/mongodb-node-reptitle.png
[https_github.com_HuangXiZhou_node-reptitle]: https://github.com/HuangXiZhou/node-reptitle
[Link 2]: http://lib.csdn.net/base/wechat
[1460000008545203_w_276_h_261]: /images/20220705/0a56ba82010f4e8bb60ec4be9e90b743.png