写一个ieee的论文爬虫

我就是我 2023-02-26 04:27 35阅读 0赞

爬取的内容很简单，就是抓取当前期刊页中所有的论文，如红框所示，初始url就是像https://ieeexplore.ieee.org/xpl/conhome/8961330/proceeding 这样子的。

# 1、研究下网站结构 #

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70_pic_center]

emm，还在加载。很明显，下面的数据都是异步加载的。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70_pic_center 1]  
F12 ->Network->F5，复制个论文标题关键词全局搜索一下，即可找到对应的文件，我们想要的东西就应该在名为“toc”的这个数据包里了。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70]  
哈哈，论文下载链接啥的果然都在这。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 1]

任务很明确了，就抓这个json文件就行了。等等，url里后面那个数字是什么鬼？  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 2]  
全局搜一下康康  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 3]  
看样子这串数字的含义是“issue number”，处于“metadata”这个数据包里  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 4]  
看一下它的url，嗯，他的url也是很好获取的，那就先抓metadata，再根据其中的issueNumber抓toc数据包  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 5]

emmm，还有个问题下面的页码也是异步加载的，找找他在哪。。。  
嘶~，难找啊，懒得找了，不找了。后续在页面循环代码里加个异常处理好了。

# 2、好，思路清晰，目标明确，开搞！ #

## 2.1先把issueNumber抓下来 ##

![在这里插入图片描述][20200714115043256.png]  
咦，没东西？加个请求头试试。  
经过我不断地尝试在请求头里加入各种东西，发现加入referer这个参数后就能爬出东西来了，看样子这个网站的反爬机制主要是针对盗链恶意访问的。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 6]  
好，这样就ok了，再把json字符串转化成字典，就很容易地要找我们的issueNumber参数了。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 7]

## 2.2抓取论文标题及其链接 ##

有了上面的issueNumber，我们就可以组装这个toc文件的url了。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 8]  
嗯，不错，东西都在这了，后面就还是解析json字符串，得到文章标题与下载链接了。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 9]

## 2.3下载pdf ##

当然直接下载肯定是不可以的，得需要登录，这里我是校园ip的登陆方式，所以就不用写登陆代码了。

好，直接下载。  
![在这里插入图片描述][20200715094054774.png]  
打开瞧瞧，嗯？啥意思啊，不让下？  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 10]  
把链接粘到浏览器上也能看啊，没问题啊  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 11]  
嗯？等等，我好像搞错了。这特么是页面链接啊，下载链接那不是在右上角嘛  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 12]  
![在这里插入图片描述][20200715094656740.png]  
![在这里插入图片描述][20200715095001508.png_pic_center]

## 2.4代码整理一下，封装一下，先下载链接 ##

import requests
    from lxml import etree
    import json
    
    
    # 输入
    conferenceNum = 8961330 #会议编号
    # 输出
    # 此会议所有文章
    
    # 获取issueNumber
    def get_issueNumber(conferenceNum):
        conferenceNum = str(conferenceNum)
        gheaders = { 
        'Referer': 'https://ieeexplore.ieee.org/xpl/conhome/'+conferenceNum+'/proceeding',
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36'
        }
        md_url = 'https://ieeexplore.ieee.org/rest/publication/home/metadata?pubid='+conferenceNum
        md_res = requests.get(md_url, headers = gheaders)
        md_dic = json.loads(md_res.text)
        issueNumber = str(md_dic['currentIssue']['issueNumber'])
        return issueNumber
    
    # 爬取论文及其下载链接
    def get_article_info(conferenceNum, issueNumber):
        conferenceNum = str(conferenceNum)
        issueNumber = str(issueNumber)
        # 将论文名和下载链接存到txt中去
        alf = open(r'%s_%s_downloadLinks.txt'%(conferenceNum, issueNumber), 'w')
        
        # 从第一页开始下载
        pageNumber = 1
        while(True):
            toc_url = 'https://ieeexplore.ieee.org/rest/search/pub/'+conferenceNum+'/issue/'+issueNumber+'/toc'
            payload = '{"pageNumber":'+str(pageNumber)+',"punumber":"'+conferenceNum+'","isnumber":'+issueNumber+'}'
            headers = { 
            'Host': 'ieeexplore.ieee.org',
            'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36',
            'Referer': 'https://ieeexplore.ieee.org/xpl/conhome/'+conferenceNum+'/proceeding?pageNumber='+str(pageNumber),
            }
            toc_res = requests.post(toc_url, headers = headers, data=payload)
            toc_dic = json.loads(toc_res.text)
            try:
                articles = toc_dic['records']
            except KeyError:
                break
            else:
                for article in articles:
                    title = article['highlightedTitle']
                    link = 'https://ieeexplore.ieee.org/stampPDF/getPDF.jsp?tp=&arnumber='+article['articleNumber']+'&ref='
                    alf.write(title.replace('\n','')+'>_<'+link+'\n')
                pageNumber = pageNumber+1
                # 停一下防禁ip
                import time
                time.sleep(1)
        
        alf.close()
        return
    
    # start
    issueNumber = get_issueNumber(conferenceNum)
    get_article_info(conferenceNum, issueNumber)

最后执行结果，一共206条，一条不多一条不少。完全OK  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70_pic_center 2]

## 2.5 后下载论文 ##

链接都摆出来了，下载就简单了。下载代码包装亿点点。

def download_pdf(download_links_file, article_folder):
        with open(download_links_file, 'r') as f:
            article_download_link_list = f.readlines()
        article_download_link_list = [i.strip() for i in article_download_link_list if i is not '\n']
        print('共计',len(article_download_link_list), '篇文章。')
        a = input('是否立即下载？（是yes/否no）：')
        if a == 'yes':
            import os
            if not os.path.exists(article_folder):
                os.makedirs(article_folder)
            for article_download_link in tqdm(article_download_link_list):
                ad = article_download_link.split('>_<')
                article_name = ad[0]
                # 处理一下article_name，因为windows中不能用\ / : * ? " < > |作为文件名
                article_name = re.sub('[\/:*?"<>|]', '_', article_name)#去掉非法字符
                article_link = ad[1]
                r = requests.get(article_link)
                filename = article_folder+'/%s.pdf'%article_name
                with open(filename, 'wb+') as f:
                    f.write(r.content)
                # 停一下防禁ip
                import time
                time.sleep(1)
        elif a == 'no':
            return
        else:
            return
    
    download_links_file = '8961330_8970627_downloadLinks.txt'
    article_folder = '8961330_8970627_articles'
    download_pdf(download_links_file, article_folder)

这里我就懒得下了，206篇太多了，找了个少的期刊下了下，下完后就是这个样子。  
![在这里插入图片描述][202007141412292.png]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70_pic_center]: /images/20230209/f785128555214d49adfa4479543e25ae.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20230209/5c461b157cb34244af8449c3029cd747.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200714111743937.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200714112616691.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200714112759400.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20200714112944482.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20200714113049183.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20200714113147401.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0,size_16,color_FFFFFF,t_70
[20200714115043256.png]: https://img-blog.csdnimg.cn/20200714115043256.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20200714120556263.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/2020071412084877.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20200714122311917.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20200714122643338.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0,size_16,color_FFFFFF,t_70
[20200715094054774.png]: https://img-blog.csdnimg.cn/20200715094054774.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20200715083946866.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 11]: https://img-blog.csdnimg.cn/20200715094352336.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70 12]: https://img-blog.csdnimg.cn/2020071509462863.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0,size_16,color_FFFFFF,t_70
[20200715094656740.png]: https://img-blog.csdnimg.cn/20200715094656740.png
[20200715095001508.png_pic_center]: https://img-blog.csdnimg.cn/20200715095001508.png#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0_size_16_color_FFFFFF_t_70_pic_center 2]: https://img-blog.csdnimg.cn/20200715101057223.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4MjM3MjE0,size_16,color_FFFFFF,t_70#pic_center
[202007141412292.png]: https://img-blog.csdnimg.cn/202007141412292.png