使用selenium爬取QS世界大学综合排名与学科排名数据

你的名字 2024-04-03 12:01 80阅读 0赞

#### 一、概述： ####

*总共有56个学科排名，5个是"BROAD SUBJECT AREA"，还有51个是"SPECIFIC SUBJECT"，下面是代码的实现。每次启动的时候都需要在浏览器打开网页之后手动重新输入网页地址，不然会跳转到qsChina的网页。 打开网页后先要选Subject，也是利用selenium执行js脚本的方式模拟浏览器点击。先打开下拉框，再选择相应的Subject，然后根据选择的Subject创建Excel表格。接着在网页下方可以获取这个Subject的条目数量。然后切换到具体的指标页面，改变每页显示的条目数量（尽可能大，可以少翻页）。再之后就可以跟综合排名一样获取具体的数据了。每遍历一页之后要翻页。*

#### 二、实现代码： ####

# encoding=utf-8
    from selenium.webdriver.common.by import By
    from selenium.webdriver.chrome.service import Service
    from selenium import webdriver
    import time
    import xlsxwriter
    import math
    
    s = Service(r'C:\Users\myComputer\Desktop\chromedriver_win32\chromedriver.exe')
    driver = webdriver.Chrome(service=s)
    curl = 'https://www.topuniversities.com/university-rankings/university-subject-rankings/2022/arts-humanities'
    driver.get(curl)
    time.sleep(1)
    
    for i in range(1, 59, 1):
        # skip the border. "1" for "BROAD SUBJECT AREA"; "7" for "SPECIFIC SUBJECT"
        if i == 1 or i == 7:
            continue
        # change subject
        SubjectSel = driver.find_element(By.XPATH, '//*[@id="ranking-fillters"]/div[7]/div/div')
        driver.execute_script('arguments[0].click();', SubjectSel)
        time.sleep(1)
    
        Subject = driver.find_element(By.XPATH, '//*[@id="ranking-fillters"]/div[7]/div/div/div[2]/div[' + str(i) + ']')
        SubjectName = Subject.text
        driver.execute_script('arguments[0].click();', Subject)
        print('Select Subject: ' + SubjectName)
        time.sleep(1)
    
        # create sheet
        Workbook = xlsxwriter.Workbook(SubjectName + '.xlsx')
        Sheet = Workbook.add_worksheet()
    
        # get item total number
        itemNumber = driver.find_element(By.XPATH, '//*[@id="_totalcountresults"]')
        itemNum = int(itemNumber.text)
        print('Total Item count in ' + SubjectName + ': ' + itemNumber.text)
    
        # change tab to rank index
        rankInd = driver.find_element(By.XPATH,
                                      '//*[@id="block-tu-d8-content"]/div/article/div/div[3]/div/div[1]/div/div[1]/div/div/ul/li[2]/a')
        driver.execute_script('arguments[0].click();', rankInd)
        time.sleep(1)
        print('Change tab to Ranking Indicators')
    
        # change items number in every page
        dropdown = driver.find_element(By.XPATH,
                                       '//*[@id="block-tu-d8-content"]/div/article/div/div[3]/div/div[1]/div/div[3]/div[4]/div[1]/div[2]/i')
        driver.execute_script('arguments[0].click();', dropdown)
        time.sleep(1)
        itemsPerPage = driver.find_element(By.XPATH,
                                           '//*[@id="block-tu-d8-content"]/div/article/div/div[3]/div/div[1]/div/div[3]/div[4]/div[1]/div[2]/div[2]/div[4]')
        driver.execute_script('arguments[0].click();', itemsPerPage)
        time.sleep(1)
        print('Now there are 100 items in every page')
    
        # initial the table head
        Sheet.write(0, 0, 'Rank')
        Sheet.write(0, 1, 'University')
        Sheet.write(0, 2, 'Location')
        Sheet.write(0, 3, 'Overall Score')
        Sheet.write(0, 4, 'H-index Citations')
        Sheet.write(0, 5, 'Citations per Paper')
        Sheet.write(0, 6, 'Academic Reputation')
        Sheet.write(0, 7, 'Employer Reputation')
    
        CycleCnt = int(math.ceil(itemNum / 100))
    
        currentRow = 1
        for j in range(CycleCnt):
            k = 1
            while True:
                try:
                    eachItem = driver.find_element(By.XPATH, '//*[@id="ranking-data-load_ind"]/div[' + str(k) + ']')
                except:
                    break
                attr = eachItem.get_attribute('customblock')  # exclude ad
                time.sleep(0.5)
                if not attr:
                    subItem = driver.find_element(By.XPATH, '//*[@id="ranking-data-load_ind"]/div[' + str(
                        k) + ']/div/div/div/div[1]/div/div/div/div/div[1]/div')
                    Sheet.write(currentRow, 0, subItem.text)
                    subItem = driver.find_element(By.XPATH, '//*[@id="ranking-data-load_ind"]/div[' + str(
                        k) + ']/div/div/div/div[1]/div/div/div/div/div[2]/div/div[1]/div')
                    Sheet.write(currentRow, 1, subItem.text)
                    subItem = driver.find_element(By.XPATH, '//*[@id="ranking-data-load_ind"]/div[' + str(
                        k) + ']/div/div/div/div[1]/div/div/div/div/div[2]/div/div[2]')
                    Sheet.write(currentRow, 2, subItem.text)
                    for q in range(3, 8, 1):
                        subItem = driver.find_element(By.XPATH, '//*[@id="ranking-data-load_ind"]/div[' + str(
                            k) + ']/div/div/div/div[2]/div/div/div/div[' + str(q - 2) + ']')
                        Sheet.write(currentRow, q, subItem.text)
                    print(str(currentRow) + '/' + str(itemNum) + ' finished!')
                    currentRow = currentRow + 1
                k = k + 1
            # next page
            if j < CycleCnt - 1:
                q = 3
                while True:
                    try:
                        nextPage = driver.find_element(By.XPATH, '//*[@id="alt-style-pagination"]/li[' + str(q) + ']/a')
                    except:
                        q = q + 1
                        continue
                    attr = nextPage.get_attribute('class')
                    if attr == 'page-link next':
                        break
                    q = q + 1
                driver.execute_script('arguments[0].click();', nextPage)
                time.sleep(1)
                print('chaneg to page ' + str(j + 2))
    
        print('finish ' + SubjectName)
        Workbook.close()
    driver.close()

#### 三、Tips： ####

###### 1. 首先需要根据浏览器与浏览器版本下载对应的driver ######

Chrome：[http://chromedriver.storage.googleapis.com/index.html][http_chromedriver.storage.googleapis.com_index.html]  
Edge：[https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/][https_developer.microsoft.com_en-us_microsoft-edge_tools_webdriver]

###### 2. 可执行路径的方式已经不能使用，要替换为Service对象，否则会报DeprecationWarning错误 ######

之前的写法：（报错：DeprecationWarning: executable\_path has been deprecated, please pass in a Service object）

driver = webdriver.Chrome(r'C:\Users\myComputer\Desktop\chromedriver_win32\chromedriver.exe')	#这里添加的是driver的绝对路径

替换写法为：

s = Service(r'C:\Users\myComputer\Desktop\chromedriver_win32\chromedriver.exe')
    driver = webdriver.Chrome(service=s)

######  ######

持续更新中…

注：本文基于[QS世界大学综合排名与学科排名数据获取][QS]扩展补充

[http_chromedriver.storage.googleapis.com_index.html]: http://chromedriver.storage.googleapis.com/index.html
[https_developer.microsoft.com_en-us_microsoft-edge_tools_webdriver]: https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/
[QS]: https://blog.csdn.net/qq_35787848/article/details/121007644