selenium库的使用

迷南。 2022-10-01 02:49 180阅读 0赞

1.前言

一般的爬虫的是用python脚本直接对目标网站进行访问，访问速度很快，这样目标网站很容易就识别出你是机器人。而使用selenium写爬虫，python脚本操控浏览器进行访问，这样的行为更像是人类行为，这样很多难爬的网站也可以轻而易举的抓数据了。

2.准备

（1）Chrome用户

Chrome driver下载地址：

[http://chromedriver.storage.googleapis.com/index.html][http_chromedriver.storage.googleapis.com_index.html]

chrome和chrome driver对应版本：

<table> 
 <thead> 
  <tr> 
   <th align="center">chromedriver版本</th> 
   <th align="center">支持的Chrome版本</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td align="center">v2.28</td> 
   <td align="center">v55-57</td> 
  </tr> 
  <tr> 
   <td align="center">v2.27</td> 
   <td align="center">v54-56</td> 
  </tr> 
  <tr> 
   <td align="center">v2.26</td> 
   <td align="center">v53-55</td> 
  </tr> 
  <tr> 
   <td align="center">v2.25</td> 
   <td align="center">v53-55</td> 
  </tr> 
  <tr> 
   <td align="center">v2.24</td> 
   <td align="center">v52-54</td> 
  </tr> 
  <tr> 
   <td align="center">v2.23</td> 
   <td align="center">v51-53</td> 
  </tr> 
  <tr> 
   <td align="center">v2.22</td> 
   <td align="center">v49-52</td> 
  </tr> 
  <tr> 
   <td align="center">v2.21</td> 
   <td align="center">v46-50</td> 
  </tr> 
  <tr> 
   <td align="center">v2.20</td> 
   <td align="center">v43-48</td> 
  </tr> 
  <tr> 
   <td align="center">v2.19</td> 
   <td align="center">v43-47</td> 
  </tr> 
  <tr> 
   <td align="center">v2.18</td> 
   <td align="center">v43-46</td> 
  </tr> 
  <tr> 
   <td align="center">v2.17</td> 
   <td align="center">v42-43</td> 
  </tr> 
  <tr> 
   <td align="center">v2.13</td> 
   <td align="center">v42-45</td> 
  </tr> 
  <tr> 
   <td align="center">v2.15</td> 
   <td align="center">v40-43</td> 
  </tr> 
  <tr> 
   <td align="center">v2.14</td> 
   <td align="center">v39-42</td> 
  </tr> 
  <tr> 
   <td align="center">v2.13</td> 
   <td align="center">v38-41</td> 
  </tr> 
  <tr> 
   <td align="center">v2.12</td> 
   <td align="center">v36-40</td> 
  </tr> 
  <tr> 
   <td align="center">v2.11</td> 
   <td align="center">v36-40</td> 
  </tr> 
  <tr> 
   <td align="center">v2.10</td> 
   <td align="center">v33-36</td> 
  </tr> 
  <tr> 
   <td align="center">v2.9</td> 
   <td align="center">v31-34</td> 
  </tr> 
  <tr> 
   <td align="center">v2.8</td> 
   <td align="center">v30-33</td> 
  </tr> 
  <tr> 
   <td align="center">v2.7</td> 
   <td align="center">v30-33</td> 
  </tr> 
  <tr> 
   <td align="center">v2.6</td> 
   <td align="center">v29-32</td> 
  </tr> 
  <tr> 
   <td align="center">v2.5</td> 
   <td align="center">v29-32</td> 
  </tr> 
  <tr> 
   <td align="center">v2.4</td> 
   <td align="center">v29-32</td> 
  </tr> 
 </tbody> 
</table>

（2）Firefox用户

安装firebug，下载网址为[https://addons.mozilla.org/zh-CN/firefox/addon/firebug/][https_addons.mozilla.org_zh-CN_firefox_addon_firebug]

再安装firepath，下载网址为[https://addons.mozilla.org/zh-CN/firefox/addon/firepath/][https_addons.mozilla.org_zh-CN_firefox_addon_firepath]

selenium库从3.x开始，需要额外安装geckodriver下载网址为[https://github.com/mozilla/geckodriver/releases][https_github.com_mozilla_geckodriver_releases]下载解压后放在任何地方并将其路径加入path，个人推荐放在python的目录，这样省去了配置path的麻烦

![SouthEast][]

3.基本使用

**导包**

from selenium import webdriver

from bs4 import BeautifulSoup

**初始化浏览器**

driver = webdriver.Firefox()

**打开某个网址**

driver.get(url)

**如果网站需要输入登录账号密码**

**这里用到firepath找到目标位置的xpath**

**找到输入账号框，清除框内信息，再输入你的账号**

driver.find\_element\_by\_xpath(xpath).clear()

driver.find\_element\_by\_xpath(xpath).send\_keys("你的账号")

**找到输入密码框，清除框内信息，再输入你的密码**

driver.find\_element\_by\_xpath(xpath).clear()

driver.find\_element\_by\_xpath(xpath).send\_keys("你的密码")

**定位“点击登录”框的位置的xpath，执行登录**

driver.find\_element\_by\_xpath(xpath).click()

**获取该网页的源码**

html = driver.page\_source

**BeautifulSoup定位标签**

bsObj = BeautifulSoup（html，‘html.parser’）

再之后如何操作和普通爬虫一样了。

4.获取xpath

**前提**，确保已经安装好firefox，并且安装好插件firebug和firepath。

**首先，**点击图中1，打开firebug

**然后**，点击图中2，打开firepath

**再然后**，点击图中的3，箭头

**再然后**，点击目标位置，例如图中的4为账号输入框

**最后**，在图中5复制xpath的值

![SouthEast 1][]

5.示例代码

from selenium import webdriver
    from bs4 import BeautifulSoup
    
    def getHTMLText(url):
        driver = webdriver.Firefox()
        driver.get(url)
        userNameXpath = ".//*[@id='UserName']"
        driver.find_element_by_xpath(userNameXpath).clear()
        driver.find_element_by_xpath(userNameXpath).send_keys('替换为你的用户名')
        pasWrdXpath = ".//*[@id='Password']"
        driver.find_element_by_xpath(pasWrdXpath).clear()
        driver.find_element_by_xpath(pasWrdXpath).send_keys('替换为你的密码')
        loginXpath = ".//*[@id='main']/div[2]/div/form/div/fieldset/div[4]/input"
        driver.find_element_by_xpath(loginXpath).click()
        html = diver.page_source
        return html
    
    
    
    def main():
        url = 'http://www.ituring.com.cn/account/logon'
        html = getHTMLText(url)
    
    main()

运行后你就能看见爬虫自动打开浏览器并登陆账户啦。当前程序还不能处理验证码的操作，所以实验时选取没有验证码的网站。关于增加验证码的代码，将在后续补充。

[http_chromedriver.storage.googleapis.com_index.html]: http://chromedriver.storage.googleapis.com/index.html
[https_addons.mozilla.org_zh-CN_firefox_addon_firebug]: https://addons.mozilla.org/zh-CN/firefox/addon/firebug/
[https_addons.mozilla.org_zh-CN_firefox_addon_firepath]: https://addons.mozilla.org/zh-CN/firefox/addon/firepath/
[https_github.com_mozilla_geckodriver_releases]: https://github.com/mozilla/geckodriver/releases
[SouthEast]: /images/20220705/01195893b16547afbd5ac286be8eb8fa.png
[SouthEast 1]: /images/20220705/8f8389d710d24ed1b9ab89b276cd2831.png