Python之Selenium的爬虫用法

浅浅的花香味﹌ 2022-01-12 06:35 333阅读 0赞

Selenium 2，又名 WebDriver，它的主要新功能是集成了 Selenium 1.0 以及 WebDriver（WebDriver 曾经是 Selenium 的竞争对手）。也就是说 Selenium 2 是 Selenium 和 WebDriver 两个项目的合并，即 Selenium 2 兼容 Selenium，它既支持 Selenium API 也支持 WebDriver API。

更多详情可以查看 Webdriver 的简介。

[Webdriver][]

本文参考内容来自

[Selenium官网][Selenium] [SeleniumPython文档][SeleniumPython]

首先安装 Selenium

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>pip <span>install <span>selenium</span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

或者下载源码

[下载源码][Link 1]

然后解压后运行下面的命令进行安装

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>python <span>setup<span>.<span>py <span>install</span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

安装好了之后我们便开始探索抓取方法了。

## 快速开始 ##

### 初步体验 ###

我们先来一个小例子感受一下 Selenium，这里我们用 Chrome 浏览器来测试，方便查看效果，到真正爬取的时候换回 PhantomJS 即可。

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>from<span> <span>selenium <span>import<span> <span>webdriver</span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>browser<span> <span>=<span> <span>webdriver<span>.<span>Chrome<span>(<span>)</span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>browser<span>.<span>get<span>(<span>'http://www.baidu.com/'<span>)</span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

运行这段代码，会自动打开浏览器，然后访问百度。

如果程序执行错误，浏览器没有打开，那么应该是没有装 Chrome 浏览器或者 Chrome 驱动没有配置在环境变量里。下载驱动，然后将驱动文件路径配置在环境变量即可。

[浏览器驱动下载][Link 2]

比如是 Mac OS，就把下载好的文件放在 /usr/bin 目录下就可以了。

### 模拟提交 ###

下面的代码实现了模拟提交搜索的功能，首先等页面加载完成，然后输入到搜索框文本，点击提交。

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div> 
      <span>from<span> <span>selenium <span>import<span> <span>webdriver</span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>from<span> <span>selenium<span>.<span>webdriver<span>.<span>common<span>.<span>keys <span>import<span> <span>Keys</span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>driver<span> <span>=<span> <span>webdriver<span>.<span>Chrome<span>(<span>)</span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>driver<span>.<span>get<span>(<span>"http://www.python.org"<span>)</span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>assert<span> <span>"Python"<span> <span>in<span> <span>driver<span>.<span>title</span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>elem<span> <span>=<span> <span>driver<span>.<span>find_element_by_name<span>(<span>"q"<span>)</span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>elem<span>.<span>send_keys<span>(<span>"pycon"<span>)</span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>elem<span>.<span>send_keys<span>(<span>Keys<span>.<span>RETURN<span>)</span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>print<span> <span>driver<span>.<span>page_source</span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

同样是在 Chrome 里面测试，感受一下。

其中 driver.get 方法会打开请求的URL，WebDriver 会等待页面完全加载完成之后才会返回，即程序会等待页面的所有内容加载完成，JS渲染完毕之后才继续往下执行。注意：如果这里用到了特别多的 Ajax 的话，程序可能不知道是否已经完全加载完毕。

WebDriver 提供了许多寻找网页元素的方法，譬如 find\_element\_by\_\* 的方法。例如一个输入框可以通过 find\_element\_by\_name 方法寻找 name 属性来确定。

然后我们输入来文本然后模拟点击了回车，就像我们敲击键盘一样。我们可以利用 Keys 这个类来模拟键盘输入。

最后最重要的一点

**获取网页渲染后的源代码。**

输出 **page\_source** 属性即可。

这样，我们就可以做到网页的动态爬取了。

### 测试用例 ###

有了以上特性，我们当然可以用来写测试样例了。

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
     <div>
       5 
     </div> 
     <div>
       6 
     </div> 
     <div>
       7 
     </div> 
     <div>
       8 
     </div> 
     <div>
       9 
     </div> 
     <div>
       10 
     </div> 
     <div>
       11 
     </div> 
     <div>
       12 
     </div> 
     <div>
       13 
     </div> 
     <div>
       14 
     </div> 
     <div>
       15 
     </div> 
     <div>
       16 
     </div> 
     <div>
       17 
     </div> 
     <div>
       18 
     </div> 
     <div>
       19 
     </div> 
     <div>
       20 
     </div> 
     <div>
       21 
     </div> 
     <div>
       22 
     </div> 
     <div>
       23 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>import<span> <span>unittest</span></span></span> 
     </div> 
     <div> 
      <span>from<span> <span>selenium <span>import<span> <span>webdriver</span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>from<span> <span>selenium<span>.<span>webdriver<span>.<span>common<span>.<span>keys <span>import<span> <span>Keys</span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>class<span> <span>PythonOrgSearch<span>(<span>unittest<span>.<span>TestCase<span>)<span>:</span></span></span></span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;<span>def<span> <span>setUp<span>(<span>self<span>)<span>:</span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>self<span>.<span>driver<span> <span>=<span> <span>webdriver<span>.<span>Chrome<span>(<span>)</span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;<span>def<span> <span>test_search_in_python_org<span>(<span>self<span>)<span>:</span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>driver<span> <span>=<span> <span>self<span>.<span>driver</span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>driver<span>.<span>get<span>(<span>"http://www.python.org"<span>)</span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>self<span>.<span>assertIn<span>(<span>"Python"<span>,<span> <span>driver<span>.<span>title<span>)</span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>elem<span> <span>=<span> <span>driver<span>.<span>find_element_by_name<span>(<span>"q"<span>)</span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>elem<span>.<span>send_keys<span>(<span>"pycon"<span>)</span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>elem<span>.<span>send_keys<span>(<span>Keys<span>.<span>RETURN<span>)</span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>assert<span> <span>"No results found."<span> <span>not<span> <span>in<span> <span>driver<span>.<span>page_source</span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;<span>def<span> <span>tearDown<span>(<span>self<span>)<span>:</span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>self<span>.<span>driver<span>.<span>close<span>(<span>)</span></span></span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>if<span> <span>__name__<span> <span>==<span> <span>"__main__"<span>:</span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;<span>unittest<span>.<span>main<span>(<span>)</span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

运行程序，同样的功能，我们将其封装为测试标准类的形式。

测试用例是继承了 unittest.TestCase 类，继承这个类表明这是一个测试类。setUp方法是初始化的方法，这个方法会在每个测试类中自动调用。每一个测试方法命名都有规范，必须以 test 开头，会自动执行。最后的 tearDown 方法会在每一个测试方法结束之后调用。这相当于最后的析构方法。在这个方法里写的是 close 方法，你还可以写 quit 方法。不过 close 方法相当于关闭了这个 TAB 选项卡，然而 quit 是退出了整个浏览器。当你只开启了一个 TAB 选项卡的时候，关闭的时候也会将整个浏览器关闭。

## 页面操作 ##

### 页面交互 ###

仅仅抓取页面没有多大卵用，我们真正要做的是做到和页面交互，比如点击，输入等等。那么前提就是要找到页面中的元素。WebDriver提供了各种方法来寻找元素。例如下面有一个表单输入框。

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>&lt;<span>input <span>type<span>=<span>"text"<span> <span>name<span>=<span>"passwd"<span> <span>id<span>=<span>"passwd-id"<span> <span>/<span>&gt;</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

我们可以这样获取它

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>element<span> <span>=<span> <span>driver<span>.<span>find_element_by_id<span>(<span>"passwd-id"<span>)</span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>element<span> <span>=<span> <span>driver<span>.<span>find_element_by_name<span>(<span>"passwd"<span>)</span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>element<span> <span>=<span> <span>driver<span>.<span>find_elements_by_tag_name<span>(<span>"input"<span>)</span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>element<span> <span>=<span> <span>driver<span>.<span>find_element_by_xpath<span>(<span>"//input[@id='passwd-id']"<span>)</span></span></span></span></span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

你还可以通过它的文本链接来获取，但是要小心，文本必须完全匹配才可以，所以这并不是一个很好的匹配方式。

而且你在用 xpath 的时候还需要注意的是，如果有多个元素匹配了 xpath，它只会返回第一个匹配的元素。如果没有找到，那么会抛出 NoSuchElementException 的异常。

获取了元素之后，下一步当然就是向文本输入内容了，可以利用下面的方法

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>element<span>.<span>send_keys<span>(<span>"some text"<span>)</span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

同样你还可以利用 Keys 这个类来模拟点击某个按键。

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>element<span>.<span>send_keys<span>(<span>"and some"<span>,<span> <span>Keys<span>.<span>ARROW_DOWN<span>)</span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

你可以对任何获取到到元素使用 send\_keys 方法，就像你在 GMail 里面点击发送键一样。不过这样会导致的结果就是输入的文本不会自动清除。所以输入的文本都会在原来的基础上继续输入。你可以用下面的方法来清除输入文本的内容。

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>element<span>.<span>clear<span>(<span>)</span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

这样输入的文本会被清除。

### 填充表单 ###

我们已经知道了怎样向文本框中输入文字，但是其它的表单元素呢？例如下拉选项卡的的处理可以如下

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
     <div>
       5 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>element<span> <span>=<span> <span>driver<span>.<span>find_element_by_xpath<span>(<span>"//select[@name='name']"<span>)</span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>all_options<span> <span>=<span> <span>element<span>.<span>find_elements_by_tag_name<span>(<span>"option"<span>)</span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>for<span> <span>option <span>in<span> <span>all_options<span>:</span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;<span>print<span>(<span>"Value is: %s"<span> <span>%<span> <span>option<span>.<span>get_attribute<span>(<span>"value"<span>)<span>)</span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;<span>option<span>.<span>click<span>(<span>)</span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

首先获取了第一个 select 元素，也就是下拉选项卡。然后轮流设置了 select 选项卡中的每一个 option 选项。你可以看到，这并不是一个非常有效的方法。

其实 WebDriver 中提供了一个叫 Select 的方法，可以帮助我们完成这些事情。

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
     <div>
       5 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>from<span> <span>selenium<span>.<span>webdriver<span>.<span>support<span>.<span>ui <span>import<span> <span>Select</span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>select<span> <span>=<span> <span>Select<span>(<span>driver<span>.<span>find_element_by_name<span>(<span>'name'<span>)<span>)</span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>select<span>.<span>select_by_index<span>(<span>index<span>)</span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>select<span>.<span>select_by_visible_text<span>(<span>"text"<span>)</span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>select<span>.<span>select_by_value<span>(<span>value<span>)</span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

如你所见，它可以根据索引来选择，可以根据值来选择，可以根据文字来选择。是十分方便的。

全部取消选择怎么办呢？很简单

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>select<span> <span>=<span> <span>Select<span>(<span>driver<span>.<span>find_element_by_id<span>(<span>'id'<span>)<span>)</span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>select<span>.<span>deselect_all<span>(<span>)</span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

这样便可以取消所有的选择。

另外我们还可以通过下面的方法获取所有的已选选项。

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>select<span> <span>=<span> <span>Select<span>(<span>driver<span>.<span>find_element_by_xpath<span>(<span>"xpath"<span>)<span>)</span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>all_selected_options<span> <span>=<span> <span>select<span>.<span>all_selected_options</span></span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

获取所有可选选项是

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>options<span> <span>=<span> <span>select<span>.<span>options</span></span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

如果你把表单都填好了，最后肯定要提交表单对吧。怎吗提交呢？很简单

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>driver<span>.<span>find_element_by_id<span>(<span>"submit"<span>)<span>.<span>click<span>(<span>)</span></span></span></span></span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

这样就相当于模拟点击了 submit 按钮，做到表单提交。

当然你也可以单独提交某个元素

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>element<span>.<span>submit<span>(<span>)</span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

方法，WebDriver 会在表单中寻找它所在的表单，如果发现这个元素并没有被表单所包围，那么程序会抛出 NoSuchElementException 的异常。

### 元素拖拽 ###

要完成元素的拖拽，首先你需要指定被拖动的元素和拖动目标元素，然后利用 ActionChains 类来实现。

Python

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
     <div>
       5 
     </div> 
     <div>
       6 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>element<span> <span>=<span> <span>driver<span>.<span>find_element_by_name<span>(<span>"source"<span>)</span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>target<span> <span>=<span> <span>driver<span>.<span>find_element_by_name<span>(<span>"target"<span>)</span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>from<span> <span>selenium<span>.<span>webdriver <span>import<span> <span>ActionChains</span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>action_chains<span> <span>=<span> <span>ActionChains<span>(<span>driver<span>)</span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>action_chains<span>.<span>drag_and_drop<span>(<span>element<span>,<span> <span>target<span>)<span>.<span>perform<span>(<span>)</span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

这样就实现了元素从 source 拖动到 target 的操作。

### 页面切换 ###

一个浏览器肯定会有很多窗口，所以我们肯定要有方法来实现窗口的切换。切换窗口的方法如下

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>driver<span>.<span>switch_to_window<span>(<span>"windowName"<span>)</span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

另外你可以使用 window\_handles 方法来获取每个窗口的操作对象。例如

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>for<span> <span>handle <span>in<span> <span>driver<span>.<span>window_handles<span>:</span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;<span>driver<span>.<span>switch_to_window<span>(<span>handle<span>)</span></span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

另外切换 frame 的方法如下

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>driver<span>.<span>switch_to_frame<span>(<span>"frameName.0.child"<span>)</span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

这样焦点会切换到一个 name 为 child 的 frame 上。

### 弹窗处理 ###

当你出发了某个事件之后，页面出现了弹窗提示，那么你怎样来处理这个提示或者获取提示信息呢？

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>alert<span> <span>=<span> <span>driver<span>.<span>switch_to_alert<span>(<span>)</span></span></span></span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

通过上述方法可以获取弹窗对象。

### 历史记录 ###

那么怎样来操作页面的前进和后退功能呢？

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>driver<span>.<span>forward<span>(<span>)</span></span></span></span></span> 
     </div> 
     <div> 
      <span>driver<span>.<span>back<span>(<span>)</span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

嗯，简洁明了。

### Cookies处理 ###

为页面添加 Cookies，用法如下

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
     <div>
       5 
     </div> 
     <div>
       6 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span># Go to the correct domain</span> 
     </div> 
     <div> 
      <span>driver<span>.<span>get<span>(<span>"http://www.example.com"<span>)</span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span># Now set the cookie. This one's valid for the entire domain</span> 
     </div> 
     <div> 
      <span>cookie<span> <span>=<span> <span>{‘<span>name’<span> <span>:<span> ‘<span>foo’<span>,<span> ‘<span>value’<span> <span>:<span> ‘<span>bar’<span>}</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>driver<span>.<span>add_cookie<span>(<span>cookie<span>)</span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

获取页面 Cookies，用法如下

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
     <div>
       5 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span># Go to the correct domain</span> 
     </div> 
     <div> 
      <span>driver<span>.<span>get<span>(<span>"http://www.example.com"<span>)</span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span># And now output all the available cookies for the current URL</span> 
     </div> 
     <div> 
      <span>driver<span>.<span>get_cookies<span>(<span>)</span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

以上便是 Cookies 的处理，同样是非常简单的。

## 元素选取 ##

关于元素的选取，有如下的API  
单个元素选取

>  *  find\_element\_by\_id
>  *  find\_element\_by\_name
>  *  find\_element\_by\_xpath
>  *  find\_element\_by\_link\_text
>  *  find\_element\_by\_partial\_link\_text
>  *  find\_element\_by\_tag\_name
>  *  find\_element\_by\_class\_name
>  *  find\_element\_by\_css\_selector

多个元素选取

>  *  find\_elements\_by\_name
>  *  find\_elements\_by\_xpath
>  *  find\_elements\_by\_link\_text
>  *  find\_elements\_by\_partial\_link\_text
>  *  find\_elements\_by\_tag\_name
>  *  find\_elements\_by\_class\_name
>  *  find\_elements\_by\_css\_selector

另外还可以利用 By 类来确定哪种选择方式

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>from<span> <span>selenium<span>.<span>webdriver<span>.<span>common<span>.<span>by <span>import<span> <span>By</span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>driver<span>.<span>find_element<span>(<span>By<span>.<span>XPATH<span>,<span> <span>'//button[text()="Some text"]'<span>)</span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>driver<span>.<span>find_elements<span>(<span>By<span>.<span>XPATH<span>,<span> <span>'//button'<span>)</span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

By 类的一些属性如下

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
     <div>
       5 
     </div> 
     <div>
       6 
     </div> 
     <div>
       7 
     </div> 
     <div>
       8 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>ID<span> <span>=<span> <span>"id"</span></span></span></span></span> 
     </div> 
     <div> 
      <span>XPATH<span> <span>=<span> <span>"xpath"</span></span></span></span></span> 
     </div> 
     <div> 
      <span>LINK_TEXT<span> <span>=<span> <span>"link text"</span></span></span></span></span> 
     </div> 
     <div> 
      <span>PARTIAL_LINK_TEXT<span> <span>=<span> <span>"partial link text"</span></span></span></span></span> 
     </div> 
     <div> 
      <span>NAME<span> <span>=<span> <span>"name"</span></span></span></span></span> 
     </div> 
     <div> 
      <span>TAG_NAME<span> <span>=<span> <span>"tag name"</span></span></span></span></span> 
     </div> 
     <div> 
      <span>CLASS_NAME<span> <span>=<span> <span>"class name"</span></span></span></span></span> 
     </div> 
     <div> 
      <span>CSS_SELECTOR<span> <span>=<span> <span>"css selector"</span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

更详细的元素选择方法参见官方文档

## 页面等待 ##

这是非常重要的一部分，现在的网页越来越多采用了 Ajax 技术，这样程序便不能确定何时某个元素完全加载出来了。这会让元素定位困难而且会提高产生 ElementNotVisibleException 的概率。

所以 Selenium 提供了两种等待方式，一种是隐式等待，一种是显式等待。

隐式等待是等待特定的时间，显式等待是指定某一条件直到这个条件成立时继续执行。

### 显式等待 ###

显式等待指定某个条件，然后设置最长等待时间。如果在这个时间还没有找到元素，那么便会抛出异常了。

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
     <div>
       5 
     </div> 
     <div>
       6 
     </div> 
     <div>
       7 
     </div> 
     <div>
       8 
     </div> 
     <div>
       9 
     </div> 
     <div>
       10 
     </div> 
     <div>
       11 
     </div> 
     <div>
       12 
     </div> 
     <div>
       13 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>from<span> <span>selenium <span>import<span> <span>webdriver</span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>from<span> <span>selenium<span>.<span>webdriver<span>.<span>common<span>.<span>by <span>import<span> <span>By</span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>from<span> <span>selenium<span>.<span>webdriver<span>.<span>support<span>.<span>ui <span>import<span> <span>WebDriverWait</span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>from<span> <span>selenium<span>.<span>webdriver<span>.<span>support <span>import<span> <span>expected_conditions <span>as<span> <span>EC</span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>driver<span> <span>=<span> <span>webdriver<span>.<span>Chrome<span>(<span>)</span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>driver<span>.<span>get<span>(<span>"http://somedomain/url_that_delays_loading"<span>)</span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>try<span>:</span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;<span>element<span> <span>=<span> <span>WebDriverWait<span>(<span>driver<span>,<span> <span>10<span>)<span>.<span>until<span>(</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>EC<span>.<span>presence_of_element_located<span>(<span>(<span>By<span>.<span>ID<span>,<span> <span>"myDynamicElement"<span>)<span>)</span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;<span>)</span></span> 
     </div> 
     <div> 
      <span>finally<span>:</span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;<span>driver<span>.<span>quit<span>(<span>)</span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

程序默认会 500ms 调用一次来查看元素是否已经生成，如果本来元素就是存在的，那么会立即返回。

下面是一些内置的等待条件，你可以直接调用这些条件，而不用自己写某些等待条件了。

>  *  title\_is
>  *  title\_contains
>  *  presence\_of\_element\_located
>  *  visibility\_of\_element\_located
>  *  visibility\_of
>  *  presence\_of\_all\_elements\_located
>  *  text\_to\_be\_present\_in\_element
>  *  text\_to\_be\_present\_in\_element\_value
>  *  frame\_to\_be\_available\_and\_switch\_to\_it
>  *  invisibility\_of\_element\_located
>  *  element\_to\_be\_clickable – it is Displayed and Enabled.
>  *  staleness\_of
>  *  element\_to\_be\_selected
>  *  element\_located\_to\_be\_selected
>  *  element\_selection\_state\_to\_be
>  *  element\_located\_selection\_state\_to\_be
>  *  alert\_is\_present

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>from<span> <span>selenium<span>.<span>webdriver<span>.<span>support <span>import<span> <span>expected_conditions <span>as<span> <span>EC</span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>wait<span> <span>=<span> <span>WebDriverWait<span>(<span>driver<span>,<span> <span>10<span>)</span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>element<span> <span>=<span> <span>wait<span>.<span>until<span>(<span>EC<span>.<span>element_to_be_clickable<span>(<span>(<span>By<span>.<span>ID<span>,<span>'someid'<span>)<span>)<span>)</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

### 隐式等待 ###

隐式等待比较简单，就是简单地设置一个等待时间，单位为秒。

Python

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
     <div>
       5 
     </div> 
     <div>
       6 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>from<span> <span>selenium <span>import<span> <span>webdriver</span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>driver<span> <span>=<span> <span>webdriver<span>.<span>Chrome<span>(<span>)</span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>driver<span>.<span>implicitly_wait<span>(<span>10<span>)<span> <span># seconds</span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>driver<span>.<span>get<span>(<span>"http://somedomain/url_that_delays_loading"<span>)</span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>myDynamicElement<span> <span>=<span> <span>driver<span>.<span>find_element_by_id<span>(<span>"myDynamicElement"<span>)</span></span></span></span></span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

当然如果不设置，默认等待时间为0。

## 结语 ##

以上就是 Selenium 的基本用法，我们讲解了页面交互，页面渲染之后的源代码的获取。这样，即使页面是 JS 渲染而成的，我们也可以手到擒来了

转载于:https://www.cnblogs.com/dengshihuang/p/7943477.html

[Webdriver]: http://www.51testing.com/zhuanti/webdriver.htm
[Selenium]: http://www.seleniumhq.org/docs/index.jsp
[SeleniumPython]: http://selenium-python.readthedocs.org/index.html
[Link 1]: https://pypi.python.org/pypi/selenium
[Link 2]: https://sites.google.com/a/chromium.org/chromedriver/downloads