Python通过urllib访问网页

水深无声 2023-10-17 22:26 30阅读 0赞

通过urllib.request.urlopen()函数就可以访问网页了

>>> import urllib.request

>>> response =urllib.request.urlopen("[http://www.finsh.com][http_www.finsh.com]")

>>> html =response.read()

>>> print(html)

--------------------

b'<!DOCTYPE html><html itemscope="" itemtype="[http://schema.org/WebSite"><head><link][http_schema.org_WebSite_head_link] rel="stylesheet" media="screen" href="[http://assets0.undeveloped.com/assets/application-50d340b4770955a3305bd75703b1fbd40448de80151abddce4336475a5959192.css][http_assets0.undeveloped.com_assets_application-50d340b4770955a3305bd75703b1fbd40448de80151abddce4336475a5959192.css]" /><link href=\\'http://assets2.undeveloped.com/assets/icons/favicon-db66290d1eb239acedfed4e94028fc8697ba119452f28461f3c40d517d006f3d.ico\\' rel=\\'icon\\' type=\\'image/x-icon\\'>\\n<link href=\\'http://assets3.undeveloped.com/assets/icons/touch-icon-iphone-cef164b4d6d29e1bff2bad9e49abaf143593a07d8a6e584f472b545b9e0c5631.png\\' rel=\\'apple-touch-icon\\'>\\n<link href=\\'http://assets1.undeveloped.com/assets/icons/touch-icon-ipad-7e9822ba1e8fa34ce37262f6746dbc72819d754f805a410dbeb2cedb08a05789.png\\' rel=\\'apple-touch-icon\\' sizes=\\'76x76\\'>\\n<link href=\\'http://assets3.undeveloped.com/assets/icons/touch-icon-iphone-retina-42b99bfa7f037e7773362a76bf1ee70632541e4054f9003ce6f190dc9070fd8d.png\\' rel=\\'apple-touch-icon\\' sizes=\\'120x120\\'>\\n<link href=\\'http://assets2.undeveloped.com/assets/icons/touch-icon-ipad-retina-56a8f2519ed554a55e6084d77c1ab7ef3511f8ed5e7877db50a9865621a79290.png\\' rel=\\'apple-touch-icon\\' sizes=\\'152x152\\'>\\n<meta charset="utf-8" /><meta content="IE=edge" http-equiv="X-UA-Compatible" /><meta content="width=device-width, minimum-scale=1, initial-scale=1" name="viewport" /><meta name="csrf-param" content="authenticity\_token" />\\n<meta name="csrf-token" content="TWquJFnvGbztzfhFmUHFn7fuWGDnd/hNQMqWkGhKDMlFh1bwgIC0f+eUpJ5DgeFbUCcjMOCPmFuBXAVPExHzBQ==" /><title>The domain name finsh.com is for sale | Undeveloped</title><meta content="The domain name finsh.com is for sale. Make an offer or buy it now at a set price. Undeveloped keeps you safe." name="description" /><meta content="nofollow" name="robots" /><link href="[http://finsh.com][http_finsh.com]" rel="canonical" /><link href="[http://finsh.com][http_finsh.com]" hreflang="en-us" itemprop="url" rel="alternate" /><link href="[http://finsh.com/nl-nl][http_finsh.com_nl-nl]" hreflang="nl-nl" itemprop="" rel="alternate" /><link href="[http://finsh.com/es-es][http_finsh.com_es-es]" hreflang="es-es" itemprop="" rel="alternate" /><link href="[http://finsh.com/hi-in][http_finsh.com_hi-in]" hreflang="hi-in" itemprop="" rel="alternate" /><link href="[http://finsh.com/de-de][http_finsh.com_de-de]" hreflang="de-de" itemprop="" rel="alternate" /><link href="[http://finsh.com/fr-fr][http_finsh.com_fr-fr]" hreflang="fr-fr" itemprop="" rel="alternate" /><link href="[http://finsh.com/zh-cn][http_finsh.com_zh-cn]" hreflang="zh-cn" itemprop="" rel="alternate" /><meta content="finsh.com - Domain Name For Sale | Undeveloped" property="og:title" /><meta content="product" property="og:type" /><meta content="[http://finsh.com/][http_finsh.com]" property="og:url" /><meta content="I found a great domain name for sale. Check it out!" property="og:description" /><meta content="Undeveloped" property="og:site\_name" /><meta content="4995" property="og:price:amount" /><meta content="USD" property="og:price:currency" /><meta content="summary" name="twitter:card" /><meta content="@undeveloped" name="twitter:site" /><meta content="finsh.com - Domain Name For Sale | Undeveloped" name="twitter:title" /><meta content="I found a great domain name for sale. Check it out!" name="twitter:description" /><meta content="@undeveloped" name="twitter:creator" /></head><body><div itemprop="author" itemscope="" itemtype="[http://schema.org/Organization][http_schema.org_Organization]" style="display:none!important;"><meta content="Undeveloped" itemprop="name" name="author" /><meta content="[http://assets1.undeveloped.com/assets/logos/logo-undeveloped-gray-41752f2131b3c81f59add0f28168beb14da3e422562addda9348e3ea2b53ae8e.png][http_assets1.undeveloped.com_assets_logos_logo-undeveloped-gray-41752f2131b3c81f59add0f28168beb14da3e422562addda9348e3ea2b53ae8e.png]" itemprop="logo" /></div><header class="headroom"><nav class="navbar navbar-default navbar-parking"><div class="navbar-header"><div class="col-xs-12"><a target="\_blank" class="navbar-brand logo-with-text" href="[https://undeveloped.com/?utm\_campaign=Parking&utm\_medium=parking&utm\_source=.com&utm\_term=finsh.com"><img][https_undeveloped.com_utm_campaign_Parking_utm_medium_parking_utm_source_.com_utm_term_finsh.com_img] class="img-responsive" alt="Undeveloped" src="[http://assets0.undeveloped.com/assets/logos/logo-undeveloped-gray-safe-d2bbdd8aeb867666a6399c0a26866e77e46deee61af0b5622822ab6f7ed80149.png][http_assets0.undeveloped.com_assets_logos_logo-undeveloped-gray-safe-d2bbdd8aeb867666a6399c0a26866e77e46deee61af0b5622822ab6f7ed80149.png]" /></a></div><div class="col-xs-12 p-t-1 p-x-0 m-x-0 visible-xs"> <div class="trustpilot-widget" data-locale="en-US" data-template-id="5418015fb0d04a0c9cf721f2" data-businessunit-id="5aa29819243d5a0001f9665a"data-style-height="30px" data-style-width="100%" data-theme="light" data-stars="1,2,3,4,5"><ahref="[https://www.trustpilot.com/review/undeveloped.com"target="\_blank"></a][https_www.trustpilot.com_review_undeveloped.com_target_blank_a]> </div> \\n</div></div><div class="pull-right p-y-2 p-x-2 hidden-xs"> <div class="trustpilot-widget" data-locale="en-US" data-template-id="5418015fb0d04a0c9cf721f2" data-businessunit-id="5aa29819243d5a0001f9665a"data-style-height="30px" data-style-width="100%" data-theme="light" data-stars="1,2,3,4,5"><ahref="[https://www.trustpilot.com/review/undeveloped.com"target="\_blank"></a][https_www.trustpilot.com_review_undeveloped.com_target_blank_a]> </div> \\n</div></nav></header><nav class="navbar navbar-default"></nav><div class="full-page"><div itemscope="" itemtype="[http://schema.org/Product][http_schema.org_Product]" style="display:none!important;"><meta content="finsh.com" itemprop="name" /><meta content=".com" itemprop="category" /><meta content="[http://finsh.com/][http_finsh.com]" itemprop="url" /><div itemprop="offers" itemscope="" itemtype="[http://schema.org/Offer"><meta][http_schema.org_Offer_meta] content="finsh.com" itemprop="name" /><meta content=".com" itemprop="category" /><meta content="[http://finsh.com/][http_finsh.com]" itemprop="url" /><meta href="[http://schema.org/InStock][http_schema.org_InStock]" itemprop="availability" /><meta content="4995" itemprop="price" /><meta content="USD" itemprop="priceCurrency" /><div itemscope="" itemtype="[http://schema.org/Organization"><meta][http_schema.org_Organization_meta] content="Mega Domains" itemprop="name" /><meta content="[https://undeveloped.com/domain-seller/mega-domains-c6335228-31a9-4e05-be5e-a8a646cc65fe][https_undeveloped.com_domain-seller_mega-domains-c6335228-31a9-4e05-be5e-a8a646cc65fe]" itemprop="url" /></div></div></div><script>window.currencyRates = \{"ISK":107.569888,"AUD":1.351703,"BGN":1.6757,"BRL":3.769397,"CAD":1.32954,"CHF":0.99022,"CNY":6.575796,"CZK":22.199499,"DKK":6.38507,"GBP":0.75567,"HKD":7.849198,"HRK":6.3255,"HUF":278.910004,"IDR":14173.0,"ILS":3.631606,"INR":68.333994,"JPY":109.904999,"KRW":1117.630005,"MXN":19.860701,"MYR":4.020977,"NOK":8.114299,"NZD":1.456602,"PHP":53.630001,"PLN":3.711201,"RON":3.993102,"RUB":63.108101,"SEK":8.86245,"SGD":1.36251,"THB":32.990002,"TRY":4.618006,"ZAR":13.531697,"EUR":0.856699\};</script><input id="domain-price" type="hidden" value="4995" /><input id="domain-currency" type="hidden" value="USD" /><script type="application/json" id="js-react-on-rails-context">\{"inMailer":false,"i18nLocale":"en-us","i18nDefaultLocale":"en-us","href":"<a href="http://finsh.com/%22,%22location%22:%22/%22,%22scheme%22:%22http%22,%22host%22:%22finsh.com%22,%22port%22:null,%22pathname%22:%22/%22,%22search%22:null,%22httpAcceptLanguage%22:null,%22serverSide%22:false%7D%3C/script%3E/n%3Cscript" designtimeurl="'http://finsh.com/" script="" ?,?search?:null,?httpacceptlanguage?:null,?serverside?:false\}<="" ?,?scheme?:?http?,?host?:?finsh.com?,?port?:null,?pathname?:?="" ,?location?:?="">\\n<script'" designtimesp="14403">\\n

........

Python爬取的是以utf-8编码的bytes对象，要还原为带中文的html代码，需要对其进行解码，将它变成Unicode编码

>>> html=html.decode("utf-8")  
  
>>> print(html)

[http_www.finsh.com]: http://www.finsh.com/
[http_schema.org_WebSite_head_link]: http://schema.org/WebSite%22%3E%3Chead%3E%3Clink
[http_assets0.undeveloped.com_assets_application-50d340b4770955a3305bd75703b1fbd40448de80151abddce4336475a5959192.css]: http://assets0.undeveloped.com/assets/application-50d340b4770955a3305bd75703b1fbd40448de80151abddce4336475a5959192.css
[http_finsh.com]: http://finsh.com/
[http_finsh.com_nl-nl]: http://finsh.com/nl-nl
[http_finsh.com_es-es]: http://finsh.com/es-es
[http_finsh.com_hi-in]: http://finsh.com/hi-in
[http_finsh.com_de-de]: http://finsh.com/de-de
[http_finsh.com_fr-fr]: http://finsh.com/fr-fr
[http_finsh.com_zh-cn]: http://finsh.com/zh-cn
[http_schema.org_Organization]: http://schema.org/Organization
[http_assets1.undeveloped.com_assets_logos_logo-undeveloped-gray-41752f2131b3c81f59add0f28168beb14da3e422562addda9348e3ea2b53ae8e.png]: http://assets1.undeveloped.com/assets/logos/logo-undeveloped-gray-41752f2131b3c81f59add0f28168beb14da3e422562addda9348e3ea2b53ae8e.png
[https_undeveloped.com_utm_campaign_Parking_utm_medium_parking_utm_source_.com_utm_term_finsh.com_img]: https://undeveloped.com/?utm_campaign=Parking&amp;utm_medium=parking&amp;utm_source=.com&amp;utm_term=finsh.com%22%3E%3Cimg
[http_assets0.undeveloped.com_assets_logos_logo-undeveloped-gray-safe-d2bbdd8aeb867666a6399c0a26866e77e46deee61af0b5622822ab6f7ed80149.png]: http://assets0.undeveloped.com/assets/logos/logo-undeveloped-gray-safe-d2bbdd8aeb867666a6399c0a26866e77e46deee61af0b5622822ab6f7ed80149.png
[https_www.trustpilot.com_review_undeveloped.com_target_blank_a]: https://www.trustpilot.com/review/undeveloped.com%22target=%22_blank%22%3E%3C/a
[http_schema.org_Product]: http://schema.org/Product
[http_schema.org_Offer_meta]: http://schema.org/Offer%22%3E%3Cmeta
[http_schema.org_InStock]: http://schema.org/InStock
[http_schema.org_Organization_meta]: http://schema.org/Organization%22%3E%3Cmeta
[https_undeveloped.com_domain-seller_mega-domains-c6335228-31a9-4e05-be5e-a8a646cc65fe]: https://undeveloped.com/domain-seller/mega-domains-c6335228-31a9-4e05-be5e-a8a646cc65fe