快捷導(dǎo)航

python爬蟲(chóng)租房信息在地圖上顯示的方法

更新時(shí)間：2019年05月13日 15:27:45 作者：DIVI_ZHANG

這篇文章主要介紹了python爬蟲(chóng)租房信息在地圖上顯示的方法，文中通過(guò)示例代碼介紹的非常詳細(xì)，對(duì)大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值，需要的朋友們下面隨著小編來(lái)一起學(xué)習(xí)學(xué)習(xí)吧

本人初學(xué)python是菜鳥(niǎo)級(jí)，寫(xiě)的不好勿噴。

python爬蟲(chóng)用了比較簡(jiǎn)單的urllib.parse和requests，把爬來(lái)的數(shù)據(jù)顯示在地圖上。接下里我們?cè)挷欢嗾f(shuō)直接上代碼:

1.安裝python環(huán)境和編輯器（自行度娘）

2.本人以58品牌公寓為例，爬取在杭州地區(qū)價(jià)格在2000-4000的公寓。

#-*- coding:utf-8 -*-
from bs4 import BeautifulSoup
from urllib.parse import urljoin
import requests
import csv
import time

以上是需要引入的模塊

url = "http://hz.58.com/pinpaigongyu/pn/{page}/?minprice=2000_4000"
#已完成的頁(yè)數(shù)序號(hào)，初時(shí)為0
page = 0

以上的全局變量

csv_file = open(r"c:\users\****\Desktop\houoseNew.csv","a+",newline='')
csv_writer = csv.writer(csv_file, delimiter=',')

自定義某個(gè)位置來(lái)保存爬取得數(shù)據(jù)，本人把爬取得數(shù)據(jù)保存為csv格式便于編輯（其中”a+”表示可以多次累加編輯在后面插入數(shù)據(jù)，建議不要使用“wb”哦！newline=”表示沒(méi)有隔行）

while True:
  #為了防止網(wǎng)站屏蔽ip，設(shè)置了時(shí)間定時(shí)器每隔5秒爬一下。打完一局農(nóng)藥差不多都爬取過(guò)來(lái)了。
  time.sleep(5)
  page +=1
  #替換URL中page變量
  print (url.format(page=page)+"ok")
  response = requests.get(url.format(page=page))
  html=BeautifulSoup(response.text)
  #尋找html中DOM節(jié)點(diǎn)li
  house_list = html.select(".list > li")

  # 循環(huán)在讀不到新的房源時(shí)結(jié)束
  if not house_list:
    break

  for house in house_list:
    #根據(jù)hml的DOM節(jié)點(diǎn)獲取自己需要的數(shù)據(jù)
    house_title = house.select("h2")[0].string
    house_url = urljoin(url, house.select("a")[0]["href"])
    house_pic = urljoin(url, house.select("img")[0]["lazy_src"])
    house_info_list = house_title.split()

    # 如果第一列是公寓名 則取第二列作為地址
    if "公寓" in house_info_list[0] or "青年社區(qū)" in house_info_list[0]:
      house_location = house_info_list[0]
    else:
      house_location = house_info_list[1]

    house_money = house.select(".money")[0].select("b")[0].string
    csv_writer.writerow([house_title, house_location, house_money,house_pic ,house_url])
 #最后不要忘記關(guān)閉節(jié)流
 csv_file.close()

如果網(wǎng)站屏蔽了你的ip，你可以做一個(gè)ip地址數(shù)組放在http的頭部具體度娘一下吧。

接下來(lái)我們寫(xiě)html

只是簡(jiǎn)單的寫(xiě)了一下寫(xiě)的不好見(jiàn)諒。用的是高德地圖，具體的js api可以到高德開(kāi)發(fā)者上去看。

<body>
<div id="container"></div>
<div class="control-panel">
  <div class="control-entry">
    <label>選擇工作地點(diǎn)：</label>
    <div class="control-input">
      <input id="work-location" type="text">
    </div>
  </div>
  <div class="control-entry">
    <label>選擇通勤方式：</label>
    <div class="control-input">
      <input type="radio" name="vehicle" value="SUBWAY,BUS" onClick="takeBus(this)" checked/> 公交+地鐵
      <input type="radio" name="vehicle" value="SUBWAY" onClick="takeSubway(this)"/> 地鐵
      <input type="radio" name="vehicle" value="WALK" onClick="takeWalk(this)"/> 走路
      <input type="radio" name="vehicle" value="BIKE" onClick="takeBike(this)"/> 騎車(chē)
    </div>
  </div>
  <div class="control-entry">
    <label>導(dǎo)入房源文件：</label>
    <div class="control-input">
      <input type="file" name="file" id="fileCsv"/>
      <button style="margin-top: 10px;width: 50%;" onclick="changeCsv()">開(kāi)始</button>
    </div>
  </div>
</div>
<div id="transfer-panel"></div>
<script>
  var map = new AMap.Map("container", {
    resizeEnable: true,
    zoomEnable: true,
    center: [120.1256856402492, 30.27289264553506],
    zoom: 12
  });

  //添加標(biāo)尺
  var scale = new AMap.Scale();
  map.addControl(scale);

  //公交到達(dá)圈對(duì)象
  var arrivalRange = new AMap.ArrivalRange();
  //經(jīng)度，緯度，時(shí)間（用不到），通勤方式（默認(rèn)是地鐵＋公交+走路+騎車(chē)）
  var x, y, t, vehicle = "SUBWAY,BUS";
  //工作地點(diǎn)，工作標(biāo)記
  var workAddress, workMarker;
  //房源標(biāo)記隊(duì)列
  var rentMarkerArray = [];
  //多邊形隊(duì)列，存儲(chǔ)公交到達(dá)的計(jì)算結(jié)果
  var polygonArray = [];
  //路徑規(guī)劃
  var amapTransfer;

  //信息窗體對(duì)象
  var infoWindow = new AMap.InfoWindow({
    offset: new AMap.Pixel(0, -30)
  });

  //地址補(bǔ)完的使用
  var auto = new AMap.Autocomplete({
    //通過(guò)id指定輸入元素
    input: "work-location"
  });
  //添加事件監(jiān)聽(tīng)，在選擇補(bǔ)完的地址后調(diào)用workLocationSelected
  AMap.event.addListener(auto, "select", workLocationSelected);


  function takeBus(radio) {
    vehicle = radio.value;
    loadWorkLocation()
  }

  function takeSubway(radio) {
    vehicle = radio.value;
    loadWorkLocation()
  }
  function takeWalk(radio){
    vehicle = radio.value;
    loadWorkLocation()
  }
  function takeBike(radio) {
    vehicle = radio.value;
    loadWorkLocation()
  }
  //獲取加載的文件
  function changeCsv() {
    $("#fileCsv").csv2arr(function (res) {
      $.each(res, function (k, p) {
        if (res[k][1]) {
          //addMarkerByAddress(地址，價(jià)格，展示的圖片)
          addMarkerByAddress(res[k][1], res[k][2],res[k][3])
        }
      })
    });
  }

  function workLocationSelected(e) {
    workAddress = e.poi.name;
    loadWorkLocation();
  }

  function loadWorkMarker(x, y, locationName) {
    workMarker = new AMap.Marker({
      map: map,
      title: locationName,
      icon: 'http://webapi.amap.com/theme/v1.3/markers/n/mark_r.png',
      position: [x, y]

    });
  }


  function loadWorkRange(x, y, t, color, v) {
    arrivalRange.search([x, y], t, function (status, result) {
      if (result.bounds) {
        for (var i = 0; i < result.bounds.length; i++) {
          //新建多邊形對(duì)象
          var polygon = new AMap.Polygon({
            map: map,
            fillColor: color,
            fillOpacity: "0.4",
            strokeColor: color,
            strokeOpacity: "0.8",
            strokeWeight: 1
          });
          //得到到達(dá)圈的多邊形路徑
          polygon.setPath(result.bounds[i]);
          polygonArray.push(polygon);
        }
      }
    }, {
      policy: v
    });
  }

  function addMarkerByAddress(address, money,imgUrl) {
    var geocoder = new AMap.Geocoder({
      city: "杭州",
      radius: 1000
    });
    geocoder.getLocation(address, function (status, result) {
      var iconValue = "";
      var _money=money;
      if (money.indexOf("-") > -1) {
        _money = money.split("-")[1];
      }
      //如果價(jià)格高于3000元/月在地圖上顯示紅色，低于的話(huà)顯示藍(lán)色
      if (parseFloat(_money) > 3000) {
        iconValue="http://webapi.amap.com/theme/v1.3/markers/n/mark_r.png";
      }else{
        iconValue = "http://webapi.amap.com/theme/v1.3/markers/n/mark_b.png";
      }
      if (status === "complete" && result.info === 'OK') {
        var geocode = result.geocodes[0];
        rentMarker = new AMap.Marker({
          map: map,
          title: address,
          icon:iconValue,
          animation:"AMAP_ANIMATION_DROP",
          position: [geocode.location.getLng(), geocode.location.getLat()]
        })
        ;
        rentMarkerArray.push(rentMarker);
        //鼠標(biāo)點(diǎn)擊標(biāo)記顯示相應(yīng)的內(nèi)容
        rentMarker.content = "<img src='"+imgUrl+"'/><div>房源：<a target = '_blank'  + address + "'>" + address + "</a><p>價(jià)格:"+money+"</p><div>"
        rentMarker.on('click', function (e) {
          infoWindow.setContent(e.target.content);
          infoWindow.open(map, e.target.getPosition());
          if (amapTransfer) amapTransfer.clear();
          amapTransfer = new AMap.Transfer({
            map: map,
            policy: AMap.TransferPolicy.LEAST_TIME,
            city: "杭州市",
            panel: 'transfer-panel'
          });
          amapTransfer.search([{
            keyword: workAddress
          }, {
            keyword: address
          }], function (status, result) {
          })
        });
      }
    })
  }

  function delWorkLocation() {
    if (polygonArray) map.remove(polygonArray);
    if (workMarker) map.remove(workMarker);
    polygonArray = [];
  }

  function delRentLocation() {
    if (rentMarkerArray) map.remove(rentMarkerArray);
    rentMarkerArray = [];
  }

  function loadWorkLocation() {
    //首先清空地圖上已有的到達(dá)圈
    delWorkLocation();
    var geocoder = new AMap.Geocoder({
      city: "杭州",
      radius: 1000
    });

    geocoder.getLocation(workAddress, function (status, result) {
      if (status === "complete" && result.info === 'OK') {
        var geocode = result.geocodes[0];
        x = geocode.location.getLng();
        y = geocode.location.getLat();
        //加載工作地點(diǎn)標(biāo)記
        loadWorkMarker(x, y);
        //加載60分鐘內(nèi)工作地點(diǎn)到達(dá)圈
        loadWorkRange(x, y, 60, "#3f67a5", vehicle);
        //地圖移動(dòng)到工作地點(diǎn)的位置
        map.setZoomAndCenter(12, [x, y]);
      }
    })
  }
</script>
</body>

想要獲取完整的代碼github：https://github.com/DIVIBEAR/pythonDemo.git
新手上路，老司機(jī)們勿噴！