作為python菜鳥，突如其來地想爬取智聯(lián)招聘的招聘信息，本來是想爬取職位介紹提取關(guān)鍵字做數(shù)據(jù)分析的，然而智聯(lián)的html結(jié)構(gòu)太混亂，只得放棄這個(gè)想法，先爬取了基本的職位信息存儲(chǔ)到本機(jī)的mysql數(shù)據(jù)庫中。純新手一枚，寫得相當(dāng)粗糙，湊合著看吧，具體代碼如下：

mport requests

import urllib.parse

import re

from lxml import etree

import threading

unity_url = r'http://sou.zhaopin/jobs/searchresult.ashx?jl={location}&kw={job}&sm=0&p={page}&source=0'

headers = {"User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"}

def get_info_url():

info_urls = []

location = input("請(qǐng)輸入想要搜索的城市：")

loc = urllib.parse.quote(location.encode('utf-8'))

job = input("請(qǐng)輸入想要搜索的職位：")

job = urllib.parse.quote(job.encode('utf-8'))

num = input("請(qǐng)輸入想要獲取的頁數(shù)：")

for i in range(1,int(num)+1):

url = unity_url.format(location=loc,job=job,page=str(i))

page = requests.get(url, headers=headers)

page.encoding = 'utf-8'

html = page.text

r = repile('need_urls = r.findall(html)

for n_url in need_urls:

info_urls.append(n_url[1])

return info_urls

def get_infos():

info_urls = get_info_url()

jobs = []

companies = []

work_years = []

degrees = []

salarys = []

places = []

for info_url in info_urls:

print (info_url)

info_page = requests.get(info_url, headers=headers)

info_page.encoding = 'utf-8'

info_html = info_page.text

e_html = etree.HTML(info_html)

try:

job = e_html.xpath('/html/body/p[5]/p[1]/p[1]/h1')[0].text

jobs.append(job)

company = e_html.xpath('/html/body/p[5]/p[1]/p[1]/h2/a')[0].text

companies.append(company)

work_year = e_html.xpath('/html/body/p[6]/p[1]/ul/li[5]/strong')[0].text

work_years.append(work_year)

degree = e_html.xpath('/html/body/p[6]/p[1]/ul/li[6]/strong')[0].text

degrees.append(degree)

salary = e_html.xpath('/html/body/p[6]/p[1]/ul/li[1]/strong')[0].text

salarys.append(salary.split('元')[0])

place = e_html.xpath('/html/body/p[6]/p[1]/ul/li[2]/strong/a')[0].text

places.append(place)

except:

pass

return jobs, companies, work_years, degrees, salarys, places

if __name__ == '__main__':

t = threading.Thread(target=get_infos)

t.start()

此代碼只是獲取了一些職位信息的URL后寫入列表中，后續(xù)還有寫入數(shù)據(jù)庫，提取數(shù)據(jù)做數(shù)據(jù)可視化的例子

后面數(shù)據(jù)可視化的例子