Python爬虫：自动处理反爬机制与数据存储指南

一、反爬机制应对策略

1. 请求头伪装

   • 设置真实浏览器头信息：

     headers = {
         'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...',
         'Accept-Language': 'zh-CN,zh;q=0.9',
         'Referer': 'https://www.google.com/'
     }
     

2. IP轮换与代理池

   proxies = {
       'http': 'http://user:pass@10.10.1.10:3128',
       'https': 'https://user:pass@10.10.1.10:1080'
   }
   response = requests.get(url, headers=headers, proxies=proxies)
   

3. 请求频率控制

   import random, time
   time.sleep(random.uniform(1.5, 3.0))  # 随机延时
   

4. Cookie动态管理

   session = requests.Session()
   session.get(login_url)  # 获取初始Cookie
   session.cookies.update(new_cookies)  # 更新Cookie
   

二、高级反爬解决方案

1. JavaScript渲染处理

   from selenium import webdriver
   from selenium.webdriver.chrome.options import Options
   
   options = Options()
   options.add_argument('--headless')  # 无头模式
   driver = webdriver.Chrome(options=options)
   driver.get(url)
   html = driver.page_source  # 获取渲染后HTML
   

2. 验证码识别

   # 使用第三方库（需申请API密钥）
   from pytesseract import image_to_string
   captcha_text = image_to_string(captcha_image)
   

3. TLS指纹绕过

   # 使用curl_cffi库
   from curl_cffi import requests
   response = requests.get(url, impersonate="chrome110")
   

三、数据存储方案

1. CSV文件存储

   import csv
   with open('data.csv', 'a', newline='', encoding='utf-8') as f:
       writer = csv.writer(f)
       writer.writerow([title, price, rating])
   

2. SQLite数据库

   import sqlite3
   conn = sqlite3.connect('data.db')
   c = conn.cursor()
   c.execute('''CREATE TABLE IF NOT EXISTS products
                (id INTEGER PRIMARY KEY, name TEXT, price REAL)''')
   c.execute("INSERT INTO products VALUES (?, ?, ?)", (1, '商品A', 29.9))
   conn.commit()
   

3. JSON存储

   import json
   data = {"items": [...]}
   with open('data.json', 'w', encoding='utf-8') as f:
       json.dump(data, f, ensure_ascii=False)
   

四、完整爬虫示例

import requests
from bs4 import BeautifulSoup
import time, random, csv

# 反爬配置
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...'}
proxy_list = ['http://ip1:port', 'http://ip2:port']  # 实际代理池

def scrape_page(url):
    try:
        proxy = {'https': random.choice(proxy_list)}
        response = requests.get(url, headers=headers, proxies=proxy, timeout=10)
        response.raise_for_status()
        return BeautifulSoup(response.text, 'html.parser')
    except Exception as e:
        print(f"请求失败: {e}")
        return None

def save_data(data):
    with open('books.csv', 'a', newline='', encoding='utf-8') as f:
        writer = csv.writer(f)
        writer.writerow(data)

# 主程序
base_url = "https://example.com/books?page="
for page in range(1, 6):
    soup = scrape_page(base_url + str(page))
    if soup:
        books = soup.select('.book-item')
        for book in books:
            title = book.select_one('.title').text.strip()
            price = book.select_one('.price').text
            save_data([title, price])
    time.sleep(random.uniform(2, 4))  # 随机延时

五、最佳实践建议

1. 遵守robots.txt规则

   import urllib.robotparser
   rp = urllib.robotparser.RobotFileParser()
   rp.set_url("https://example.com/robots.txt")
   rp.read()
   if rp.can_fetch("*", target_url):
       # 执行爬取
   

2. 异常处理机制

   try:
       # 爬取代码
   except requests.exceptions.RequestException as e:
       print(f"网络错误: {e}")
   except (AttributeError, TypeError) as e:
       print(f"解析错误: {e}")
   

3. 分布式爬虫架构

   ┌───────────┐       ┌───────────┐
   │ 主节点     │──任务─▶│ 爬虫节点1  │
   │(任务调度)  │       └───────────┘
   └───────────┘           ▲
         │                 │
         ▼                 ▼
   ┌───────────┐       ┌───────────┐
   │ 存储中心   │◀──数据─┤ 爬虫节点N  │
   └───────────┘       └───────────┘
   

重要提示：实际开发中需根据目标网站特点调整策略，建议使用Scrapy框架构建工程化爬虫系统，并遵守相关法律法规和网站使用条款。