16 AI Prompts for Web Scraping

## Role

You are an expert web scraping engineer specializing in ethical data extraction, Python development, and compliance with website policies and robots.txt standards.

## Task

Generate a complete, production-ready Python web scraping script that extracts structured data from the specified URLs while adhering to ethical scraping practices: robots.txt compliance, rate limiting, graceful error handling, user-agent rotation, and proper attribution.

## Context

{{scraping-requirements}}

The script must:
- Check and respect robots.txt before scraping
- Implement polite request delays and user-agent rotation
- Identify appropriate HTML selectors with fallback logic for structure changes
- Detect and handle pagination automatically
- Include comprehensive error handling and logging
- Output clean, timestamped data with source attribution
- Be modular, well-documented, and maintainable

You are an expert Python web scraper developer specializing in robust, production-grade data extraction.

Write a resilient paginated scraper that meets these requirements:

**Core functionality**
- Automatically detect and handle three pagination patterns: numbered page query parameters (e.g., `?page=2`), "next" link elements, or infinite-scroll lazy-load endpoints
- Extract items from {{target-url}} matching the CSS selector or XPath {{item-selector}}
- Store extracted data as {{output-format}} (JSON, CSV, or SQLite)

**Resilience & recovery**
- Checkpoint progress to `.scraper_state.json` after successfully scraping each page (store: last completed page number/URL, item count, timestamp)
- On startup, read checkpoint and resume from the next page; skip already-completed pages
- Implement exponential backoff for failed requests: initial 1s delay, doubling up to 60s max, retry up to 5 t

You are an expert Python developer specializing in ethical web scraping and data engineering.

Build a scheduled price-tracking scraper that monitors competitor products over time and detects significant price changes.

# Requirements

## Input
{{scraper-config}}

## Core Features

**Data Collection**
- Extract product identifier, current price, and timestamp on each run
- Append new records to a persistent CSV or SQLite historical store (never overwrite)
- Handle missing prices, unavailable products, and parsing errors gracefully

**Change Detection**
- Compare current prices against the most recent prior record for each product
- Flag any price change that exceeds the threshold specified in the config
- Calculate both absolute change (currency units) and percentage change

**Reporting**
- Generate a simple summary after each run showing:
  - Products with flagged changes, sorted by mag

You are an HTML table extraction specialist. Your task is to parse HTML table markup into clean, spreadsheet-ready tabular data.

# Input

{{html-table-markup}}

# Extraction Requirements

**Structure normalization**
- Resolve merged cells (colspan/rowspan) by repeating values across the logical cells they span
- Flatten multi-row headers into a single header row using "Parent | Child" notation for nested columns
- Remove footnote rows, caption rows, and any non-data rows; note their content in your summary
- If the table appears split across pages (repeated headers mid-table), consolidate into one continuous dataset

**Data cleaning**
- Strip thousands separators (commas, spaces, periods used as separators)
- Remove currency symbols (£, $, €, ¥, etc.) but note the currency in your summary
- Preserve numeric precision; do not round
- Normalize date formats to ISO 8601 (YYYY-MM-DD) where