just-scrape CLI
AI-powered CLI for web scraping, data extraction, and page monitoring
What You Can Do
Search the web, scrape URLs into markdown/HTML/screenshots/links/images, extract structured JSON with AI prompts, crawl multi-page sites, and monitor pages for changes on a schedule. Supports browser automation features like JavaScript rendering, stealth mode, cookies, and custom headers. Perfect for data engineers, researchers, and developers who need API-free web data collection without writing scrapers.
Features
Find pages and sources with optional AI-powered data extraction and filtering by time range or country
Extract a single URL as markdown, HTML, screenshot, links, images, summary, or branding in one command
Use AI prompts and JSON schemas to pull structured data from any webpage with guaranteed output shape
Bulk extract multiple pages from a site section with depth limits, include/exclude patterns, and format control
Schedule recurring checks with cron intervals or simple shorthands (30m, 1h, 1d) and optional webhook notifications
JavaScript rendering, stealth mode, custom cookies, authentication headers, and configurable scroll depths
Track all scrape, extract, search, and crawl operations with status, timestamps, and full request/response payloads
Monitor API credits in real-time and validate ScrapeGraph API configuration before large operations
Example Output
Search for Python tutorials and extract titles
just-scrape search "python best practices" --num-results 5 -p "Extract page title and description" --json
Returns JSON with matched pages, their titles, descriptions, and URLs from the search results.
Scrape a homepage and extract multiple formats
just-scrape scrape "https://example.com" -f markdown,links,screenshot --json
Downloads the page as markdown, extracts all links, captures a screenshot, and saves all formats to JSON in a single request.
Monitor a pricing page hourly with webhook alerts
just-scrape monitor create --url "https://example.com/pricing" --interval 1h --webhook-url "https://your-webhook.com/alerts" --name "Pricing tracker"
Schedules hourly checks of the page and POSTs change notifications to your webhook endpoint automatically.
What's Included
- 8 CLI commands: search, scrape, extract, crawl, monitor, history, credits, validate
- Claude agent skill: SKILL.md for seamless integration with Claude Code workflows
- Configuration management: Supports environment variables, .env files, config.json, and interactive prompts
- NPM package: Globally installable via npm, pnpm, yarn, or bun with zero additional dependencies
Who It's For
- Data engineers
- Web developers and full-stack engineers
- Researchers and data scientists
- DevOps and SRE teams
- Business intelligence and competitive intelligence analysts
Best For
- Web scraping and data extraction
- Competitive intelligence and market research
- Page change monitoring and alerting
- API-free data collection from websites
- Structured data extraction with AI validation