Documentation

Scraping API Integration

Integrating SmartCrawl’s Scraping APIs into your applications is simple and effective, allowing you to collect the data you need from some of the world’s largest platforms. Our APIs are designed for flexibility, enabling seamless data extraction for various use cases, from market research to competitive analysis. Whether you are scraping Amazon, Google, eBay, or more, SmartCrawl’s APIs provide reliable, scalable solutions.

Getting Started with SmartCrawl Scraping API

Our Scraping API is built for ease of use. Follow these steps to get started:

1. API Key Authentication

To authenticate your API requests, you’ll need your API Key. You can find this in your SmartCrawl dashboard. Use this key in the headers of your requests to authenticate:

Authorization: Bearer <Your_API_Key>

Supported Data Sources

SmartCrawl’s APIs support data extraction from a wide range of sources, including:

Amazon: Collect product data, prices, and reviews from Amazon’s vast marketplace.

Google: Scrape search engine results pages (SERP), Google Maps data, and more.

eBay: Access auction data, product listings, and seller information.

YouTube: Extract video metadata, comments, and engagement metrics.

Walmart, Target, Etsy, and more: Scrape product data from various global retail platforms.

API Endpoints

SmartCrawl offers a variety of API endpoints based on your data needs:

Product Scraping API

For scraping eCommerce platforms like Amazon, Walmart, and Target:

GET /product/<platform>?query=<search_term>&location=<geo_target>

SERP Scraping API

For scraping search engine results:

GET /serp?keywords=<keywords>&location=<geo_target>&device=<desktop

Custom API Requests

Need custom scraping solutions? Use our flexible API to define your specific scraping needs, including custom headers and parameters.

Error Handling & Rate Limits

We implement clear error handling to ensure smooth operation:

• 400 Bad Request: Invalid request or missing parameters.

• 401 Unauthorized: Invalid or missing API key.

• 429 Too Many Requests: You have hit your rate limit. Our standard rate limit is 100 requests per minute.

For higher rate limits, contact our support team for tailored solutions.

Sample Request: Scraping Amazon

Here’s a sample API request to scrape Amazon for a product search:

GET /product/amazon?query=smartphone&location=US Authorization: Bearer <Your_API_Key

This request retrieves a list of smartphones available in the US market, including product names, prices, and availability.

API Response Format

All responses from SmartCrawl’s APIs are returned in JSON format for easy parsing and integration into your system. Here’s an example of a response:

{ 
"product": {
"name": "Smartphone XYZ",
"price": "$499",
"availability": "In Stock",
"rating": "4.5",
"reviews": "254",
"url": "https://amazon.com/product/smartphoneXYZ"
}
}

Advanced API Features

SmartCrawl offers additional advanced features to enhance your scraping efforts:

• IP Rotation: Automatically switch between multiple IPs to avoid detection and bans.

• Geo-Targeting: Specify city, state, or country to scrape localized data.

• Session Persistence: Maintain long-term sessions to scrape data consistently without interruption.

Integration with Popular Tools

SmartCrawl APIs can easily integrate with popular data scraping tools and frameworks, such as:

• Scrapy: Learn how to integrate with Scrapy

• BeautifulSoup: Learn how to integrate with BeautifulSoup

• Puppeteer: Learn how to integrate with Puppeteer

For additional support or customized integration, please contact our API support team.

Rate Limits & Upgrades

By default, SmartCrawl offers 100 requests per minute. If your use case requires higher throughput, you can easily upgrade your plan to increase the rate limit and data volume capacity.

Get Started with SmartCrawl’s Scraping API Today

Leverage SmartCrawl’s Scraping API to access data across platforms, enhance your competitive intelligence, and streamline your data collection efforts. Ready to start? Sign up now or get in touch with our team for a demo.

Getting Started ›