Web Crawler with crawl4ai

We’ve built a cool asynchronous web crawler that grabs restaurant deals from food delivery sites using the crawl4ai library for smooth data scraping. This tool works without interruptions, letting everything run at the same time for better speed and efficiency. Once we scrape the data, we neatly organize it and save it in a CSV file so it’s easy to analyze and use later.

Features

• Asynchronous Execution: Handles multiple URLs concurrently for efficient crawling.
• Data Extraction Schema: Uses JsonCssExtractionStrategy to define the structure for data extraction.
• Configurable Browser Settings: Employs Chromium in headless mode for fast and lightweight crawling.
• Chunk-based Processing: Splits URLs into manageable chunks to avoid overloading the server.
• Delay Handling: Adds delays between processing chunks to mimic human browsing behavior and prevent IP bans.
• Customizable Output: Extracted data is saved into a user-defined CSV file.

How It Works

• Input File: Provide a CSV file (input2.csv) containing the URLs to scrape.
• Extraction Schema: The script uses the following JSON-CSS schema

• Crawling: The script processes URLs in chunks of 15. For each URL, it extracts offer details and saves them into the output CSV (output2.csv).
• Output File: Results are written to output2.csv with two columns:URL: The URL that was processed.Offer Title: The extracted offer details.

Prerequisites

Ensure you have the following dependencies installed:

pip install crawl4ai asyncio        

Now the complete logic is

Input Example (input2.csv):

https://example1.com
https://example2.com
https://example3.com        

Output (output2.csv):

URL,Offer Title
https://example1.com,"20% Off on Orders"
https://example2.com,"Buy 1 Get 1 Free"        

Working Procedure

• Setup Input File: Create a CSV file (input2.csv) with a list of URLs to scrape.
• Run the Script: Execute the script using Python:python script_name.py
• Extract Data: The script processes the URLs, extracts offer details and saves them to output2.csv.
• Review Output: Check the output2.csv file for the extracted data.

Creating a price comparison model for platform-wise competitive food pricing in the industry based on this code opens up several exciting possibilities. Here's a detailed breakdown of the scope, possibilities, and how this code can be adapted and extended for such a project:

Scopes

• Real-Time Price Tracking: A system can be made that keeps an eye on food prices from places like Foodpanda, Uber Eats, and DoorDash, so you can spot trends and see what discounts competitors are offering.
• Competitive Comparison: Check out how prices stack up for similar restaurants or food items across different platforms to find the best deals for what you're craving.
• Dynamic Pricing Insights: Notice patterns in promotional pricing and time-limited offers to help businesses tweak their pricing strategies based on what competitors are doing.
• Market Segmentation: Break down prices by location, type of cuisine, or delivery fees to get a better understanding of how different market segments react to prices.
• Historical Price Trends: Build up a history of food prices to spot recurring trends or seasonal discounts that pop up over time.

Possibilities

• Data Integration for Price Comparison: Crawler can systematically gather price data and food offers, including base prices, discounted prices, and delivery fees. This process incorporates essential details such as restaurant names, food items, and platform information to facilitate comprehensive side-by-side comparisons.
• Machine Learning for Price Optimization: the development of machine learning models aimed at predicting effective pricing strategies based on historical data. This initiative will involve the application of clustering algorithms to categorize similar price points and discern discount trends.
• Dashboard Development: Create interactive dashboards utilizing advanced tools such as Power BI, Tableau, or Python web frameworks (e.g., Dash) to present price comparisons across various platforms. These dashboards will feature visualizations illustrating price disparities by platform, temporal changes, and regional trends.
• Customer-Focused Recommendations: Establishment of a recommendation system designed to assist customers in identifying the most cost-effective platform for their desired food items.