dev-resources.site
for different kinds of informations.
The Tech News Scraper
This is a submission for the Bright Data Web Scraping Challenge: Scrape Data from Complex, Interactive Websites
What I Built
This project scrapes data from websites that offer the latest technological news and updates. It uses JavaScript and Node.js, with Puppeteer and the Bright Data Scraping Browser to handle dynamic content. It scrapes data from two major websites:
Demo
You can view the source code and instructions for running the project on GitHub.
How I Used Bright Data
I leveraged Bright Data’s Scraping Browser to handle JavaScript-heavy and interactive websites that require dynamic content loading. The project scrapes real-time data, including titles, descriptions, URLs, images, and published dates. Bright Data's browser provided a smooth solution to maintain the scraping process without additional overhead.
Challenge Prompt: Bright Data Web Scraping Challenge
Installation
- Clone the repository
git clone https://github.com/chethanyadav456/Scraping_Master.git
- Install dependencies
npm install
- Run the project
node master.js
- Create a .env file and add:
MONGO_URI=
BROWSER_WS=
License
This project is licensed under the MIT License - see the LICENSE file for details
Featured ones: