Web Scrapers in Ruby

Tools used

• Ruby
• Mechanize (Crawler and Parser) github
• MongoDB (Database) github
• Mongo Ruby Driver github
• Parallel (Multi threading, Multi Processing) github
• Typhoeus (Concurrent HTTP requests) github
• Delayed Job (ASync Background Job) github

Samples

• Plain simple web scraper using mechanize
• Storing result in mongo
• Progress tracking with mongo
• Multi threading with parallel
• Concurrent HTTP requests with typhoeus
• Background Job with delayed_job