Web Crawl with out not getting your ip Blocked :P

I have had this data science project on my way where i had to crawl data from websites like amazon. Its fairly predictable that your IP might get blocked when the website detects your web-crawl pattern.

Here is the what i had to did. I took a bunch of proxies available for free ( few premium options available too) and make a random choice each time when you hit the url.

Pretty simple!!

Learner, thinker and Doer.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store