How To Make Money, Using Web Scraping
"""
文章核心思想就是利用現在的網頁爬取技術在網絡中尋找賺錢的機會，思路是好的，但是一目前的國內情況來看，一方面可能觸碰到法律紅線，另一方面可能是面臨爬去成本與收益之比過低的問題，不過機會還是有，很多，的。
如果希望掌握爬取技術，可以文章後留言，樓主會非常樂意傳給你爬取通用技術代碼，同時也非常歡迎感興趣的朋友一起技術交流
"""
I’m going to show you 3 unique ways to make money, using Web Scraping
Feb 10 · 7 min read
Did you know what your reading right now is data. It may just seem like a few words to you but on the back end everything you read online is data that can be taken, picked apart, and manipulated with. Simplified this is what a Web Scraper is. They go through the code that was created to make a website (HTML code) or database and take the data they want. Virtually any website can be scraped . Some sites do involve measures that stop these scrapers from taking their data, but if your good enough you can essentially scrape 99% of websites online.
If you didn’t know what a Web Scraper is, well now you have an idea and we can get to the point of why your reading this article. Money. Web Scraping can be a unique way to make money that isn’t as difficult as it sounds. In fact all the methods and examples I'm going to show you took less than 50 lines of code to make, and can be learned in only a couple of hours. So with that said let me show you...
3 ways to make Money Using Web Scraping
1. Creating Bots
A bot is just a technical term for a program that does a specific action. Depending what you make this action to be, you can sell it to those who don’t have the technical abilities to make it themselves.
To show how you can create a bot and sell it, I created an Airbnb bot. This bot allows the user to input a location and it will return all the houses that Airbnb offers at that location including the price, rating, number of guests allowed, bedrooms, beds, and baths. All of this being done by web scraping the data of each posting on the Airbnb website.
To demonstrate the bot in action I’m going to input a location. Lets say I want to search for Airbnb’s in Rome, Italy. I simply input Rome into the bot, and it returns 272 unique Airbnb’s within seconds in an organized excel spreadsheet.
It is now much easier to see all the houses/features and their comparisons to other postings. It is also much easier to filter through. I live in a family of 4 and if we were to go to Rome we would look for an Airbnb with at least 2 beds at a decent price. Now with this clean organized spreadsheet, excel makes it extremely easy to filter to match my needs. And out of 272 results 7 returned with my matching needs.
Within these 7 the one I would pick is the Vatican St.Peter Daniel, it has a very good rating and is cheapest out of the 7 with a cost of $61 per night. So after I pick the one I want, I would simply copy the link of the posting into a browser and book it then.
Looking for places to stay can be an extremly daunting task when going on vacation, I’m sure most of us have felt that at one time or another. Because of this there are those that are willing to pay just to make this process easier. With this bot I made the process easier. You just saw me book a room with all my matching needs at a good price within 5 minutes.
Trust me people are willing to pay to make their lives just a bit easier.
2. Reselling
One of the most common uses of web scraping, is getting prices off websites. There are those who create web scraping programs that run everyday and return the price of a specific product, and when the price drops to a certain amount the program will automatically buy the product before its sold out. Then since the demand for the product will be higher than the supply they resell the product at a higher price to make a profit. This is just one example of the many reselling tactics that web scrapers use.
Another one which I will show you an example of can save you a lot of money and make a lot for you too.
Every retail website has limited deals and sales, where they will display the original price and the sale price. But what they don’t do is show how much is actually discounted off the original price. For example if a watch originally costs $350 and the sale price is $300 you would think $50 off would be a lot of money but it’s actually only a 14.2% discount. Now if a T-shirt originally costs $50 and the sale price is $40, you might see $10 being not that much off the original price, but in fact the discount is larger than the watch at 20%. Therefore you can save/make money by buying the products with the highest discounted %.
Using Hudson's’ Bay, a department store that has numerous of sales on all kinds of brands, were going to use web scraping to get the original and sale price of all the products and find the product with the highest discount.
After scraping the website it returned over 900 products and as you can see there is only 1 product out of the 900 with over a 50% discount. That would be the Perry Ellis Solid Non-Iron Dress Shirt.
This sale price is only a limited time offer, so the price for this shirt will eventually go back up to around $90. So if I were to buy it now at $40 than sell it at $30 below its original at $60 when the limited sale ends, I would still make a profit of $20.
This is a method where if you find the right niche to do this is in, there is a potential to make a large amount of money.
3. Selling Data
There are millions of datasets online that are free and accessible to everyone. This data is often easily gathered and thereby offered to anyone who wants to use them. On the other hand some data is not as easy to get, and takes either time or a lot of work to put in a nice clean dataset. This has become the evolution of selling data. There are companies that focus on getting data that may be hard to obtain and structuring that data into a nice clean spreadsheet or dashboard that others can use at a certain cost.
BigDataBall is a sports data website that sells player logs, play-by-play data, and other stats at a price of $30 for a single seasons worth of data. The reason they can ask for this price is not because there the only ones that have this data, but there one of the only websites out there that offer this data in a very structured and clean dashboard that is easy to read.
Now what I’m going to do is get the same data as BigDataBall has for free and I’m going to put it into a structured dataset like the ones I did before.
Like I said before they aren’t the only ones with this type of data. Basketball-Reference.com has all the same data but its not structured meaning its data is all over the place and hard to read, and you simply cannot just download the dataset you want. This is where web scraping comes in. I’m going to web scrape the website of all the players logs for each game and put it into a structured dataset like BigDataBall.
Structured Dataset of all the NBA Player Logs
After web scraping Basketball-Reference.com we got over 16000 player logs for the season so far. You can see why this data in a nice clean format can be monetized, because no one in their right mind would manually get 16000 logs of data and put it into their own dataset. But with the help of web scraping we were able to get this data in a couple of minutes and save ourselves $30.
Needless to say though you can do what BigDataBall does. Find data that is hard to obtain manually, let your computer do the work, and than sell it to those interested in having that data in a structured dataset.
Conclusion
In the world where everyone wants to make money Web Scraping has become a very unique and new way to make money on the side. Where if you apply it to the right situations it can make you a ton of money, and it is easier to do than most people think. With that said if any of you are interested in how I made the programs for the examples above, send me a message and I’ll be happy to take you through the code and how I did it, and hopefully teach you a few things or too. Thanks for reading :)
Towards Data Science
Data Science | Analytics | Sports
