Search the Community

Since it's often not very clear how an economics researcher would actually go about using Python in their research, I put together a little tutorial that demonstrates a couple combined use cases. The title of the tutorial is "Estimate an Econometric Model from Scraped Data in Real-Time using Python" and you can find it here: live.economics.io The application that's looked at is one of determining the factors that influence the price of a desktop CPU using the prices and features of the processors that are currently listed at newegg.com as the data set for the model. One point of this tutorial is to illustrate how straightforward it can be to collect, clean and analyze data on the Internet. That is, turning unstructured data into structured data, then performing some analysis on that data, all using Python. Another abstraction I wanted to demonstrate was that of thinking about the data you want to analyze in your research as being some txt/xls/csv/etc file on your computer. The ability to literally build or engineer your own data set has many great properties, one of which is that your data set will likely be unique since it is a function of the collection algorithms you write in your scripts (in terms of discovering the agents/firms/products/features that are relevant to analyzing a particular market/etc), and when operating on a larger scale, the configuration of the server [and it's geographic location!] that hosts the websites you're trying to scrape. As always, comments and feedback are encouraged.