How ProPublica Illinois uses GNU Make to load 1.4GB of data every day
|
|
I avoided using GNU Make in my data journalism work for a long time, partly because the documentation was so obtuse that I couldn’t see how Make, one of many extract-transform-load (ETL) processes, could help my day-to-day data reporting. But this year, to build The Money Game, I needed to load 1.4GB of Illinois political contribution and spending data every day, and the ETL process was taking hours, so I gave Make another chance.
Now the same process takes less than 30 minutes.
read more
|
|
Full Story |
This topic does not have any threads posted yet!
You cannot post until you login.
