I really couldn’t find a COGI list anywhere either. Was hoping to see if could find one for the last 25 years or so. This made me interested in seeing if I could set up a webscraper and pull the data for the top 50 name brand products most purchased from popular stores (milk, eggs, bread, toilet paper, deodorant, sugar, flour etc), and save them daily to a database. I assume the user agents and proxies to make the requests look like they are coming from different people so they don’t get blocked would be the hardest part, but maybe I’ll see if I can set one up throughout the week.
I really couldn’t find a COGI list anywhere either. Was hoping to see if could find one for the last 25 years or so. This made me interested in seeing if I could set up a webscraper and pull the data for the top 50 name brand products most purchased from popular stores (milk, eggs, bread, toilet paper, deodorant, sugar, flour etc), and save them daily to a database. I assume the user agents and proxies to make the requests look like they are coming from different people so they don’t get blocked would be the hardest part, but maybe I’ll see if I can set one up throughout the week.
I’m ready for spectacular failure of course.
Shit, how would I do clothes…