Mike Levin SEO

Future-proof your technology-skills with Linux, Python, vim & git... and me!

Moved all the Twitter functions into ezscrape to put it through its paces

by Mike Levin SEO & Datamaster, 07/19/2012

Okay, now’s your chance to do something amazing. Don’t blow it. Functions like twitter really test all the various things I built for ezscrape. Move at least one twitter function over.

Pshwew! Just did a bunch of error checking code to make sure that making new ezscrapes isn’t infuriating from errors, and that things like urlfield (to switch which field to use for a URL) works in conjunction with the rewrite field. This is all coming from turning my twitter functions into ezscrape functions, because it takes twusername instead of url as input, so I both have to use urlfield and rewrite to turn it into a hit against the Twitter json API.

Pshwew! I just moved all my Twitter functions over to ezscrape. Hit a bunch of things I had to debug, and it’s now 5:20 PM, and it took me longer than I wanted, but there was stuff like dealing with escaped slashes in JSON. I’m grabbing values out of JSON with RegEx, which is not ideal, but I can really start rolling ezscrape out that way minimizing the number of unknowns I have to deal with.

But it really is amazing how much of Tiger’s crufty functions can be moved over to a list of dictionary parameters, so now not every case is special. Slamming out new scrape or API functions is going to be that much easier now that I don’t have to think in Python functions every time. Eventually, I’ll move fancier scraping that reaches down through iframes to get its value. It should be no harder than the twitter functions, which switch twusername into an API call through a URL rewrite trick sitting in the scraping rules.

Okay, then what now in the precious bit of time left, what can I REALLY do? Well, I can do thought-work. Forget about any real coding. It’s already 5:45, since I started typing this. It was valuable to get all the twitter functions into ezscrape, and I’m glad I did it, because it hit most of the complex cases I’m going to encounter.

Ah… test how the ezscrape tab auto-populates, and how it looks now that the twitter functions are there… nice! I should finish the pinterest functions, and all the social shopping functions. But moving advanced scrapes into ezscrape now is a done deal. I’m very confident about that, and my time really needs to go into getting the data from the tab to flow back into the system, so people can define their own scrapes.