Planning Fully Unbounded Web Scraper Kit For AI
I'm creating a project called Fully Unbounded Web Scraper Kit (FUWSK) to help me stay competitive and produce results with a compounding effect. FUWSK will be like a command-line interface command that uses genuine browser and web-scraping output to feed OpenAI and automate subsequent browsing commands. Join me on this journey to discover how FUWSK can help me stay ahead in this rapidly changing world.
Unlock the Power of AI with Fully Unbounded Web Scraper Kit (FUWSK)!
By Michael Levin
Monday, March 27, 2023
Things are hardly worth doing anymore unless they have some sort of snowballing effect, like compounding returns that produces some sort of exponential results. The reason is that it’s getting really competitive out there. AIs will surely do everything that does not honestly and truly require the human touch.
The problem with this approach is that almost everything worth doing is difficult, or else everyone would be doing it.
This leads to a tricky problem. What do I do next in my job-capacity at MOZ, and in this life and in this world and as a dad and as a lover of cats and unsure about mission and purpose, but zeroing in, for sure.
Define a project that you can
- Private Journal (truly open-ended brainstorming / wacky w/no limits)
- Public Journal (slightly refined thoughts, still rough and small audience)
- Private YouTube (experiments only for my fellow co-workers at MOZ)
- Public YouTube (mostly my personal account, and mostly raw unedited practice)
- Public MOZ (refined for as broad a general public audience as possible)
What is the project?
As I watch what’s going on with AI, ChatGPT, Bing & Bard, I’m noticing the gold rush that disgusts me. These gold rushes surrounding new technology platform opportunities brings out the worst in people, like domain squatting and other low-creativity, low-effort races by people who could never make it without the early mover advantage.
The OpenAI plug-in ecosystem… very interesting! Generic I/O, but which OpenAI has a chance to read the input and craft the output, however, in all other ways, it’s just standard coding. So… so… so… this is interesting.
I don’t want to miss out on the learnings of Co-Pilot. Is that Microsoft or OpenAI? Check! It’s Github, so Microsoft… so I’ll use it in vim.
I need a General Web Scraper… but…
It can do things that none of the sandboxed Python code execution environments made available through OpenAI plug-ins.
In particular, I want to surf from my (or any IP) and run a full-fledged unbounded web browser for browser automation. Scraping in away that’s very difficult in sandboxed cloud environments. They don’t want to give you all the resources it takes for browser-based automation. They’ll let you use Requests, URLlib (of course) and maybe httpx if you’re lucky. But they’re unlikely to let you surf as Chrome logged in with your Google account, Edge logged in as Microsoft account, etc.
Fully Unbounded Web Scraper Kit
It works a lot like a command-line interface command, i.e. a Linux Shell command. It will let you write scripts whose genuine browser and genuine web-scraping output can be fed to OpenAI, responses retreived, and used for subsequent browsing automation commands.
FUWSK… fuwsker and fuwsking… which is a way of using local unbounded web scraping abilities in ways that are difficult on the cloud and can produce results that are highly competitive in that snowballing, compounding returns way I discussed.