Concurrent Web Crawling in Python
Today, I'm celebrating President's Day by accomplishing a mission: creating an async Python function to return zipped input/output promises. I'm also working on handling a list of tuples as input and building a package called mlseo on PyPI. To top it off, I've completed a concurrent site crawler in Python that uses SQLite. Join me in celebrating President's Day with this amazing progress!
Celebrating President's Day with Amazing Progress: Concurrent Web Crawling in Python!
By Michael Levin
Monday, February 21, 2022
Happy President’s Day.
Mission accomplished on returning zipped input / output promises on an async Python function. This makes using it for key/value database recording very easy. Next step: handling a list of tuples as input instead of a list of URLs.
We’re going to make a package on PyPI (Python Package Index) where things get pip installed from. The package will be called mlseo, and it will have lots of great things for doing generic broad SEO tasks.
We finished a site crawler. It’s concurrent. It’s Python. It uses SQLite.