Concurrent Web Crawling in Python

by Mike Levin

Happy President’s Day.

Mission accomplished on returning zipped input / output promises on an async Python function. This makes using it for key/value database recording very easy. Next step: handling a list of tuples as input instead of a list of URLs.

We’re going to make a package on PyPI (Python Package Index) where things get pip installed from. The package will be called mlseo, and it will have lots of great things for doing generic broad SEO tasks.

We finished a site crawler. It’s concurrent. It’s Python. It uses SQLite.