Future-proof your tech-skills with Linux, Python, vim & git as I share with you the most timeless and love-worthy tools in tech — and on staying valuable while machines learn... and beyond.

Concurrent Web Crawling in Python

by Mike Levin

Monday, February 21, 2022

Happy President’s Day.

Mission accomplished on returning zipped input / output promises on an async Python function. This makes using it for key/value database recording very easy. Next step: handling a list of tuples as input instead of a list of URLs.

We’re going to make a package on PyPI (Python Package Index) where things get pip installed from. The package will be called mlseo, and it will have lots of great things for doing generic broad SEO tasks.

We finished a site crawler. It’s concurrent. It’s Python. It uses SQLite.