Mike Levin SEO

Future-proof your technology-skills with Linux, Python, vim & git... and me!

← Journaling As Performance Art
Okay, I'm Finally Keeping Audiences In Mind →

Concurrent Web Crawling in Python

by Mike Levin

Monday, February 21, 2022

Happy President’s Day.

Mission accomplished on returning zipped input / output promises on an async Python function. This makes using it for key/value database recording very easy. Next step: handling a list of tuples as input instead of a list of URLs.

We’re going to make a package on PyPI (Python Package Index) where things get pip installed from. The package will be called mlseo, and it will have lots of great things for doing generic broad SEO tasks.

We finished a site crawler. It’s concurrent. It’s Python. It uses SQLite.