Mike Levin SEO

Future-proof your technology-skills with Linux, Python, vim & git... and me!

← TechSmith Camtasia Took Away Screen Recorder Hide Taskbar Icon Feature
Use Python to Cycle Your IP with HMA VPN Software and Windows Automation →

Switching a Python Scheduler to Huey Task Queue Using Crontab API

by Mike Levin

Tuesday, August 09, 2022

Here’s a video I didn’t think I was going to do. I was going to switch over from one Python Scheduler to another, dealing with the changing “API-language” this entails, going from the very Pythonic “for humans” API of pip install schedule to the much less Pythonic crontab method of Huey, a scheduler with is also really a data pipeline task queue system like the much more popular Python Celery package but with much lower dependencies, so you can run it without all this devops stuff (tech liability).

Over my last several videos, I went from the pip install schedule package from PyPI that has an API that beautifully looks like this:

schedule.every(10).seconds.do(job)
schedule.every(10).minutes.do(job)
schedule.every().hour.do(job)
schedule.every().day.at("10:30").do(job)
schedule.every(5).to(10).minutes.do(job)
schedule.every().monday.do(job)
schedule.every().wednesday.at("13:15").do(job)
schedule.every().minute.at(":17").do(job)

# Or alternatively, use Python decorators:

@repeat(every(10).minutes)
def job():
    print("I am a scheduled job")

Task queue (for reliability). To pip install huey whose scheduling portion of their API looks like this: Huey is patterned after the much more popular Celery API.

@huey.periodic_task(crontab(minute='0', hour='3'))
def nightly_backup():
    sync_all_data()

@huey.periodic_task(crontab(minute='*/3'))
def every_three_minutes():
    print('This task runs every three minutes')

So you see, Schedule allows per-line standalone scheduling instructions with a well designed “API-language” plus the convenience of Python decorators where those “for humans” language conventions carry over.

Huey on the other hand only allows function decorators, which in and of itself is not a bad thing, but the API follows the long-standing Unix conventions of the cron program. Now while it’s generally good to follow the Unix/Linux API-standards, scheduling is one of the more difficult places to do so.

Switching my Munchkin container that currently uses the Python PyPI pip install Schedule package over to Huey is the subject of this video.

Just get to it and make it happen. Do a series of quick, small passes even if you’re producing videos to document it and help people at large dealing with these same issues.

How to test that your scheduler is running when all you have is periodic scheduling and you want a scheduled even to happen moments after you start the scheduler?

There were stumbling blocks having to do with “exact scheduling” vs. periodic scheduling, but we got it to work. Nuances:

Don’t forget to subscribe!