Queick: A Simple Job Queue Manager for Python
I developed a simple job queue system for Python.
Feature
- Written in Python only standard libraries
- Job-queue manager without redis
- Workingfor low-spec machines
- Retry
- Retry on network available
- Scheduling
- Thread-base job execution
Main target
Queick can be fully utilized in low-spec computers, such as Raspberry Pi. This system does not aim to be used at the server-side of a normal Web service.
Usage
First, install queick by using pip command.
$ pip install queick
Second, Second, prepare a job file (jobfunc.py) and an application (test.py).
# jobfunc.py
import time
def function(arg):
time.sleep(1)
print(arg)# test.py
from queick import JobQueue
from jobfunc import function
from time import timeq = JobQueue()
q.enqueue(function, args=(“hello”,))
q.enqueue_at(time() + 5, function, args=(“world”,)) # Run after 5 seconds
Third, start queick worker.
$ queickFinally, run the application.
$ python test.pyArchitecture
As mentioned above, a job is executed on a thread and implemented by concurrent.futures. Also, event notification is used instead of polling to receive jobs.
If a user specifies retry=True option to q.enqueue(), the job will be retried when it fails. The default retry function is Exponential backoff.
Application
Queick is integrated into an entrance recording system made of Raspberry Pi and NFC reader at my laboratory.
This system records the students’ entering/leaving history on Slack when they enter and leave by touching their student id.
However, this system has a problem. The Raspberry Pi often disconnects the Internet for some reason. Thus, a job queue system was needed to be integrated to retry the failed post.
I developed a job queue system from scratch because I did not want to install and operate Redis server on the Raspberry Pi
Conclusion
The core of Queick was made in 2 days, yes, my developing speed was quick, therefore, queick. I totally enjoyed developing it.
For further information, check repository asmsuechan/queick.
