I’m very pleased to announce the release of my Split Batch processing library.
This library provides a way to imitate cron jobs in a discrete way on any platform, whether or not native cron is actually supported.
This can be useful for where cron-like batch processing is needed, for example, if you have a large dataset to geocode, but in circumstances where cron isn’t available, or, when it’s not desirable to expect the end-user to set up a cron job themselves.
This library can imitate cron jobs on any setup in much the same way as WordPress’ various cron mechanisms work.
Presently this library only works with WordPress, however fundamentally it uses the Doctrine DBAL so any Doctrine connection can be passed to the libraries connection module to make the library function in a much wider variety of contexts.
Defining a job is as simple as subclassing the provided base class and implementing an initialisation method and an iteration method.
A split batch daemon is then used to execute the job in sections, limited either by a maximum number of iterations or a maximum amount of milliseconds, whichever the daemon hits first. At this point execution will cease until the specified run frequency has been reached.
This means you can effectively process a very large dataset, for instance a gigantic CSV spreadsheet, in a discrete manner triggered by visitor’s HTTP requests.
The maximum iterations and maximum time ensure that the HTTP request isn’t held up for too long for the end user, but processing can still be done. The frequency options ensures that busy servers who may be receiving hundreds of requests per second cannot repeatedly trigger the job. The daemon will wait until the specified frequency period has elapsed before running another set of iterations on the job.
This library is something I’ve wanted to develop for a while now, I’ve implemented this over the last couple of days for an upcoming project I’ll be working on but wanted to make this available for broader use since it’s something I’ve been conceptualising for a while now.
This library is available on Packagist
composer require perry-rylance/split-batch