Cron Jobs vs Events for async data processing

Let's consider this scenario:

You are building a system which integrates with a payment provider, and you're storing and managing user transactions while calling the third party payments API.

The caveat is that the 3rd party payment process is asynchronous, meaning you can initiate a payment with one API, and the payment will be confirmed through a webhook/callback.

Let's say the payment provider gives a spec stating that the average time to complete a transaction is up to 1 minute.

As with any other 3rd party provider, we have to consider failures and retries. One of the noted failure scenarios here, is that the payment provider fails to call our webhook/callback after the transaction is completed. This happens 2-3% of the time according to the provider.

We have to be able to sync the transaction status of the payment provider with the records in our system. There are two ways of achieving this:

Using a Cron job

A task will run every 2 minutes that fetches the 'pending' transactions, and will call a payments API to get the transaction details, check the status of each transaction, and update the records in our database accordingly.

Using delayed events

After a transaction is initiated, we can schedule a delayed event (to be fired after 2 minutes), which will be consumed by a worker. The worker can fetch teh transaction record from our database, and if the status is still pending, they can call the payment API to get the transaction details, and update the record accordingly.

Which method should you use? Of course, it depends.

Pros and cons

• Events are self managed and focused; they only deal with a single transaction, and retry is simple since the worker can trigger itself again with a delay. The code is also simpler, since you're handling a single transaction. However, if there is a large volume of transactions, let's say N, then using events has a drawback of having to fire N number of events, which will subsequently do N number of database fetch queries.
• On the flipside, Cron jobs run on a schedule and will only fetch the pending transactions. So instead of validating N number of transactions, you only validate a small portion of potential failures, while fetching them using a single database query. The code gets a bit more complex since you're handling data in batches, but you're getting a more optimized approach for database interactions.

Conclusion

The Cron job implementation is simpler since you only fetch the pending transactions, and process them accordingly, while the events based implementation is more distributed and entails validating every single transaction and multiple queries.

For this choice, it seems the trade off is centered around the volume of transactions. For high volume transactions, cron jobs would be preferred since that would reduce the number of external API calls, and also avoid flogging the database and background workers. For lower volume of transactions, the delayed events approach makes sense since you'd get near real-time updates without putting too much load on your system.