Preventing Duplicate Cron Job Executions in Distributed Systems Java base

In a distributed system, one of the most common problems developers face is duplicate execution of scheduled tasks (Cron Jobs). This typically happens when your application is deployed across multiple instances, and each instance independently triggers the same job. This can lead to issues such as:

• Sending duplicate emails or notifications.
• Processing payments multiple times.
• Redundant data processing.

To ensure only one instance executes a job at any given time, you need a distributed locking mechanism. In this article, we’ll explore and implement five solutions for preventing duplicate job execution in Spring Boot 3.4 and Spring 6:

1. ShedLock – Lightweight and database-based.
2. Quartz Scheduler – Advanced job scheduling across clusters.
3. Redis-Based Locking – Leveraging Redis for distributed locks.
4. Zookeeper-Based Locking – A reliable solution for large-scale distributed systems.
5. Custom Database Table Locking – A simple custom implementation using a relational database.

1. ShedLock

ShedLock ensures that only one instance of a scheduled job runs at a time by using a distributed lock stored in a shared database. It is lightweight, easy to set up, and works well for most distributed systems.

Implementation with ShedLock

Step 1: Add Dependencies

Add the following dependencies to your pom.xml:

<dependency>
    <groupId>net.javacrumbs.shedlock</groupId>
    <artifactId>shedlock-spring</artifactId>
    <version>5.5.0</version>
</dependency>
<dependency>
    <groupId>net.javacrumbs.shedlock</groupId>
    <artifactId>shedlock-provider-jdbc-template</artifactId>
    <version>5.5.0</version>
</dependency>
        

Step 2: Configure ShedLock

@Configuration
public class ShedLockConfig {
    @Bean
    public LockProvider lockProvider(DataSource dataSource) {
        return new JdbcTemplateLockProvider(
            JdbcTemplateLockProvider.Configuration.builder()
                .withJdbcTemplate(new JdbcTemplate(dataSource))
                .usingDbTime() // Use database time to avoid clock drift
                .build()
        );
    }
}
        

Step 3: Annotate Scheduled Jobs

@Service
public class ShedLockTask {

    @Scheduled(cron = "0 * * * * *") // Executes every minute
    @SchedulerLock(name = "ShedLockTask", lockAtMostFor = "10m", lockAtLeastFor = "1m")
    public void performTask() {
        System.out.println("Executing task with ShedLock...");
    }
}
        

2. Quartz Scheduler

Quartz Scheduler is a robust and feature-rich library designed for scheduling jobs in a cluster. It uses a Job Store to coordinate across multiple instances, ensuring that only one instance executes a job.

Implementation with Quartz Scheduler

Step 1: Add Dependency

Add the following dependency to your pom.xml:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-quartz</artifactId>
</dependency>
        

Step 2: Define a Quartz Job

public class QuartzTask implements Job {
    @Override
    public void execute(JobExecutionContext context) {
        System.out.println("Executing task with Quartz Scheduler...");
    }
}
        

Step 3: Configure Quartz

@Configuration
public class QuartzConfig {

    @Bean
    public JobDetail jobDetail() {
        return JobBuilder.newJob(QuartzTask.class)
            .withIdentity("quartzTask")
            .storeDurably()
            .build();
    }

    @Bean
    public Trigger trigger(JobDetail jobDetail) {
        return TriggerBuilder.newTrigger()
            .forJob(jobDetail)
            .withIdentity("quartzTrigger")
            .withSchedule(CronScheduleBuilder.cronSchedule("0 * * * * ?")) // Every minute
            .build();
    }
}
        

3. Redis-Based Locking

Redis can be used for distributed locking by leveraging its in-memory data store. Libraries like Redisson , Lettuce provide an easy-to-use API for managing locks.

Redisson is a Redis Java client that includes many common implementations of distributed Java collections, objects, and services. As a result, Redisson dramatically lowers the Redis learning curve for Java developers, making it easier than ever to build key-value Redis databases.

Lettuce describes itself as "a scalable Redis client for building non-blocking Reactive applications." The Lettuce project includes both synchronous and asynchronous support for the Redis API, including Java data structures, the publish/subscribe pattern, and high availability and scalability.

so for Reactive system you can use Lettuce and I prefer use Lettuce in my projects.

Implementation with Redis

Step 1: Add Dependency

Add the following dependency to your pom.xml:

<dependency>
    <groupId>org.redisson</groupId>
    <artifactId>redisson-spring-boot-starter</artifactId>
    <version>3.22.0</version>
</dependency>
        

Step 2: Configure Redisson

@Configuration
public class RedissonConfig {

    @Bean
    public RedissonClient redissonClient() {
        Config config = new Config();
        config.useSingleServer().setAddress("redis://localhost:6379");
        return Redisson.create(config);
    }
}
        

Step 3: Implement a Distributed Lock

@Service
public class RedisTask {

    private final RedissonClient redissonClient;

    public RedisTask(RedissonClient redissonClient) {
        this.redissonClient = redissonClient;
    }

    @Scheduled(cron = "0 * * * * *") // Executes every minute
    public void performTask() {
        RLock lock = redissonClient.getLock("redisTaskLock");
        if (lock.tryLock()) {
            try {
                System.out.println("Executing task with Redis lock...");
            } finally {
                lock.unlock();
            }
        } else {
            System.out.println("Task already running, skipping execution...");
        }
    }
}
        

4. Zookeeper-Based Locking

Apache Zookeeper is a high-performance, distributed coordination service. It provides advanced locking mechanisms suitable for large-scale distributed systems.

Implementation with Zookeeper

Step 1: Add Dependency

Add the following dependency to your pom.xml:

<dependency>
    <groupId>org.apache.curator</groupId>
    <artifactId>curator-recipes</artifactId>
    <version>5.5.0</version>
</dependency>
        

Step 2: Configure Zookeeper

@Configuration
public class ZookeeperConfig {

    @Bean
    public CuratorFramework curatorFramework() {
        CuratorFramework client = CuratorFrameworkFactory.newClient(
            "localhost:2181",
            new ExponentialBackoffRetry(1000, 3)
        );
        client.start();
        return client;
    }
}
        

Step 3: Implement a Distributed Lock

@Service
public class ZookeeperTask {

    private final CuratorFramework curatorFramework;

    public ZookeeperTask(CuratorFramework curatorFramework) {
        this.curatorFramework = curatorFramework;
    }

    @Scheduled(cron = "0 * * * * *") // Executes every minute
    public void performTask() {
        InterProcessMutex lock = new InterProcessMutex(curatorFramework, "/zookeeperTaskLock");
        try {
            if (lock.acquire(5, TimeUnit.SECONDS)) {
                try {
                    System.out.println("Executing task with Zookeeper lock...");
                } finally {
                    lock.release();
                }
            } else {
                System.out.println("Task already running, skipping execution...");
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}
        

5. Custom Database Table Locking

In this approach, a custom database table is used to track job execution. Instances check the table before executing a job and respect the lock if it exists.

Step 1: Create a Lock Table

CREATE TABLE job_lock (
    job_name VARCHAR(255) PRIMARY KEY,
    locked_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    locked_by VARCHAR(255)
);
        

Step 2: Implement a Lock Service

@Service
public class JobLockService {

    private final JdbcTemplate jdbcTemplate;

    public JobLockService(JdbcTemplate jdbcTemplate) {
        this.jdbcTemplate = jdbcTemplate;
    }

    public boolean tryLock(String jobName) {
        String instanceId = "Instance-" + System.currentTimeMillis();
        try {
            jdbcTemplate.update(
                "INSERT INTO job_lock (job_name, locked_at, locked_by) VALUES (?, ?, ?)",
                jobName, LocalDateTime.now(), instanceId
            );
            return true;
        } catch (DataIntegrityViolationException e) {
            return false;
        }
    }

    public void releaseLock(String jobName) {
        jdbcTemplate.update("DELETE FROM job_lock WHERE job_name = ?", jobName);
    }
}
        

Step 3: Schedule a Job

@Service
public class DatabaseTask {

    private final JobLockService jobLockService;

    public DatabaseTask(JobLockService jobLockService) {
        this.jobLockService = jobLockService;
    }

    @Scheduled(cron = "0 * * * * *") // Executes every minute
    public void performTask() {
        String jobName = "DatabaseTask";
        if (jobLockService.tryLock(jobName)) {
            try {
                System.out.println("Executing task with database lock...");
            } finally {
                jobLockService.releaseLock(jobName);
            }
        } else {
            System.out.println("Task already running, skipping execution...");
        }
    }
}
        

Conclusion

Preventing duplicate Cron Job executions is crucial in distributed systems to avoid redundant operations and ensure data consistency. Each of the solutions discussed here offers unique advantages:

• ShedLock: Lightweight and easy to implement.
• Quartz Scheduler: Ideal for advanced scheduling needs.
• Redis: High performance and in-memory locking.
• Zookeeper: Reliable for large-scale distributed systems.
• Custom Database Table: Simple and requires no additional infrastructure.

Choose the solution that best fits your system’s architecture, scale, and requirements to ensure smooth and coordinated job execution.