Quartz is an open-source job scheduling framework that can be integrated into a wide variety of Java applications. It is used to schedule jobs that can be executed at a later time or on a repeating basis. Quartz is designed to be a simple and powerful scheduling framework that is easy to use and configure. But there is no built-in feature in the Quartz framework that allows for the automatic retrying of jobs that encounter exceptions during their execution. This article is inspired by . Here we consider using Quartz in a Spring application and implement an exponential random backoff policy that reduces the number of retry attempts, prevents overloading the system, allows the system to recover slowly over time, and distributes retry attempts evenly. Harvey Delaney’s blog post about Job Exception Retrying in Quartz.NET Storing retry parameters We will use -s that are associated with and objects of our job. ’s will store exponential random backoff policy parameters , , and . JobDataMap associated with Trigger will store . JobDataMap JobDetail Trigger JobDetail JobDataMap MAX_RETRIES RETRY_INITIAL_INTERVAL_SECS RETRY_INTERVAL_MULTIPLIER RETRY_NUMBER Job failure handling listener Now let’s add a Spring component that implements the interface. is called when the job is about to be executed and is called after job completion. In we will increment the counter. will contain the main code of exception handling and creating a new trigger for the next execution of a job. JobFailureHandlingListener org.quartz.JobListener JobListener#jobToBeExecuted JobListener#jobWasExecuted jobToBeExecuted RETRY_NUMBER jobWasExecuted Exponential random backoff policy Exponential random backoff is an algorithm used to progressively introduce delays in retrying an operation that has previously failed. The idea is to increase the delay between retries in an exponential manner, with each delay being chosen randomly from a range of possible values. This helps to avoid the scenario where multiple devices or systems all attempt to retry an operation at the same time, which can further exacerbate the problem that caused the operation to fail in the first place. Exponential random backoff is often used in distributed systems that need to retry operations that have failed. For example, it might be used in a distributed database to retry transactions that fail due to conflicts with other transactions or in a distributed file system to retry operations that fail due to network errors or the temporary unavailability of servers. We will use an exponential random backoff algorithm to calculate the time of the next attempt to start the job. Show me the code JobFailureHandlingListener.java package quartzdemo.listeners; import org.quartz.*; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.stereotype.Component; import java.util.Date; @Component public class JobFailureHandlingListener implements JobListener { private final String RETRY_NUMBER_KEY = "RETRY_NUMBER"; private final String MAX_RETRIES_KEY = "MAX_RETRIES"; private final int DEFAULT_MAX_RETRIES = 5; private final String RETRY_INITIAL_INTERVAL_SECS_KEY = "RETRY_INITIAL_INTERVAL_SECS"; private final int DEFAULT_RETRY_INITIAL_INTERVAL_SECS = 60; private final String RETRY_INTERVAL_MULTIPLIER_KEY = "RETRY_INTERVAL_MULTIPLIER"; private final double DEFAULT_RETRY_INTERVAL_MULTIPLIER = 1.5; private final Logger logger = LoggerFactory.getLogger(getClass()); @Override public String getName() { return "FailJobListener"; } @Override public void jobToBeExecuted(JobExecutionContext context) { context.getTrigger().getJobDataMap().merge(RETRY_NUMBER_KEY, 1, (oldValue, initValue) -> ((int) oldValue) + 1); } @Override public void jobExecutionVetoed(JobExecutionContext context) { } @Override public void jobWasExecuted(JobExecutionContext context, JobExecutionException jobException) { if (context.getTrigger().getNextFireTime() != null || jobException == null) { return; } int maxRetries = (int) context.getJobDetail().getJobDataMap() .computeIfAbsent(MAX_RETRIES_KEY, key -> DEFAULT_MAX_RETRIES); int timesRetried = (int) context.getTrigger().getJobDataMap().get(RETRY_NUMBER_KEY); if (timesRetried > maxRetries) { logger.error("Job with ID and class: " + context.getJobDetail().getKey() +", " + context.getJobDetail().getJobClass() + " has run " + maxRetries + " times and has failed each time.", jobException); return; } TriggerKey triggerKey = context.getTrigger().getKey(); int initialIntervalSecs = (int) context.getJobDetail().getJobDataMap() .computeIfAbsent(RETRY_INITIAL_INTERVAL_SECS_KEY, key -> DEFAULT_RETRY_INITIAL_INTERVAL_SECS); double multiplier = (double) context.getJobDetail().getJobDataMap() .computeIfAbsent(RETRY_INTERVAL_MULTIPLIER_KEY, key -> DEFAULT_RETRY_INTERVAL_MULTIPLIER); Date newStartTime = ExponentialRandomBackoffFixtures.getNextStartDate(timesRetried, initialIntervalSecs, multiplier); Trigger newTrigger = TriggerBuilder.newTrigger() .withIdentity(triggerKey) .startAt(newStartTime) .usingJobData(context.getTrigger().getJobDataMap()) .build(); newTrigger.getJobDataMap().put(RETRY_NUMBER_KEY, timesRetried); try { context.getScheduler().rescheduleJob(triggerKey, newTrigger); } catch (SchedulerException e) { throw new RuntimeException(e); } } } ExponentialRandomBackoffFixtures.java package quartzdemo.listeners; import org.apache.commons.lang3.RandomUtils; import java.time.Duration; import java.time.LocalDateTime; import java.time.ZoneId; import java.util.Date; public class ExponentialRandomBackoffFixtures { public static Date getNextStartDate(int timesRetried, int initialIntervalSecs, double multiplier) { double minValue = initialIntervalSecs * Math.pow(multiplier, timesRetried - 1); double maxValue = minValue * multiplier; Duration duration = Duration.ofMillis((long) (RandomUtils.nextDouble(minValue, maxValue) * 1000)); LocalDateTime nextDateTime = LocalDateTime.now().plus(duration); return Date.from(nextDateTime.atZone(ZoneId.systemDefault()).toInstant()); } } Now we can add the listener to a scheduler package quartzdemo; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.scheduling.annotation.EnableScheduling; import org.springframework.scheduling.quartz.SchedulerFactoryBean; import quartzdemo.listeners.JobFailureHandlingListener; @Configuration @EnableScheduling public class SchedulingConfiguration { private final JobFailureHandlingListener jobFailureHandlingListener; public SchedulingConfiguration2(JobFailureHandlingListener jobFailureHandlingListener) { this.jobFailureHandlingListener = jobFailureHandlingListener; } @Bean public SchedulerFactoryBean scheduler() { SchedulerFactoryBean schedulerFactory = new SchedulerFactoryBean(); // ... schedulerFactory.setGlobalJobListeners(jobFailureHandlingListener); return schedulerFactory; } } and run a job with a custom parameter RETRY_INITIAL_INTERVAL_SECS package quartzdemo.services; import org.quartz.*; import org.springframework.stereotype.Service; import quartzdemo.jobs.SimpleJob; import java.time.LocalDateTime; import java.time.ZoneId; import java.util.Date; import static quartzdemo.listeners.JobFailureHandlingListener.RETRY_INITIAL_INTERVAL_SECS_KEY; @Service public class SchedulingService { private final Scheduler scheduler; public SchedulingService(Scheduler scheduler) { this.scheduler = scheduler; } // ... private void scheduleJob() throws SchedulerException { Date afterFiveSeconds = Date.from(LocalDateTime.now().plusSeconds(5).atZone(ZoneId.systemDefault()).toInstant()); JobDetail jobDetail = JobBuilder.newJob(SimpleJob.class).usingJobData(RETRY_INITIAL_INTERVAL_SECS_KEY, 30).build(); Trigger trigger = TriggerBuilder.newTrigger().startAt(afterFiveSeconds).usingJobData("", "").build(); scheduler.scheduleJob(jobDetail, trigger); } } Conclusion Quartz is a robust framework that provides a wide range of capabilities. Even if it lacks something, it is always possible to implement the necessary functionality using the provided API. This allows developers to extend the framework to meet their specific requirements, so Quartz is a great choice for any scheduling needs.
