TT#15: "Tech talk on Spring batch"

?? Unlocking the Power of Spring Batch! ??

???? Analogy: Picture yourself as a chef in a bustling restaurant, tasked with preparing a feast for hundreds of guests. You wouldn’t attempt to cook all the dishes simultaneously. Instead, you’d divide the task into smaller parts or “batches”. You might chop all the vegetables first, then marinate the meat, and finally bake the bread. Each of these tasks is manageable on its own, and by tackling them sequentially, you can whip up a meal for hundreds without breaking a sweat.

?? Spring Batch operates on a similar principle. It’s a framework within the Spring ecosystem designed to handle data in large volumes. It’s used for batch processing - that is, processing data in “batches” or chunks, rather than all at once.

Here are the technical details:

1. ?? Job: In Spring Batch, a Job is the entire process - it’s the feast you’re cooking. A job consists of one or more Steps.
2. ??♂? Step: A Step is a single task, like chopping the vegetables. In Spring Batch, a step involves reading data, processing it, and then writing it.
3. ?? Read: Spring Batch can read data from various sources, like databases, CSV files, XML files, etc.
4. ?? Process: Once the data is read, Spring Batch processes it. This could involve various operations like filtering, transforming, or aggregating the data.
5. ?? Write: After processing, Spring Batch writes the data to a specified location, like a database or a file.
6. ?? Chunk: A Chunk is a block of items that are processed together within a transaction boundary. For example, if you have 1000 records to process, you might decide to process them in chunks of 100. This means that Spring Batch would read 100 records, process them, write them, and then move on to the next 100.
7. ??? JobRepository: The JobRepository is a mechanism for storing all the metadata about configured jobs and executed job instances, step executions and their contexts.
8. ?? JobLauncher: The JobLauncher is a simple interface for launching a Job.

By breaking down the task into manageable chunks and handling each chunk one at a time, Spring Batch enables efficient and reliable data processing. ??

?? Unlocking the Power of Spring Batch: Pros and Cons!

?? Pros:

1. ?? Efficient Processing: Spring Batch is designed for efficient processing of large volumes of data. It can handle complex transformations and perform I/O operations with minimal system resources.
2. ?? Transaction Management: Spring Batch provides a robust transaction management system that ensures data consistency and reliability. If a job fails at any point, it can be restarted from where it left off.
3. ?? Scalability: Spring Batch supports parallel processing, which can significantly improve performance for large jobs. It can be scaled horizontally (across multiple servers) to handle larger data volumes.
4. ??? Error Handling: Spring Batch provides comprehensive error handling capabilities. It can skip records, retry operations, and even send notifications when a job fails.
5. ?? Integration: As part of the Spring ecosystem, Spring Batch integrates well with other Spring projects and Java technologies. This makes it a good choice for enterprise applications.

?? Cons:

1. ??♂? Complexity: Spring Batch has a steep learning curve. It’s a powerful tool, but it can be complex and intimidating for beginners.
2. ?? Overkill for Simple Tasks: If you only need to perform simple, one-off batch operations, using Spring Batch might be overkill. There are lighter, simpler tools available for such tasks.
3. ?? Debugging: Debugging Spring Batch applications can be challenging. The framework does a lot of work behind the scenes, which can make it difficult to track down and fix issues.
4. ?? Limited Documentation: While there is a fair amount of documentation available for Spring Batch, it can sometimes be lacking in detail, especially for more advanced features.

?? Unleashing the Power of Spring Batch: Real-world Marvels! ??

1. ?? E-commerce Order Processing:Scenario: Picture an e-commerce platform that needs to process large volumes of orders daily. Usage: Spring Batch can be employed to handle order processing in batches, validating orders, updating inventory, and generating invoices efficiently.
2. ?? Billing System for Utilities:Scenario: Utility companies often generate bills for millions of customers periodically. Usage: Spring Batch can process billing data, calculate charges, and generate bills in batches, ensuring accuracy and handling high volumes of billing information.
3. ?? Data Warehousing:Scenario: Data warehousing involves the extraction, transformation, and loading (ETL) of data from various sources into a centralized repository. Usage: Spring Batch can facilitate ETL processes, allowing for the extraction and processing of data from diverse sources into a data warehouse.
4. ?? Customer Onboarding in Banking:Scenario: Banks need to onboard new customers with diverse data sources and compliance checks. Usage: Spring Batch can handle customer data validation, identity verification, and compliance checks, ensuring a smooth onboarding process in batches.
5. ?? Report Generation in Healthcare:Scenario: Healthcare systems often require batch processing for generating reports from patient data. Usage: Spring Batch can be employed to process patient records, aggregate data, and generate various reports, such as monthly health summaries or billing reports.
6. ?? Log Analysis in IT Operations:Scenario: IT operations generate extensive logs, and analyzing them is crucial for identifying issues. Usage: Spring Batch can process log data in batches, perform analysis, and generate reports or alerts based on predefined criteria.
7. ?? Inventory Management in Retail:Scenario: Retailers deal with constant updates in inventory levels and stock replenishment. Usage: Spring Batch can handle inventory updates, reorder processing, and stock level checks in batches, ensuring accurate inventory management.
8. ?? Human Resources Data Processing:Scenario: HR departments often deal with bulk data, such as employee onboarding, payroll processing, and attendance records. Usage: Spring Batch can automate HR data processing tasks, from onboarding new employees to generating payroll reports, enhancing efficiency.

In these real-world scenarios, Spring Batch proves beneficial by providing a structured and scalable approach to handle large datasets, ensuring reliability, and facilitating the automation of complex business processes. ??

Below is the code snippet with detailed explanation. ??

@Configuration
@AllArgsConstructor
public class CustomerBatchJobConfig {

    private final CustomerRepository customerRepository;

    public FlatFileItemReader<Customer> itemReader() {
        FlatFileItemReader<Customer> itemReader = new FlatFileItemReader<>();
        itemReader.setResource(new FileSystemResource("src/main/resources/customers.csv"));
        itemReader.setName("csv-reader");
        itemReader.setLinesToSkip(1);
        itemReader.setLineMapper(lineMapper());
        return itemReader;
    }

    private LineMapper<Customer> lineMapper() {
        DefaultLineMapper lineMapper = new DefaultLineMapper();

        DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
        tokenizer.setDelimiter(",");
        tokenizer.setNames("id", "name", "age", "country", "email");
        tokenizer.setStrict(false);

        BeanWrapperFieldSetMapper mapper = new BeanWrapperFieldSetMapper();
        mapper.setTargetType(Customer.class);

        lineMapper.setFieldSetMapper(mapper);
        lineMapper.setLineTokenizer(tokenizer);
        return lineMapper;
    }

    @Bean
    public CustomerItemProcessor processor() {
        return new CustomerItemProcessor();
    }

    @Bean
    public RepositoryItemWriter<Customer> itemWriter() {
        RepositoryItemWriter<Customer> writer = new RepositoryItemWriter<>();
        writer.setRepository(customerRepository);
        writer.setMethodName("save");
        return writer;
    }

    @Bean
    public Step step(JobRepository repository, PlatformTransactionManager transactionManager) {
        return new StepBuilder("csv-step", repository)
                .<Customer, Customer>chunk(2, transactionManager)
                .reader(itemReader())
                .processor(processor())
                .writer(itemWriter())
//                .taskExecutor(taskExecutor())
                .build();
    }

    // Below method can be used if you want to run the batch concurrently
    private TaskExecutor taskExecutor() {
        SimpleAsyncTaskExecutor asyncTaskExecutor = new SimpleAsyncTaskExecutor();
        asyncTaskExecutor.setConcurrencyLimit(10);
        return asyncTaskExecutor;
    }

    @Bean
    public Job job(JobRepository jobRepository, PlatformTransactionManager transactionManager) {
        return new JobBuilder("csv-job", jobRepository)
                .flow(step(jobRepository, transactionManager))
                .end()
                .build();
    }
}        

@RestController
@AllArgsConstructor
public class CustomerBatchController {
    private final JobLauncher jobLauncher;

    @Autowired
    private final Job job;

    @GetMapping(value = "/startBatch")
    public BatchStatus startBatch() throws JobInstanceAlreadyCompleteException, JobExecutionAlreadyRunningException, JobParametersInvalidException, JobRestartException {
        JobParameters jobParameters = new JobParametersBuilder()
                .addLong("startAt", System.currentTimeMillis())
                .toJobParameters();
        JobExecution run = jobLauncher.run(job, jobParameters);
        return run.getStatus();
    }
}        

# H2 Database Configuration
spring.datasource.url=jdbc:h2:mem:testdb
spring.datasource.driverClassName=org.h2.Driver
spring.datasource.username=admin
spring.datasource.password=admin
spring.h2.console.enabled=true
spring.jpa.database-platform=org.hibernate.dialect.H2Dialect
spring.batch.initialize-jobs=false
spring.batch.job.enabled=false        

Explanation:

1. CustomerBatchJobConfig: This class is annotated with @Configuration, indicating that it’s a source of bean definitions for the application context. The @AllArgsConstructor annotation automatically generates a constructor with one parameter for each field in the class. In this case, it’s used to inject CustomerRepository.
2. itemReader(): This method sets up a FlatFileItemReader to read from a CSV file. The setResource() method sets the file to read from. setName() sets the name of the reader. setLinesToSkip(1) is used to skip the header line of the CSV file. setLineMapper(lineMapper()) sets the LineMapper to use, which is defined in the lineMapper() method.
3. lineMapper(): This method sets up a DefaultLineMapper to map lines from the CSV file to Customer objects. The DelimitedLineTokenizer is used to tokenize the lines read from the CSV file. The BeanWrapperFieldSetMapper is used to map the tokens to a Customer object.
4. processor(): This method sets up a CustomerItemProcessor that processes Customer objects. The processor filters out customers from Canada.
5. itemWriter(): This method sets up a RepositoryItemWriter that writes Customer objects to a database using a CustomerRepository.
6. step(): This method sets up a Step that reads, processes, and writes Customer objects. It uses the reader, processor, and writer defined earlier. It also sets the chunk size to 2, meaning it will read, process, and write two records at a time.
7. job(): This method sets up a Job that includes the step defined earlier.
8. CustomerItemProcessor: This class implements ItemProcessor interface and overrides the process() method. It filters out customers from Canada.
9. CustomerBatchController: This is a REST controller that exposes an endpoint to start the batch job. It uses JobLauncher to launch the job.
10. H2 Database Configuration: These are properties set in the application.properties file for configuring the H2 database and disabling the automatic job execution feature of Spring Batch.

In summary, this code reads customer data from a CSV file, filters out customers from Canada, and writes the remaining customers to a database. The batch job can be started by making a GET request to the /startBatch endpoint. The job reads and writes data in chunks of 2 and can be configured to run tasks concurrently. The H2 database is used for storing the data and the job metadata. The CustomerItemProcessor class is used to implement the business logic of filtering out customers from Canada. The CustomerBatchController class exposes an endpoint to start the batch job. The startBatch() method uses JobLauncher to launch the job with the current timestamp as a parameter. The H2 database configuration is used to set up an in-memory database for testing purposes. The spring.batch.initialize-jobs and spring.batch.job.enabled properties are set to false to disable the automatic execution of jobs on startup.

?? I’m thrilled to share that I’ve created a ‘Hello World’ project that demonstrates the workings of Spring Batch! This project is a great starting point for anyone looking to understand the basics of Spring Batch. ??

You can find the repository for this project on GitLab. Feel free to clone, explore, and let the learning begin! Here’s the link to the repo: Spring Batch Example ??

Happy Coding! ??

?? Join the Conversation: Share this post with your friends and colleagues who are passionate about web development and tech innovation. Let's learn and grow together. Your network will thank you! ??