Here is a screenshot of the CSV open in Microsoft Excel<\/p>\n
View and Download the code from <\/b>Github<\/a><\/p>\n Besides typical Spring Boot dependencies, we include spring-boot-starter-batch, which is the dependency for Spring Batch as the name suggests, and hsqldb for an in-memory database. We also include commons-lang3 for ToStringBuilder.<\/p>\n This is a POJO that models the fields of an anime. The fields are:<\/p>\n What is important to note is our class constructor for the three fields: public AnimeDTO(String id, String title, String description). This will be used in our application. Also, as usual, we need to make a default constructor with no parameters or else Java will throw an error.<\/p>\n There is a lot going on in this class and it is not all written at once, so we are going to go through the code in steps. Visit Github<\/a> to see the code in its entirety. As the Spring Batch documentation<\/a> states FlatFileIteamReader will \u201cread lines of data from a flat file that typically describe records with fields of data defined by fixed positions in the file or delimited by some special character (e.g. Comma)\u201d.<\/p>\n We are dealing with a CSV, so of course the data is delimited by a comma, making this the perfect for use with our file.<\/p>\n Important points:<\/p>\n If we want to transform the data before writing it to the database, an ItemProcessor is necessary. Our code does not actually apply any business logic to transform the data, but we allow for the capability to.<\/p>\n csvAnimeProcessor<\/em> returns a new instance of the AnimeProcessor object which we review below.<\/p>\n If we wanted to apply business logic before writing to the database, you could manipulate the Strings before writing to the database. For instance, you could add toUpperCase()<\/em> after getTitle<\/em> to make the title upper case before writing to the database. However, I decided not to do that or apply any other business logic for this example processor, so no manipulation is being done. The Processor is here simply for demonstration.<\/p>\n The csvAnimeWriter<\/em> method is responsible for actually writing the values into our database. Our database is an in-memory HSQLDB however this application allows us to easily swap out one database for another. The dataSource<\/em> is autowired.<\/p>\n A Step<\/a> is a domain object that contains an independent, sequential phase of a batch job and contains all of the information needed to define and control the actual batch processing.<\/p>\n Now that we\u2019ve created the reader and processor for data we need to write it. For the reading, we\u2019ve been using chunk-oriented processing, meaning we\u2019ve been reading the data one at a time. Chunk-oriented processing also includes creating \u2018chunks\u2019 that will be written out, within a transaction boundary. For chunk-oriented processing, you set a commit interval and once the number of items read equals the commit interval that has been set, the entire chunk is written out via the ItemWriter, and the transaction is committed. We set the chunk interval size to 1.<\/p>\n I suggest reading the Spring Batch documentation about chunk-oriented processing<\/a>.<\/p>\n Then the reader, processor, and writer call the methods we wrote.<\/p>\n A Job consists of Steps. We pass a parameter into the Job below because we want to track the completion of the Job.<\/p>\n <\/p>\n The class below autowires the JdbcTemplate because we\u2019ve already set the dataSource and we want to easily make our query. The results of our are query are a list of AnimeDTO objects. For each object returned, we will create a message in our console to show that the item has been written to the database.<\/p>\n We need to create a schema for our database. As mentioned, we have made all fields Strings for ease of use, so we have made their data types VARCHAR.<\/p>\n This is a standard class with main(). As the Spring Documentation states, @SpringBootApplication<\/em> is a convenience annotation that includes @Configuration<\/em>, @EnableAutoConfiguration<\/em>, @EnableWebMvc<\/em>, and @ComponentScan<\/em>.<\/p>\n The FieldSet is fed through the processor and \u201cConverting\u201d is printed to the console. When the Spring Batch Job is finished, we select all the records and print them out to the console individually. When the Batch Process is complete this is what is printed to the console.<\/p>\n Spring Batch builds upon the POJO-based development approach and user-friendliness of the Spring Framework\u2019s to make it easy for developers to create enterprise grade batch processing.<\/p>\n The source code is on<\/b> Github<\/a><\/p>\n![\"\"](\"http:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2017\/10\/Screen-Shot-2017-10-12-at-8.19.19-AM.png\")
<\/a><\/p>\n1 \u2013 Project Structure<\/h2>\n
![\"\"](\"http:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2017\/10\/Screen-Shot-2017-10-12-at-7.33.41-AM.png\")
<\/a><\/p>\n2 \u2013 Project Dependencies<\/h2>\n
<?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\n<project xmlns=\"http:\/\/maven.apache.org\/POM\/4.0.0\" xmlns:xsi=\"http:\/\/www.w3.org\/2001\/XMLSchema-instance\"\r\n\txsi:schemaLocation=\"http:\/\/maven.apache.org\/POM\/4.0.0 http:\/\/maven.apache.org\/xsd\/maven-4.0.0.xsd\">\r\n\t<modelVersion>4.0.0<\/modelVersion>\r\n\r\n\t<groupId>com.michaelcgood<\/groupId>\r\n\t<artifactId>michaelcgood-spring-batch-csv<\/artifactId>\r\n\t<version>0.0.1<\/version>\r\n\t<packaging>jar<\/packaging>\r\n\r\n\t<name>michaelcgood-spring-batch-csv<\/name>\r\n\t<description>Michael C Good - Spring Batch CSV Example Application<\/description>\r\n\r\n\t<parent>\r\n\t\t<groupId>org.springframework.boot<\/groupId>\r\n\t\t<artifactId>spring-boot-starter-parent<\/artifactId>\r\n\t\t<version>1.5.7.RELEASE<\/version>\r\n\t\t<relativePath \/> <!-- lookup parent from repository -->\r\n\t<\/parent>\r\n\r\n\t<properties>\r\n\t\t<project.build.sourceEncoding>UTF-8<\/project.build.sourceEncoding>\r\n\t\t<project.reporting.outputEncoding>UTF-8<\/project.reporting.outputEncoding>\r\n\t\t<java.version>1.8<\/java.version>\r\n\t<\/properties>\r\n\r\n\t<dependencies>\r\n\t\t<dependency>\r\n\t\t\t<groupId>org.springframework.boot<\/groupId>\r\n\t\t\t<artifactId>spring-boot-starter-batch<\/artifactId>\r\n\t\t<\/dependency>\r\n\t\t<dependency>\r\n\t\t\t<groupId>org.springframework.boot<\/groupId>\r\n\t\t\t<artifactId>spring-boot-starter-data-jpa<\/artifactId>\r\n\t\t<\/dependency>\r\n\t\t<dependency>\r\n\t\t\t<groupId>org.springframework.boot<\/groupId>\r\n\t\t\t<artifactId>spring-boot-starter-web<\/artifactId>\r\n\t\t<\/dependency>\r\n\r\n\t\t<dependency>\r\n\t\t\t<groupId>org.hsqldb<\/groupId>\r\n\t\t\t<artifactId>hsqldb<\/artifactId>\r\n\t\t<\/dependency>\r\n\t\t<dependency>\r\n\t\t\t<groupId>org.springframework.boot<\/groupId>\r\n\t\t\t<artifactId>spring-boot-starter-test<\/artifactId>\r\n\t\t\t<scope>test<\/scope>\r\n\t\t<\/dependency>\r\n\t\t<!-- https:\/\/mvnrepository.com\/artifact\/org.apache.commons\/commons-lang3 -->\r\n\t\t<dependency>\r\n\t\t\t<groupId>org.apache.commons<\/groupId>\r\n\t\t\t<artifactId>commons-lang3<\/artifactId>\r\n\t\t\t<version>3.6<\/version>\r\n\t\t<\/dependency>\r\n\t<\/dependencies>\r\n\r\n\t<build>\r\n\t\t<plugins>\r\n\t\t\t<plugin>\r\n\t\t\t\t<groupId>org.springframework.boot<\/groupId>\r\n\t\t\t\t<artifactId>spring-boot-maven-plugin<\/artifactId>\r\n\t\t\t<\/plugin>\r\n\t\t<\/plugins>\r\n\t<\/build>\r\n\r\n\r\n<\/project><\/pre>\n
3 \u2013 Model<\/h2>\n
\n
package com.michaelcgood;\r\n\r\nimport org.apache.commons.lang3.builder.ToStringBuilder;\r\n\/**\r\n * Contains the information of a single anime\r\n *\r\n * @author Michael C Good michaelcgood.com\r\n *\/\r\n\r\npublic class AnimeDTO {\r\n\t\r\n\tpublic String getId() {\r\n\t\treturn id;\r\n\t}\r\n\r\n\tpublic void setId(String id) {\r\n\t\tthis.id = id;\r\n\t}\r\n\r\n\tpublic String getTitle() {\r\n\t\treturn title;\r\n\t}\r\n\r\n\tpublic void setTitle(String title) {\r\n\t\tthis.title = title;\r\n\t}\r\n\r\n\tpublic String getDescription() {\r\n\t\treturn description;\r\n\t}\r\n\r\n\tpublic void setDescription(String description) {\r\n\t\tthis.description = description;\r\n\t}\r\n\r\n\r\n\r\n\tprivate String id;\r\n\t\r\n\r\n\r\n\r\n\tprivate String title;\r\n\tprivate String description;\r\n\t\r\n\tpublic AnimeDTO(){\r\n\t\t\r\n\t}\r\n\t\r\n\tpublic AnimeDTO(String id, String title, String description){\r\n\t\tthis.id = id;\r\n\t\tthis.title = title;\r\n\t\tthis.description = title;\r\n\t}\r\n\t\r\n\r\n\t\r\n\t @Override\r\n\t public String toString() {\r\n\t\t return new ToStringBuilder(this)\r\n\t\t\t\t .append(\"id\", this.id)\r\n\t\t\t\t .append(\"title\", this.title)\r\n\t\t\t\t .append(\"description\", this.description)\r\n\t\t\t\t .toString();\r\n\t }\r\n\r\n\r\n}<\/pre>\n4 \u2013 CSV File to Database Configuration<\/h2>\n
4.1 \u2013 Reader<\/h3>\n
@Bean\r\n public FlatFileItemReader<AnimeDTO> csvAnimeReader(){\r\n FlatFileItemReader<AnimeDTO> reader = new FlatFileItemReader<AnimeDTO>();\r\n reader.setResource(new ClassPathResource(\"animescsv.csv\"));\r\n reader.setLineMapper(new DefaultLineMapper<AnimeDTO>() {{\r\n setLineTokenizer(new DelimitedLineTokenizer() {{\r\n setNames(new String[] { \"id\", \"title\", \"description\" });\r\n }});\r\n setFieldSetMapper(new BeanWrapperFieldSetMapper<AnimeDTO>() {{\r\n setTargetType(AnimeDTO.class);\r\n }});\r\n }});\r\n return reader;\r\n }<\/pre>\n\n
\n
4.2 \u2013 Processor<\/h3>\n
4.2.1 \u2013 Processor in CsvFileToDatabaseConfig.Java<\/h4>\n
@Bean\r\n\tItemProcessor<AnimeDTO, AnimeDTO> csvAnimeProcessor() {\r\n\t\treturn new AnimeProcessor();\r\n\t}<\/pre>\n4.2.2 \u2013 AnimeProcessor.Java<\/h4>\n
package com.michaelcgood;\r\n\r\nimport org.slf4j.Logger;\r\nimport org.slf4j.LoggerFactory;\r\n\r\nimport org.springframework.batch.item.ItemProcessor;\r\n\r\npublic class AnimeProcessor implements ItemProcessor<AnimeDTO, AnimeDTO> {\r\n\t\r\n private static final Logger log = LoggerFactory.getLogger(AnimeProcessor.class);\r\n \r\n @Override\r\n public AnimeDTO process(final AnimeDTO AnimeDTO) throws Exception {\r\n \t\r\n \tfinal String id = AnimeDTO.getId();\r\n final String title = AnimeDTO.getTitle();\r\n final String description = AnimeDTO.getDescription();\r\n\r\n final AnimeDTO transformedAnimeDTO = new AnimeDTO(id, title, description);\r\n\r\n log.info(\"Converting (\" + AnimeDTO + \") into (\" + transformedAnimeDTO + \")\");\r\n\r\n return transformedAnimeDTO;\r\n }\r\n\r\n}<\/pre>\n4.3 \u2013 Writer<\/h3>\n
@Bean\r\n\tpublic JdbcBatchItemWriter<AnimeDTO> csvAnimeWriter() {\r\n\t\t JdbcBatchItemWriter<AnimeDTO> excelAnimeWriter = new JdbcBatchItemWriter<AnimeDTO>();\r\n\t\t excelAnimeWriter.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<AnimeDTO>());\r\n\t\t excelAnimeWriter.setSql(\"INSERT INTO animes (id, title, description) VALUES (:id, :title, :description)\");\r\n\t\t excelAnimeWriter.setDataSource(dataSource);\r\n\t return excelAnimeWriter;\r\n\t}<\/pre>\n4.4 \u2013 Step<\/h3>\n
@Bean\r\n\tpublic Step csvFileToDatabaseStep() {\r\n\t\treturn stepBuilderFactory.get(\"csvFileToDatabaseStep\")\r\n\t\t\t\t.<AnimeDTO, AnimeDTO>chunk(1)\r\n\t\t\t\t.reader(csvAnimeReader())\r\n\t\t\t\t.processor(csvAnimeProcessor())\r\n\t\t\t\t.writer(csvAnimeWriter())\r\n\t\t\t\t.build();\r\n\t}<\/pre>\n4.5 \u2013 Job<\/h3>\n
@Bean\r\n\tJob csvFileToDatabaseJob(JobCompletionNotificationListener listener) {\r\n\t\treturn jobBuilderFactory.get(\"csvFileToDatabaseJob\")\r\n\t\t\t\t.incrementer(new RunIdIncrementer())\r\n\t\t\t\t.listener(listener)\r\n\t\t\t\t.flow(csvFileToDatabaseStep())\r\n\t\t\t\t.end()\r\n\t\t\t\t.build();\r\n\t}<\/pre>\n5 \u2013 Job Completion Notification Listener<\/h2>\n
package com.michaelcgood;\r\n\r\nimport java.sql.ResultSet;\r\nimport java.sql.SQLException;\r\nimport java.util.List;\r\n\r\nimport org.slf4j.Logger;\r\nimport org.slf4j.LoggerFactory;\r\n\r\nimport org.springframework.batch.core.BatchStatus;\r\nimport org.springframework.batch.core.JobExecution;\r\nimport org.springframework.batch.core.listener.JobExecutionListenerSupport;\r\nimport org.springframework.beans.factory.annotation.Autowired;\r\nimport org.springframework.jdbc.core.JdbcTemplate;\r\nimport org.springframework.jdbc.core.RowMapper;\r\nimport org.springframework.stereotype.Component;\r\n\r\n@Component\r\npublic class JobCompletionNotificationListener extends JobExecutionListenerSupport {\r\n\r\n\tprivate static final Logger log = LoggerFactory.getLogger(JobCompletionNotificationListener.class);\r\n\r\n\tprivate final JdbcTemplate jdbcTemplate;\r\n\r\n\t@Autowired\r\n\tpublic JobCompletionNotificationListener(JdbcTemplate jdbcTemplate) {\r\n\t\tthis.jdbcTemplate = jdbcTemplate;\r\n\t}\r\n\r\n\t@Override\r\n\tpublic void afterJob(JobExecution jobExecution) {\r\n\t\tif(jobExecution.getStatus() == BatchStatus.COMPLETED) {\r\n\t\t\tlog.info(\"============ JOB FINISHED ============ Verifying the results....\\n\");\r\n\r\n\t\t\tList<AnimeDTO> results = jdbcTemplate.query(\"SELECT id, title, description FROM animes\", new RowMapper<AnimeDTO>() {\r\n\t\t\t\t@Override\r\n\t\t\t\tpublic AnimeDTO mapRow(ResultSet rs, int row) throws SQLException {\r\n\t\t\t\t\treturn new AnimeDTO(rs.getString(1), rs.getString(2), rs.getString(3));\r\n\t\t\t\t}\r\n\t\t\t});\r\n\r\n\t\t\tfor (AnimeDTO AnimeDTO : results) {\r\n\t\t\t\tlog.info(\"Discovered <\" + AnimeDTO + \"> in the database.\");\r\n\t\t\t}\r\n\r\n\t\t}\r\n\t}\r\n\t\r\n}<\/pre>\n6 \u2013 SQL<\/h2>\n
DROP TABLE animes IF EXISTS;\r\nCREATE TABLE animes (\r\n id VARCHAR(10),\r\n title VARCHAR(400),\r\n description VARCHAR(999)\r\n);<\/pre>\n
6 \u2013 Main<\/h2>\n
package com.michaelcgood;\r\n\r\nimport org.springframework.boot.SpringApplication;\r\nimport org.springframework.boot.autoconfigure.SpringBootApplication;\r\n\r\n@SpringBootApplication\r\npublic class SpringBatchCsvApplication {\r\n\r\n\tpublic static void main(String[] args) {\r\n\t\tSpringApplication.run(SpringBatchCsvApplication.class, args);\r\n\t}\r\n}<\/pre>\n7 \u2013 Demo<\/h2>\n
7.1 \u2013 Converting<\/h3>\n
\n![\"\"](\"http:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2017\/10\/Screen-Shot-2017-10-12-at-7.53.11-AM.png\")
<\/a><\/p>\n7.2 \u2013 Discovering New Items In Database<\/h3>\n
\n![\"\"](\"http:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2017\/10\/Screen-Shot-2017-10-12-at-7.54.01-AM.png\")
<\/a><\/p>\n7.3 \u2013 Batch Process Complete<\/h3>\n
Job: [FlowJob: [name=csvFileToDatabaseJob]] completed with the following parameters: [{run.id=1, -spring.output.ansi.enabled=always}] and the following status: [COMPLETED]\r\nStarted SpringBatchCsvApplication in 36.0 seconds (JVM running for 46.616)<\/pre>\n8 \u2013 Conclusion<\/h2>\n