• 통계/정산 배치 작업이 동기적 실행되어 전체적인 속도가 느림.

  • 통계 → 정산 → 누적조회수 업데이트 순으로 처리하되 서로 독립적으로 실행되는 동영상과 광고 작업을 그룹화하여 각 작업이 비동기적으로 실행되도록 TaskExecutor를 사용해 구현함.

  • SchedulerConfig

    @Configuration
    @EnableScheduling
    public class SchedulerConfig4 {
    
        private final JobLauncher jobLauncher;
        private final TaskExecutor taskExecutor;
        private final Job videoJob;
        private final Job adJob;
    
        public SchedulerConfig4(JobLauncher jobLauncher,
                                Job videoJob, Job adJob,
                                @Qualifier("batchTaskExecutor") TaskExecutor taskExecutor) {
            this.jobLauncher = jobLauncher;
            this.videoJob = videoJob;
            this.adJob = adJob;
            this.taskExecutor = taskExecutor;
        }
    
        @Scheduled(cron = "*/5 * * * * *") // 매 5초마다 실행
        public void runJobs() {
            try {
                System.out.println("[병렬] 잡 실행 시작");
    
                List<Runnable> tasks = new ArrayList<>();
                tasks.add(() -> runJob(videoJob, "병렬"));
                tasks.add(() -> runJob(adJob, "병렬"));
    
                CountDownLatch latch = new CountDownLatch(tasks.size());
    
                for (Runnable task : tasks) {
                    taskExecutor.execute(() -> {
                        task.run();
                        latch.countDown();
                    });
                }
    
                latch.await();
    
                // 요약 출력
                printSummary();
    
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    
        private JobExecution runJob(Job job, String mode) {
            try {
                JobParameters jobParameters = new JobParametersBuilder()
                        .addString("uuid", UUID.randomUUID().toString())
                        .addString("mode", mode)
                        .toJobParameters();
    
                JobExecution jobExecution = jobLauncher.run(job, jobParameters);
                while (jobExecution.isRunning()) {
                    Thread.sleep(100);
                }
                return jobExecution;
            } catch (JobExecutionAlreadyRunningException | JobRestartException | JobInstanceAlreadyCompleteException |
                     JobParametersInvalidException | InterruptedException e) {
                e.printStackTrace();
                return null;
            }
        }
    
        private void printSummary() {
            System.out.println("\n=== 배치 Job 병렬처리 테스트 Summary ===");
            if (CustomJobListener2.parallelStartTime > 0 && CustomJobListener2.parallelEndTime > 0) {
                long parallelTotalTime = CustomJobListener2.parallelEndTime - CustomJobListener2.parallelStartTime;
                System.out.println("병렬 총 소요시간: " + parallelTotalTime + " ms");
            }
        }
    }
    
    
  • 데이터규모

    데이터규모
    동영상 시청기록 1,000,000
    광고 시청기록 82,400,000
  • Tasklet 방식으로 구현한 Step/Job에 병렬 처리를 적용했을 때, 오히려 성능이 저하되어 Chunk 방식으로 구현함.

    구분 순차 병렬 개선율
    Tasklet 285003 ms 358111 ms −25.65%
    Chunk 275078 ms 213056 ms 22.55%
  • 최종 결과

    • 순차처리

      [순차] 잡 실행 시작
      2024-08-01T13:57:35.064+09:00  INFO 7326 --- [catube-batch] [   scheduling-1] o.s.b.c.l.support.SimpleJobLauncher      : Job: [SimpleJob: [name=statJob]] launched with the following parameters: [{'mode':'{value=순차, type=class java.lang.String, identifying=true}','uuid':'{value=90145d8f-3b90-485c-a5e0-f95602d0eaea, type=class java.lang.String, identifying=true}'}]
      2024-08-01T13:57:35.090+09:00  INFO 7326 --- [catube-batch] [   scheduling-1] o.s.batch.core.job.SimpleStepHandler     : Executing step: [videoStatStep]
      2024-08-01T13:57:37.205+09:00  INFO 7326 --- [catube-batch] [   scheduling-1] o.s.batch.core.step.AbstractStep         : Step: [videoStatStep] executed in 2s114ms
      2024-08-01T13:57:37.213+09:00  INFO 7326 --- [catube-batch] [   scheduling-1] o.s.batch.core.job.SimpleStepHandler     : Executing step: [adStatStep]
      2024-08-01T14:02:09.974+09:00  INFO 7326 --- [catube-batch] [   scheduling-1] o.s.batch.core.step.AbstractStep         : Step: [adStatStep] executed in 4m32s760ms
      [순차 Job] statJob completed in 274901 ms
      2024-08-01T14:02:09.982+09:00  INFO 7326 --- [catube-batch] [   scheduling-1] o.s.b.c.l.support.SimpleJobLauncher      : Job: [SimpleJob: [name=statJob]] completed with the following parameters: [{'mode':'{value=순차, type=class java.lang.String, identifying=true}','uuid':'{value=90145d8f-3b90-485c-a5e0-f95602d0eaea, type=class java.lang.String, identifying=true}'}] and the following status: [COMPLETED] in 4m34s906ms
      2024-08-01T14:02:09.992+09:00  INFO 7326 --- [catube-batch] [   scheduling-1] o.s.b.c.l.support.SimpleJobLauncher      : Job: [SimpleJob: [name=billJob]] launched with the following parameters: [{'mode':'{value=순차, type=class java.lang.String, identifying=true}','uuid':'{value=61847db6-31c3-4b57-97e5-7be1f7cabba3, type=class java.lang.String, identifying=true}'}]
      2024-08-01T14:02:09.999+09:00  INFO 7326 --- [catube-batch] [   scheduling-1] o.s.batch.core.job.SimpleStepHandler     : Executing step: [videoBillStep]
      2024-08-01T14:02:10.022+09:00  INFO 7326 --- [catube-batch] [   scheduling-1] o.s.batch.core.step.AbstractStep         : Step: [videoBillStep] executed in 22ms
      2024-08-01T14:02:10.036+09:00  INFO 7326 --- [catube-batch] [   scheduling-1] o.s.batch.core.job.SimpleStepHandler     : Executing step: [adBillStep]
      2024-08-01T14:02:10.150+09:00  INFO 7326 --- [catube-batch] [   scheduling-1] o.s.batch.core.step.AbstractStep         : Step: [adBillStep] executed in 113ms
      [순차 Job] billJob completed in 157 ms
      2024-08-01T14:02:10.153+09:00  INFO 7326 --- [catube-batch] [   scheduling-1] o.s.b.c.l.support.SimpleJobLauncher      : Job: [SimpleJob: [name=billJob]] completed with the following parameters: [{'mode':'{value=순차, type=class java.lang.String, identifying=true}','uuid':'{value=61847db6-31c3-4b57-97e5-7be1f7cabba3, type=class java.lang.String, identifying=true}'}] and the following status: [COMPLETED] in 159ms
      2024-08-01T14:02:10.160+09:00  INFO 7326 --- [catube-batch] [   scheduling-1] o.s.b.c.l.support.SimpleJobLauncher      : Job: [SimpleJob: [name=afterJob]] launched with the following parameters: [{'mode':'{value=순차, type=class java.lang.String, identifying=true}','uuid':'{value=3dd6559c-0906-4440-b3c6-4ad0c0ec0fa1, type=class java.lang.String, identifying=true}'}]
      2024-08-01T14:02:10.164+09:00  INFO 7326 --- [catube-batch] [   scheduling-1] o.s.batch.core.job.SimpleStepHandler     : Executing step: [videoDoneStep]
      2024-08-01T14:02:10.170+09:00  INFO 7326 --- [catube-batch] [   scheduling-1] o.s.batch.core.step.AbstractStep         : Step: [videoDoneStep] executed in 6ms
      2024-08-01T14:02:10.177+09:00  INFO 7326 --- [catube-batch] [   scheduling-1] o.s.batch.core.job.SimpleStepHandler     : Executing step: [adDoneStep]
      2024-08-01T14:02:10.180+09:00  INFO 7326 --- [catube-batch] [   scheduling-1] o.s.batch.core.step.AbstractStep         : Step: [adDoneStep] executed in 3ms
      [순차 Job] afterJob completed in 20 ms
      2024-08-01T14:02:10.183+09:00  INFO 7326 --- [catube-batch] [   scheduling-1] o.s.b.c.l.support.SimpleJobLauncher      : Job: [SimpleJob: [name=afterJob]] completed with the following parameters: [{'mode':'{value=순차, type=class java.lang.String, identifying=true}','uuid':'{value=3dd6559c-0906-4440-b3c6-4ad0c0ec0fa1, type=class java.lang.String, identifying=true}'}] and the following status: [COMPLETED] in 21ms
      
      === 순차 처리 Summary ===
      순차 총 소요시간: 275078 ms
      
    • 병렬처리

      [병렬] 잡 실행 시작
      2024-08-01T13:49:45.065+09:00  INFO 6994 --- [catube-batch] [atch-Executor-2] o.s.b.c.l.support.SimpleJobLauncher      : Job: [SimpleJob: [name=adJob]] launched with the following parameters: [{'mode':'{value=병렬, type=class java.lang.String, identifying=true}','uuid':'{value=0e02cb89-ab88-4573-86a4-637fe5d9e631, type=class java.lang.String, identifying=true}'}]
      2024-08-01T13:49:45.065+09:00  INFO 6994 --- [catube-batch] [atch-Executor-1] o.s.b.c.l.support.SimpleJobLauncher      : Job: [SimpleJob: [name=videoJob]] launched with the following parameters: [{'mode':'{value=병렬, type=class java.lang.String, identifying=true}','uuid':'{value=d796bd05-768b-46ec-b071-8fce65bd1561, type=class java.lang.String, identifying=true}'}]
      2024-08-01T13:49:45.091+09:00  INFO 6994 --- [catube-batch] [atch-Executor-2] o.s.batch.core.job.SimpleStepHandler     : Executing step: [adStatStep]
      2024-08-01T13:49:45.092+09:00  INFO 6994 --- [catube-batch] [atch-Executor-1] o.s.batch.core.job.SimpleStepHandler     : Executing step: [videoStatStep]
      2024-08-01T13:49:47.145+09:00  INFO 6994 --- [catube-batch] [atch-Executor-1] o.s.batch.core.step.AbstractStep         : Step: [videoStatStep] executed in 2s53ms
      2024-08-01T13:49:47.155+09:00  INFO 6994 --- [catube-batch] [atch-Executor-1] o.s.batch.core.job.SimpleStepHandler     : Executing step: [videoBillStep]
      2024-08-01T13:49:47.176+09:00  INFO 6994 --- [catube-batch] [atch-Executor-1] o.s.batch.core.step.AbstractStep         : Step: [videoBillStep] executed in 20ms
      2024-08-01T13:49:47.182+09:00  INFO 6994 --- [catube-batch] [atch-Executor-1] o.s.batch.core.job.SimpleStepHandler     : Executing step: [videoDoneStep]
      2024-08-01T13:49:47.194+09:00  INFO 6994 --- [catube-batch] [atch-Executor-1] o.s.batch.core.step.AbstractStep         : Step: [videoDoneStep] executed in 11ms
      [병렬 Job] videoJob completed in 2118 ms
      2024-08-01T13:49:47.203+09:00  INFO 6994 --- [catube-batch] [atch-Executor-1] o.s.b.c.l.support.SimpleJobLauncher      : Job: [SimpleJob: [name=videoJob]] completed with the following parameters: [{'mode':'{value=병렬, type=class java.lang.String, identifying=true}','uuid':'{value=d796bd05-768b-46ec-b071-8fce65bd1561, type=class java.lang.String, identifying=true}'}] and the following status: [COMPLETED] in 2s125ms
      2024-08-01T13:53:17.989+09:00  INFO 6994 --- [catube-batch] [atch-Executor-2] o.s.batch.core.step.AbstractStep         : Step: [adStatStep] executed in 3m32s897ms
      2024-08-01T13:53:18.034+09:00  INFO 6994 --- [catube-batch] [atch-Executor-2] o.s.batch.core.job.SimpleStepHandler     : Executing step: [adBillStep]
      2024-08-01T13:53:18.124+09:00  INFO 6994 --- [catube-batch] [atch-Executor-2] o.s.batch.core.step.AbstractStep         : Step: [adBillStep] executed in 90ms
      2024-08-01T13:53:18.128+09:00  INFO 6994 --- [catube-batch] [atch-Executor-2] o.s.batch.core.job.SimpleStepHandler     : Executing step: [adDoneStep]
      2024-08-01T13:53:18.131+09:00  INFO 6994 --- [catube-batch] [atch-Executor-2] o.s.batch.core.step.AbstractStep         : Step: [adDoneStep] executed in 3ms
      [병렬 Job] adJob completed in 213056 ms
      2024-08-01T13:53:18.134+09:00  INFO 6994 --- [catube-batch] [atch-Executor-2] o.s.b.c.l.support.SimpleJobLauncher      : Job: [SimpleJob: [name=adJob]] completed with the following parameters: [{'mode':'{value=병렬, type=class java.lang.String, identifying=true}','uuid':'{value=0e02cb89-ab88-4573-86a4-637fe5d9e631, type=class java.lang.String, identifying=true}'}] and the following status: [COMPLETED] in 3m33s61ms
      
      === 배치 Job 병렬처리 테스트 Summary ===
      병렬 총 소요시간: 213056 ms