통계/정산 배치 작업이 동기적 실행되어 전체적인 속도가 느림.
통계 → 정산 → 누적조회수 업데이트 순으로 처리하되 서로 독립적으로 실행되는 동영상과 광고 작업을 그룹화하여 각 작업이 비동기적으로 실행되도록 TaskExecutor를 사용해 구현함.
SchedulerConfig
@Configuration
@EnableScheduling
public class SchedulerConfig4 {
private final JobLauncher jobLauncher;
private final TaskExecutor taskExecutor;
private final Job videoJob;
private final Job adJob;
public SchedulerConfig4(JobLauncher jobLauncher,
Job videoJob, Job adJob,
@Qualifier("batchTaskExecutor") TaskExecutor taskExecutor) {
this.jobLauncher = jobLauncher;
this.videoJob = videoJob;
this.adJob = adJob;
this.taskExecutor = taskExecutor;
}
@Scheduled(cron = "*/5 * * * * *") // 매 5초마다 실행
public void runJobs() {
try {
System.out.println("[병렬] 잡 실행 시작");
List<Runnable> tasks = new ArrayList<>();
tasks.add(() -> runJob(videoJob, "병렬"));
tasks.add(() -> runJob(adJob, "병렬"));
CountDownLatch latch = new CountDownLatch(tasks.size());
for (Runnable task : tasks) {
taskExecutor.execute(() -> {
task.run();
latch.countDown();
});
}
latch.await();
// 요약 출력
printSummary();
} catch (Exception e) {
e.printStackTrace();
}
}
private JobExecution runJob(Job job, String mode) {
try {
JobParameters jobParameters = new JobParametersBuilder()
.addString("uuid", UUID.randomUUID().toString())
.addString("mode", mode)
.toJobParameters();
JobExecution jobExecution = jobLauncher.run(job, jobParameters);
while (jobExecution.isRunning()) {
Thread.sleep(100);
}
return jobExecution;
} catch (JobExecutionAlreadyRunningException | JobRestartException | JobInstanceAlreadyCompleteException |
JobParametersInvalidException | InterruptedException e) {
e.printStackTrace();
return null;
}
}
private void printSummary() {
System.out.println("\n=== 배치 Job 병렬처리 테스트 Summary ===");
if (CustomJobListener2.parallelStartTime > 0 && CustomJobListener2.parallelEndTime > 0) {
long parallelTotalTime = CustomJobListener2.parallelEndTime - CustomJobListener2.parallelStartTime;
System.out.println("병렬 총 소요시간: " + parallelTotalTime + " ms");
}
}
}
데이터규모
| 데이터규모 | |
|---|---|
| 동영상 시청기록 | 1,000,000 |
| 광고 시청기록 | 82,400,000 |
Tasklet 방식으로 구현한 Step/Job에 병렬 처리를 적용했을 때, 오히려 성능이 저하되어 Chunk 방식으로 구현함.
| 구분 | 순차 | 병렬 | 개선율 |
|---|---|---|---|
| Tasklet | 285003 ms | 358111 ms | −25.65% |
| Chunk | 275078 ms | 213056 ms | 22.55% |
최종 결과
순차처리
[순차] 잡 실행 시작
2024-08-01T13:57:35.064+09:00 INFO 7326 --- [catube-batch] [ scheduling-1] o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=statJob]] launched with the following parameters: [{'mode':'{value=순차, type=class java.lang.String, identifying=true}','uuid':'{value=90145d8f-3b90-485c-a5e0-f95602d0eaea, type=class java.lang.String, identifying=true}'}]
2024-08-01T13:57:35.090+09:00 INFO 7326 --- [catube-batch] [ scheduling-1] o.s.batch.core.job.SimpleStepHandler : Executing step: [videoStatStep]
2024-08-01T13:57:37.205+09:00 INFO 7326 --- [catube-batch] [ scheduling-1] o.s.batch.core.step.AbstractStep : Step: [videoStatStep] executed in 2s114ms
2024-08-01T13:57:37.213+09:00 INFO 7326 --- [catube-batch] [ scheduling-1] o.s.batch.core.job.SimpleStepHandler : Executing step: [adStatStep]
2024-08-01T14:02:09.974+09:00 INFO 7326 --- [catube-batch] [ scheduling-1] o.s.batch.core.step.AbstractStep : Step: [adStatStep] executed in 4m32s760ms
[순차 Job] statJob completed in 274901 ms
2024-08-01T14:02:09.982+09:00 INFO 7326 --- [catube-batch] [ scheduling-1] o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=statJob]] completed with the following parameters: [{'mode':'{value=순차, type=class java.lang.String, identifying=true}','uuid':'{value=90145d8f-3b90-485c-a5e0-f95602d0eaea, type=class java.lang.String, identifying=true}'}] and the following status: [COMPLETED] in 4m34s906ms
2024-08-01T14:02:09.992+09:00 INFO 7326 --- [catube-batch] [ scheduling-1] o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=billJob]] launched with the following parameters: [{'mode':'{value=순차, type=class java.lang.String, identifying=true}','uuid':'{value=61847db6-31c3-4b57-97e5-7be1f7cabba3, type=class java.lang.String, identifying=true}'}]
2024-08-01T14:02:09.999+09:00 INFO 7326 --- [catube-batch] [ scheduling-1] o.s.batch.core.job.SimpleStepHandler : Executing step: [videoBillStep]
2024-08-01T14:02:10.022+09:00 INFO 7326 --- [catube-batch] [ scheduling-1] o.s.batch.core.step.AbstractStep : Step: [videoBillStep] executed in 22ms
2024-08-01T14:02:10.036+09:00 INFO 7326 --- [catube-batch] [ scheduling-1] o.s.batch.core.job.SimpleStepHandler : Executing step: [adBillStep]
2024-08-01T14:02:10.150+09:00 INFO 7326 --- [catube-batch] [ scheduling-1] o.s.batch.core.step.AbstractStep : Step: [adBillStep] executed in 113ms
[순차 Job] billJob completed in 157 ms
2024-08-01T14:02:10.153+09:00 INFO 7326 --- [catube-batch] [ scheduling-1] o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=billJob]] completed with the following parameters: [{'mode':'{value=순차, type=class java.lang.String, identifying=true}','uuid':'{value=61847db6-31c3-4b57-97e5-7be1f7cabba3, type=class java.lang.String, identifying=true}'}] and the following status: [COMPLETED] in 159ms
2024-08-01T14:02:10.160+09:00 INFO 7326 --- [catube-batch] [ scheduling-1] o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=afterJob]] launched with the following parameters: [{'mode':'{value=순차, type=class java.lang.String, identifying=true}','uuid':'{value=3dd6559c-0906-4440-b3c6-4ad0c0ec0fa1, type=class java.lang.String, identifying=true}'}]
2024-08-01T14:02:10.164+09:00 INFO 7326 --- [catube-batch] [ scheduling-1] o.s.batch.core.job.SimpleStepHandler : Executing step: [videoDoneStep]
2024-08-01T14:02:10.170+09:00 INFO 7326 --- [catube-batch] [ scheduling-1] o.s.batch.core.step.AbstractStep : Step: [videoDoneStep] executed in 6ms
2024-08-01T14:02:10.177+09:00 INFO 7326 --- [catube-batch] [ scheduling-1] o.s.batch.core.job.SimpleStepHandler : Executing step: [adDoneStep]
2024-08-01T14:02:10.180+09:00 INFO 7326 --- [catube-batch] [ scheduling-1] o.s.batch.core.step.AbstractStep : Step: [adDoneStep] executed in 3ms
[순차 Job] afterJob completed in 20 ms
2024-08-01T14:02:10.183+09:00 INFO 7326 --- [catube-batch] [ scheduling-1] o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=afterJob]] completed with the following parameters: [{'mode':'{value=순차, type=class java.lang.String, identifying=true}','uuid':'{value=3dd6559c-0906-4440-b3c6-4ad0c0ec0fa1, type=class java.lang.String, identifying=true}'}] and the following status: [COMPLETED] in 21ms
=== 순차 처리 Summary ===
순차 총 소요시간: 275078 ms
병렬처리
[병렬] 잡 실행 시작
2024-08-01T13:49:45.065+09:00 INFO 6994 --- [catube-batch] [atch-Executor-2] o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=adJob]] launched with the following parameters: [{'mode':'{value=병렬, type=class java.lang.String, identifying=true}','uuid':'{value=0e02cb89-ab88-4573-86a4-637fe5d9e631, type=class java.lang.String, identifying=true}'}]
2024-08-01T13:49:45.065+09:00 INFO 6994 --- [catube-batch] [atch-Executor-1] o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=videoJob]] launched with the following parameters: [{'mode':'{value=병렬, type=class java.lang.String, identifying=true}','uuid':'{value=d796bd05-768b-46ec-b071-8fce65bd1561, type=class java.lang.String, identifying=true}'}]
2024-08-01T13:49:45.091+09:00 INFO 6994 --- [catube-batch] [atch-Executor-2] o.s.batch.core.job.SimpleStepHandler : Executing step: [adStatStep]
2024-08-01T13:49:45.092+09:00 INFO 6994 --- [catube-batch] [atch-Executor-1] o.s.batch.core.job.SimpleStepHandler : Executing step: [videoStatStep]
2024-08-01T13:49:47.145+09:00 INFO 6994 --- [catube-batch] [atch-Executor-1] o.s.batch.core.step.AbstractStep : Step: [videoStatStep] executed in 2s53ms
2024-08-01T13:49:47.155+09:00 INFO 6994 --- [catube-batch] [atch-Executor-1] o.s.batch.core.job.SimpleStepHandler : Executing step: [videoBillStep]
2024-08-01T13:49:47.176+09:00 INFO 6994 --- [catube-batch] [atch-Executor-1] o.s.batch.core.step.AbstractStep : Step: [videoBillStep] executed in 20ms
2024-08-01T13:49:47.182+09:00 INFO 6994 --- [catube-batch] [atch-Executor-1] o.s.batch.core.job.SimpleStepHandler : Executing step: [videoDoneStep]
2024-08-01T13:49:47.194+09:00 INFO 6994 --- [catube-batch] [atch-Executor-1] o.s.batch.core.step.AbstractStep : Step: [videoDoneStep] executed in 11ms
[병렬 Job] videoJob completed in 2118 ms
2024-08-01T13:49:47.203+09:00 INFO 6994 --- [catube-batch] [atch-Executor-1] o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=videoJob]] completed with the following parameters: [{'mode':'{value=병렬, type=class java.lang.String, identifying=true}','uuid':'{value=d796bd05-768b-46ec-b071-8fce65bd1561, type=class java.lang.String, identifying=true}'}] and the following status: [COMPLETED] in 2s125ms
2024-08-01T13:53:17.989+09:00 INFO 6994 --- [catube-batch] [atch-Executor-2] o.s.batch.core.step.AbstractStep : Step: [adStatStep] executed in 3m32s897ms
2024-08-01T13:53:18.034+09:00 INFO 6994 --- [catube-batch] [atch-Executor-2] o.s.batch.core.job.SimpleStepHandler : Executing step: [adBillStep]
2024-08-01T13:53:18.124+09:00 INFO 6994 --- [catube-batch] [atch-Executor-2] o.s.batch.core.step.AbstractStep : Step: [adBillStep] executed in 90ms
2024-08-01T13:53:18.128+09:00 INFO 6994 --- [catube-batch] [atch-Executor-2] o.s.batch.core.job.SimpleStepHandler : Executing step: [adDoneStep]
2024-08-01T13:53:18.131+09:00 INFO 6994 --- [catube-batch] [atch-Executor-2] o.s.batch.core.step.AbstractStep : Step: [adDoneStep] executed in 3ms
[병렬 Job] adJob completed in 213056 ms
2024-08-01T13:53:18.134+09:00 INFO 6994 --- [catube-batch] [atch-Executor-2] o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=adJob]] completed with the following parameters: [{'mode':'{value=병렬, type=class java.lang.String, identifying=true}','uuid':'{value=0e02cb89-ab88-4573-86a4-637fe5d9e631, type=class java.lang.String, identifying=true}'}] and the following status: [COMPLETED] in 3m33s61ms
=== 배치 Job 병렬처리 테스트 Summary ===
병렬 총 소요시간: 213056 ms