- I have a paginated REST API which I would like to poll
- Since the source data is growing, I would like to set this up as a scheduled sync job
and rerun it from the point where the previous execution left off (even if it was
COMPLETED
) basically advancing on the new data periodically AbstractPaginatedDataItemReader
seems to be suitable, it is able to:- Track the read item count (actually it is
AbstractItemCountingItemStreamItemReader
) - Maintain and persist its state in the
ExecutionContext
(read.count
)
- Track the read item count (actually it is
- What I'm missing is creating the new
JobParameters
based on the previousExecutionContext
- It seems
DefaultJobParametersExtractor
does the trick I need - And it is used from the
JobStep
class in a way I need - Unfortunately this is only available to
JobStep
and I have aTaskletStep
- So I ended up reimplementing the parts I need from
JobStep
andDefaultJobParametersExtractor
Is there a better way to do this?
The code in Example.java
is what I would like to get rid off by providing the RunIdIncrementer
and a JobParametersExtractor
.
Thank you very much, I really appreciate your help especially your effort providing the example. I have a few other questions, could you please take a look?
I'm not sure your example will work in my use case, could you please verify my thought process? When the job is finished, I don't know how much data was actually written. So based on the previous
JobParameters
I don't know how much I should increment the index. Maybe it wrote thousands of items, maybe zero, maybe it failed after the 10th item. So in the next round, I would like to know when the previous execution stopped not where it started.So the only way to know where the previous execution stopped is looking into its
ExecutionContext
where this information is persisted after every writes, right?I'm only using the
RunIdIncrementer
to increment therun.id
(so that I will not get an exception for the duplicateJobInstance
s), The index is "incremented" via getting it from the previousExecutionContext
:createJobParameters(job, executionContext.getLong("ResourceReader.index"))
.The
JobParametersIncrementer
(RunIdIncrementer
in my case) is called because I'm calling.getNextJobParameters(job)
on theJobParametersBuilder
, and it also seems, it works. (Example.java line#29)The
JobOperator
is a very good point, and I think I will use it, the only thing I need to take care about (this is not in the example above) is that I want to start the job from the scheduled method and from a controller. For the scheduled method, I want the start/run call to block while from the controller, I want a non-blocking call. Right now I have twoJobLauncher
s and I'm injecting them to the right place. I might need to create twoJobOperator
s one is using the blocking and the other one is the non-blockingJobLauncher
.