Skip to content

Instantly share code, notes, and snippets.

@damc-dev
Created August 14, 2018 17:29
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save damc-dev/e08bd52d7152e243a7edc5afb5c77e1a to your computer and use it in GitHub Desktop.
Save damc-dev/e08bd52d7152e243a7edc5afb5c77e1a to your computer and use it in GitHub Desktop.
[Note] Spring Batch - Cursor based vs page based item readers for retrieving records from a database

So from what I can tell from the docs [https://docs.spring.io/spring-batch/trunk/reference/html/readersAndWriters.html#database]

Cursor based item readers use a DB single connection load the entire ResultSet into app memory by default then the application iterates over the ResultSet with a cursor (So not a DB cursor) mapping one row and writing it out at a time so previous rows can be garbage collected.

Although you can set the maxRows to limit the amount of rows in the ResultSet at one time, then when the ResultSet needs more rows it will (using the same connection) fetch more (amount depends on the value of fetchSize). This continues until all rows from the query are loaded into the ResultSet and read.

Page based item readers make multiple queries each returning a different "page" of the results (size configurable with setPageSize method)

I assume cursor based item readers probably use more memory (unless configured appropriately) and be faster, where as the page based would typically consume less memory overall.

TLDR:

It really depends on the scenario, for our team we didn't see it making much difference given the low overhead of mapping each row and the number of records we would be processing in a batch.

The great thing about Spring Batch as it should be very easy to swap out readers later on if we have a need to optimize!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment