Pagination with backpressure

Pager.java

import rx.Observable;

public class Pager {

    public static void main(String[] args) {
        Observable.range(1, 1000)
                  .window(10)
                  .doOnNext(nextBatch -> System.out.println("Next page."))
                  .flatMap(nums -> nums, 1)
                  .forEach(System.out::println);
    }
}

In order to simulate "user delay" the code can be modified as

        Observable.range(1, 1000)
                  .window(10)
                  .doOnNext(nextBatch -> System.out.println("Next page."))
                  .flatMap(nums -> {
                      return nums.concatWith(Observable.timer(10, TimeUnit.SECONDS)
                                                       .map(Long::intValue));
                  }, 1)
                  .toBlocking()
                  .forEach(System.out::println);

This will emit one page every 10 seconds (maintaining backpressure to the source). If the timer is replaced by a user action, then it is equivalent to user clicking to get the next page.

I like the idea of approaching the problem by applying backpressure! We should explore that more. But you're still just buffering, not paging. What your code is doing is fetching 1000 JSON elements over the network, then traversing them in batches of 10. Not very efficient! Your code is based on the assumption that we have access to all elements right out the gate, which is ignoring the very problem of paging.

What you want to be doing instead is have a lazy stream: You emit a single page n, then only when the caller keeps subscribing, you even begin to look at how you fetch page n + 1. Perhaps if we bake defer into your solution somehow to construct them lazily it would work?

Another thing to keep in mind, how would you retry the last page in your example?

What your code is doing is fetching 1000 JSON elements over the network, then traversing them in batches of 10. Not very efficient!

Actually not, if the source supports backpressure. If you were to replace Observable.range(1, 1000) with a real HTTP request over the wire, you would make calls for a page per request(n) from the subscriber. So, you would just be querying one page at a time from the server.

Another thing to keep in mind, how would you retry the last page in your example?

I think what is more interesting in what you are asking is how does someone navigate between pages, eg: prev, next, jump to page # usecases. Since, RxJava backpressure always "moves forward", there is no way to address that solely by using backpressure.