Skip to content

Instantly share code, notes, and snippets.

@MarkMurphy
Last active February 5, 2024 19:59
Show Gist options
  • Save MarkMurphy/170e776940566f96e444adc2c54c6315 to your computer and use it in GitHub Desktop.
Save MarkMurphy/170e776940566f96e444adc2c54c6315 to your computer and use it in GitHub Desktop.
Cursor based pagination for Rails models.

Example Usage

Users

id name
1 Jane
2 Max
3 John
4 Scott
5 Mark
result = User.after(3) 
=>
[
  { id: 4, name: "Scott" },
  { id: 5, name: "Mark" }
]

result.has_more?
=> false

result.total_size
=> 2

result.total_count
=> 6
result = User.before(3) 
=>
[
  { id: 2, name: "Max" },
  { id: 1, name: "Jane" }
]

result.has_more?
=> false

result.total_size
=> 2

result.total_count
=> 6
result = User.after(1).limit(1)
=>
[
  { id: 2, name: "Max" }
]

result.has_more?
=> true

result.total_size
=> 5

result.total_count
=> 6
result = User.after(1).before(3)
=>
[
  { id: 2, name: "Max" }
]

result.has_more?
=> false

result.total_size
=> 1

result.total_count
=> 6
module Arel
module Nodes
module Cursor
class LessThan < Arel::Nodes::LessThan; end
class GreaterThan < Arel::Nodes::GreaterThan; end
end
end
module Predications
def before right
Nodes::Cursor::LessThan.new self, quoted_node(right)
end
def after right
Nodes::Cursor::GreaterThan.new self, quoted_node(right)
end
end
end
module Paginatable
extend ActiveSupport::Concern
included do
def self.all(*args)
super.extending(ActiveRecordRelationMethods)
end
def self.after(cursor)
reorder(primary_key => :asc).where(
arel_table[primary_key].after(cursor)
).extending(ActiveRecordRelationMethods)
end
def self.before(cursor)
reorder(primary_key => :desc).where(
arel_table[primary_key].before(cursor)
).extending(ActiveRecordRelationMethods)
end
end
module ActiveRecordRelationMethods
def has_more?
@has_more ||= begin
# Cache #size otherwise multiple calls to the database will occur.
(results_size = size) > 0 && results_size < total_size
end
end
# Returns number of records that exist in scope of the current cursor
def total_size(column_name = :all) #:nodoc:
# #count overrides the #select which could include generated columns
# referenced in #order, so skip #order here, where it's irrelevant to the
# result anyway.
@total_size ||= begin
context = except(:offset, :limit, :order)
# Remove includes only if they are irrelevant
context = context.except(:includes) unless references_eager_loaded_tables?
args = [column_name]
# .group returns an OrderedHash that responds to #count
context = context.count(*args)
if context.is_a?(Hash) || context.is_a?(ActiveSupport::OrderedHash)
context.count
else
context.respond_to?(:count) ? context.count(*args) : context
end
end
end
# Returns number of records that exist without :offset, :limit, :order, :before or :after
def total_count(column_name = :all) #:nodoc:
# #count overrides the #select which could include generated columns
# referenced in #order, so skip #order here, where it's irrelevant to the
# result anyway.
@total_count ||= begin
context = except(:offset, :limit, :order)
context.where_values = where_values.reject do |value|
value.is_a?(Arel::Nodes::Cursor::GreaterThan) ||
value.is_a?(Arel::Nodes::Cursor::LessThan)
end
# Remove includes only if they are irrelevant
context = context.except(:includes) unless references_eager_loaded_tables?
args = [column_name]
# .group returns an OrderedHash that responds to #count
context = context.count(*args)
if context.is_a?(Hash) || context.is_a?(ActiveSupport::OrderedHash)
context.count
else
context.respond_to?(:count) ? context.count(*args) : context
end
end
end
end
end
@overdrivemachines
Copy link

Where do we put this file paginatable.rb?

@and0x000
Copy link

and0x000 commented Aug 8, 2018

@overdrivemachines that file goes to the directory app/models/concerns

@and0x000
Copy link

and0x000 commented Sep 17, 2018

I'd like to point out, that line 81 seems not work with Rails 5 due to some renamings.

This seems to do the trick:

[...]
        context.where_clause = where_values_hash.reject do |value|
[...]

@jerrygreen
Copy link

jerrygreen commented Sep 27, 2019

That's a bad implementation of cursor based pagination. You break the sorting. What if my collection sorted by name? But I want to get new collection skipping objects up to some id, but remain the same sorting options. I.e we have this:

id name
1 Jane
2 Max
3 John
4 Scott
5 Mark

User.order(:name) will return me full collection:

id name
1 Jane
3 John
5 Mark
2 Max
4 Scott

Of course I want to paginate, so I limit my collection with User.order(:name).limit(2) (in real life it could be not 2 but 20):

id name
1 Jane
3 John

And what if I want the second page? But I don't want to use offset, if you've got here to see "cursor pagination" instead of a simple pagination, then you probably know why you need this. So what?

I use User.order(:name).after(3).limit(2) and should get this:

id name
5 Mark
2 Max

But your solution will reorder my collection by id, returning some nonsense

@MarkMurphy
Copy link
Author

MarkMurphy commented Sep 27, 2019

@jerrygreen

That's a bad implementation of cursor based pagination. You break the sorting.

It's not a bad implementation or nonsense. It served my use case just fine. Just because it doesn't fit yours doesn't make it bad. I did not intend for it to account for sorting on anything other than the primary key. This is how most cursor based apis work. It doesn't break your sorting, it intentionally removes it. You can't sort by arbitrary columns. The field you compare the cursor to has to be unique, sequential and immutable.

It's been several years since I posted this. I think the solution I would implement now is to generate an opaque token for the cursor that contains the necessary information. One cursor for the next page and one for the previous would be sent back to the client.

Making it opaque to the client means you can change the implementation server side, which is a win.

Slack has a decent write up on this: https://slack.engineering/evolving-api-pagination-at-slack-1c1f644f8e12

@shrdlu68
Copy link

shrdlu68 commented Apr 6, 2022

@jerrygreen you can always sort by name, id.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment