Skip to content

Instantly share code, notes, and snippets.

@MarkMurphy
Last active February 5, 2024 19:59
Show Gist options
  • Save MarkMurphy/170e776940566f96e444adc2c54c6315 to your computer and use it in GitHub Desktop.
Save MarkMurphy/170e776940566f96e444adc2c54c6315 to your computer and use it in GitHub Desktop.
Cursor based pagination for Rails models.

Example Usage

Users

id name
1 Jane
2 Max
3 John
4 Scott
5 Mark
result = User.after(3) 
=>
[
  { id: 4, name: "Scott" },
  { id: 5, name: "Mark" }
]

result.has_more?
=> false

result.total_size
=> 2

result.total_count
=> 6
result = User.before(3) 
=>
[
  { id: 2, name: "Max" },
  { id: 1, name: "Jane" }
]

result.has_more?
=> false

result.total_size
=> 2

result.total_count
=> 6
result = User.after(1).limit(1)
=>
[
  { id: 2, name: "Max" }
]

result.has_more?
=> true

result.total_size
=> 5

result.total_count
=> 6
result = User.after(1).before(3)
=>
[
  { id: 2, name: "Max" }
]

result.has_more?
=> false

result.total_size
=> 1

result.total_count
=> 6
module Arel
module Nodes
module Cursor
class LessThan < Arel::Nodes::LessThan; end
class GreaterThan < Arel::Nodes::GreaterThan; end
end
end
module Predications
def before right
Nodes::Cursor::LessThan.new self, quoted_node(right)
end
def after right
Nodes::Cursor::GreaterThan.new self, quoted_node(right)
end
end
end
module Paginatable
extend ActiveSupport::Concern
included do
def self.all(*args)
super.extending(ActiveRecordRelationMethods)
end
def self.after(cursor)
reorder(primary_key => :asc).where(
arel_table[primary_key].after(cursor)
).extending(ActiveRecordRelationMethods)
end
def self.before(cursor)
reorder(primary_key => :desc).where(
arel_table[primary_key].before(cursor)
).extending(ActiveRecordRelationMethods)
end
end
module ActiveRecordRelationMethods
def has_more?
@has_more ||= begin
# Cache #size otherwise multiple calls to the database will occur.
(results_size = size) > 0 && results_size < total_size
end
end
# Returns number of records that exist in scope of the current cursor
def total_size(column_name = :all) #:nodoc:
# #count overrides the #select which could include generated columns
# referenced in #order, so skip #order here, where it's irrelevant to the
# result anyway.
@total_size ||= begin
context = except(:offset, :limit, :order)
# Remove includes only if they are irrelevant
context = context.except(:includes) unless references_eager_loaded_tables?
args = [column_name]
# .group returns an OrderedHash that responds to #count
context = context.count(*args)
if context.is_a?(Hash) || context.is_a?(ActiveSupport::OrderedHash)
context.count
else
context.respond_to?(:count) ? context.count(*args) : context
end
end
end
# Returns number of records that exist without :offset, :limit, :order, :before or :after
def total_count(column_name = :all) #:nodoc:
# #count overrides the #select which could include generated columns
# referenced in #order, so skip #order here, where it's irrelevant to the
# result anyway.
@total_count ||= begin
context = except(:offset, :limit, :order)
context.where_values = where_values.reject do |value|
value.is_a?(Arel::Nodes::Cursor::GreaterThan) ||
value.is_a?(Arel::Nodes::Cursor::LessThan)
end
# Remove includes only if they are irrelevant
context = context.except(:includes) unless references_eager_loaded_tables?
args = [column_name]
# .group returns an OrderedHash that responds to #count
context = context.count(*args)
if context.is_a?(Hash) || context.is_a?(ActiveSupport::OrderedHash)
context.count
else
context.respond_to?(:count) ? context.count(*args) : context
end
end
end
end
end
@MarkMurphy
Copy link
Author

MarkMurphy commented Sep 27, 2019

@jerrygreen

That's a bad implementation of cursor based pagination. You break the sorting.

It's not a bad implementation or nonsense. It served my use case just fine. Just because it doesn't fit yours doesn't make it bad. I did not intend for it to account for sorting on anything other than the primary key. This is how most cursor based apis work. It doesn't break your sorting, it intentionally removes it. You can't sort by arbitrary columns. The field you compare the cursor to has to be unique, sequential and immutable.

It's been several years since I posted this. I think the solution I would implement now is to generate an opaque token for the cursor that contains the necessary information. One cursor for the next page and one for the previous would be sent back to the client.

Making it opaque to the client means you can change the implementation server side, which is a win.

Slack has a decent write up on this: https://slack.engineering/evolving-api-pagination-at-slack-1c1f644f8e12

@shrdlu68
Copy link

shrdlu68 commented Apr 6, 2022

@jerrygreen you can always sort by name, id.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment