:keyset


:keyset is the fastest KEYSET paginator for ActiveRecord::Relation or Sequel::Dataset SQL ordered collections.

It uses a much faster pagination technique than OFFSET.

Check it out with bundle exec pagy keyset

Depending on your order requirements, here is how you set it up for maximum efficiency:

Just order by :id. It's fast out of the box without any setup...

set = collection.order(:id)

Option 1   If the combination of the values of the ordered columns is certainly unique...

set = collection.order(:brand).order(:model)

Option 2   If it's possibly not unique, just append a unique column to the order (e.g., your primary keys).

set = collection.order(:last_name).order(:first_name).order(:id)

•1   The index must include the exact same columns and ordering direction of your set.
•2   The index type must minimize the data scan (e.g. B-tree, B+ Tree, ...)

Controller
@pagy, @records = pagy(:keyset, set, **options)
  • @pagy is the pagination instance. It provides the readers and the helpers to use in your code.
  • @records is the eager-loaded Array of the page records.
<!-- The only supported UI helper -->
<%== @pagy.next_tag(text: 'Next page &gt;') %>
keyset: {...}
Set it only to force the keyset hash of column/order pairs. (It is set automatically from the set order)
tuple_comparison: true
Enable the tuple comparison e.g. (brand, id) > (:brand, :id). It works only with the same direction order, hence, it's ignored for mixed order. Check how your DB supports it (your keyset should include only NOT NULL columns).
pre_serialize: serialize

Set it to a lambda that receives the keyset_attributes hash. Modify this hash directly to customize the serialization of specific values (e.g., to preserve Time object precision). The lambda's return value is ignored.

serialize = lambda do |attributes|
# Convert it to a string matching the stored value/format in SQLite DB
attributes[:created_at] = attributes[:created_at].strftime('%F %T.%6N')
end
limit: 10
Specifies the number of items per page (default: 20)
client_max_limit: 1_000

Set the maximum :limit that the client is allowed to request. Higher requested :limits are silently capped.

IMPORTANT If falsey, the client cannot request any :limit.

page: force_page
Set it only to force the current :page. (It is set automatically from the request param).
request: request || hash

Pagy tries to find the Rake::Request at self.request. Set it only when it's not directly available in your code (e.g., Hanami, standalone app, test,...). For example:

hash_request = { base_url: 'http://www.example.com',
                 path:     '/path',
                 params:   { 'param1' => 1234 }, # The string-keyed params hash from the request
                 cookie:   'xyz' }               # The 'pagy' cookie, only for keynav
jsonapi: true
Enables JSON:API-compliant URLs with nested query string (e.g., ?page[number]=2&page[size]=100).
root_key: 'my_root'
Set it to enable nested URLs with nested query string ?my_root[page]=2&my_root[limit]=100)). Use it to handle multiple pagination objects in the same request.
page_key: 'my_page'
Set it to change the key string used for the :page in URLs (default 'page').
limit_key: 'my_limit'
Set it to change the key string used for the :limit in URLs (default 'limit').
next
The next page
page
The current page
limit
The items per page
in
The actual items in the page
records
The fetched records for the current page.
options
The hash of options of the object

There are a few peculiar aspects of the keyset pagination technique that you might not be familiar with. Here is a concise list:

offset pagination
The technique to fetch each page by incrementing the offset from the collection start.
It requires two queries per page (or only one if you use countless): it's slow toward the end of big tables.
It can be used for a rich frontend: it's the classic pagy pagination.
keyset pagination
The technique to fetch the next page starting after the latest fetched record in a uniquely ordered collection.
It requires only one query per page (without OFFSET). if properly indexed, it's the fastest technique, regardless of the table size and position. Supports only infinite pagination, with no other frontend helpers.
keynav pagination
The pagy exclusive technique to use keyset pagination, providing nearly complete UI support. The fasted technique with UI capabilities.
uniquely ordered
The property of a set, when the concatenation of the values of the ordered columns is unique for each record. It is similar to a composite primary key for the ordered table, but dynamically based on the keyset columns.
set
The uniquely ordered ActiveRecord::Relation or Sequel::Dataset collection to paginate.
keyset
The hash of column/order pairs. Pagy extracts it from the order of the set.
keyset attributes
The hash of attributes of the keyset columns of a record.
cutoff
The value that identifies where the page ends, and the next one begins. It is the Base64 URL-safe encoded string of the serialized array of the keyet attribute values.
cutoffs
The array of cutoffs of the known pagination state, used only by keynav to keep track of the visited pages during the navigation. They are cached in the sessionStorage of the client.
page
The current page, i.e. the page of records beginning after the cutoff of the previous page. Also the :page option, which is set to the cutoff of the previous page
next
The next page, i.e. the page of records beginning after the cutoff. Also the cutoff value retured by the next method.

The cutoff of a page is the value that identifies where the page has ended, and the next one begins.

Let's consider an example of a simple set of 29 records, with an id column populated by character keys, and its order is: order(:id).

Assuming a LIMIT of 10, the "first page" will just include the first 10 records in the set: no cutoff value is known so far...

                  │ first page (10)  >│ rest (19)                          >│
beginning of set >[· · · · · · · · · ·]· · · · · · · · · · · · · · · · · · ·]< end of set

At this point, it's the exact same first page pulled with OFFSET pagination, however, we don't want to use OFFSET to get the records after the first 10: that would be slow in a big table, so we need a way to identify the beginning of the next page without COUNTing the records (i.e., the whole point we want to avoid for performance).

So we read the id of the last one, which is the value X in our example... and that is the cutoff value of the first page. It can be described as: "the point up to the value X in the id column".

Notice that this is not like saying "up to the record X". It's important to understand that a cutoff refers just to a cutoff value of a column (or the values of multiple column, in case of multi-columns keysets).

Indeed, that very record could be deleted right after we read it, and our cutoff-X will still be the valid truth that we paginated the set up to the "X" value, cutting any further record off the page...

                  │ first page (10)  >│ second page (10) >│ rest (9)       >│
beginning of set >[· · · · · · · · · X]· · · · · · · · · ·]· · · · · · · · ·]< end of set
                                      â–²
                                   cutoff-X

For getting the next page of records (i.e. the "second page") we pull the next 10 records AFTER the cutoff-X. Again, we read the id of the last one, which is Y: so we have our new cutoff-Y, which is the end of the current page, and the next will go AFTER it...

                  │ first page (10)  >│ second page (10) >│ last page (9)  >│
beginning of set >[· · · · · · · · · X]· · · · · · · · · Y]· · · · · · · · ·]< end of set
                                      â–²                   â–²
                                   cutoff-X            cutoff-Y

When we pull the next page from the cutoff-Y, we find only the remaining 9 records, which means that it's the "last page", which naturally ends with the end of the set, so it doesn't have any cutoff value to separate it from further records.

Order issue
Append the primary key to the order

Not unique combination...

Product.order(:name, :production_date)

The :id is usually the primary key...

Product.order(:name, :production_date, :id)
Encoding issue
Solution

The generic to_json method used to encode the page may lose some information when decoded

  • Check the actual executed DB query and the actual stored value
  • Identify the column that has a format that doesn't match with the keyset
  • Override the encoding with the :pre_serialize option
Index issue
Solutions

The index has the wrong order, or it's the wrong type

  • Ensure that the index reflects exactly the columns sequence and order of your keyset
  • Use a B-tree or B+ Tree index. Use SQL EXPLAIN ANALYZE or similar tool to confirm no table scan is performed.
  • With a same-direction order keysets, enabling the :tuple_comparison option may help if your DB supports it.