:keyset
:keyset is the fastest KEYSET paginator for ActiveRecord::Relation or Sequel::Dataset SQL ordered collections.
It uses a much faster pagination technique than OFFSET.
Check it out with bundle exec pagy keyset
When an UI is needed, and using KEYSET Pagination is possible, use the :keynav_js paginator to overcome almost all these constraints.
With any KEYSET pagination technique...
- You can only paginate from one page to the next: no jumping to arbitrary pages.
- The
setmust beuniquely ordered. Add the primary key (usually:id) as the last order column to be sure. - You should add the most suitable DB index for your ordering strategy, or it there will be no performance gain.
With Pagy :keyset...
You don't know the previous and the last page; you only know the first and next pages.
If you want to paginate backward, just call reverse_order on your set, and proceed forward.
Depending on your order requirements, here is how you set it up for maximum efficiency:
Just order by :id. It's fast out of the box without any setup...
set = collection.order(:id)
uniquely ordered
Option 1 Â If the combination of the values of the ordered columns is certainly unique...
set = collection.order(:brand).order(:model)
Option 2 Â If it's possibly not unique, just append a unique column to the order (e.g., your primary keys).
set = collection.order(:last_name).order(:first_name).order(:id)
•1
 The index must include the exact same columns and ordering direction of your set.
•2
 The index type must minimize the data scan (e.g. B-tree, B+ Tree, ...)
@pagy, @records = pagy(:keyset, set, **options)
@pagyis the pagination instance. It provides thereaders and the helpers to use in your code.@recordsis the eager-loadedArrayof the page records.
<!-- The only supported UI helper -->
<%== @pagy.next_tag(text: 'Next page >') %>
keyset: {...}- Set it only to force the
keysethash of column/order pairs. (It is set automatically from the set order) tuple_comparison: true- Enable the tuple comparison e.g.
(brand, id) > (:brand, :id). It works only with the same direction order, hence, it's ignored for mixed order. Check how your DB supports it (yourkeysetshould include onlyNOT NULLcolumns). pre_serialize: serializeSet it to a
lambdathat receives thekeyset_attributeshash. Modify this hash directly to customize the serialization of specific values (e.g., to preserveTimeobject precision). The lambda's return value is ignored.serialize = lambda do |attributes| # Convert it to a string matching the stored value/format in SQLite DB attributes[:created_at] = attributes[:created_at].strftime('%F %T.%6N') endlimit: 10- Specifies the number of items per page (default:
20) client_max_limit: 1_000Set the maximum
:limitthat the client is allowed torequest. Higher requested:limits are silently capped.IMPORTANT If falsey, the client cannot request any
:limit.page: force_page- Set it only to force the current
:page. (It is set automatically from the request param). request: request || hashPagy tries to find the
Rake::Requestatself.request. Set it only when it's not directly available in your code (e.g., Hanami, standalone app, test,...). For example:hash_request = { base_url: 'http://www.example.com', path: '/path', params: { 'param1' => 1234 }, # The string-keyed params hash from the request cookie: 'xyz' } # The 'pagy' cookie, only for keynavjsonapi: true- Enables JSON:API-compliant URLs with nested query string (e.g.,
?page[number]=2&page[size]=100). root_key: 'my_root'- Set it to enable nested URLs with nested query string
?my_root[page]=2&my_root[limit]=100)). Use it to handle multiple pagination objects in the same request. page_key: 'my_page'- Set it to change the key string used for the
:pagein URLs (default'page'). limit_key: 'my_limit'- Set it to change the key string used for the
:limitin URLs (default'limit').
next- The next page
page- The current page
limit- The items per page
in- The actual items in the page
records- The fetched records for the current page.
options- The hash of options of the object
There are a few peculiar aspects of the keyset pagination technique that you might not be familiar with. Here is a concise list:
offset pagination- The technique to fetch each page by incrementing the
offsetfrom the collection start.
It requires two queries per page (or only one if you use countless): it's slow toward the end of big tables.
It can be used for a rich frontend: it's the classic pagy pagination. keyset pagination- The technique to fetch the next page starting after the latest fetched record in a
uniquely orderedcollection.
It requires only one query per page (without OFFSET). if properly indexed, it's the fastest technique, regardless of the table size and position. Supports only infinite pagination, with no other frontend helpers. keynav pagination- The pagy exclusive technique to use
keysetpagination, providing nearly complete UI support. The fasted technique with UI capabilities. uniquely ordered- The property of a
set, when the concatenation of the values of the ordered columns is unique for each record. It is similar to a composite primarykeyfor the ordered table, but dynamically based on thekeysetcolumns. set- The
uniquely orderedActiveRecord::RelationorSequel::Datasetcollection to paginate. keyset- The hash of column/order pairs. Pagy extracts it from the order of the
set. keyset attributes- The hash of attributes of the
keysetcolumns of a record. cutoff- The value that identifies where the
pageends, and thenextone begins. It is theBase64URL-safe encoded string of the serialized array of thekeyet attributevalues. cutoffs- The array of
cutoffs of the known pagination state, used only by keynav to keep track of the visited pages during the navigation. They are cached in thesessionStorageof the client. page- The current
page, i.e. the page of records beginning after thecutoffof the previous page. Also the:pageoption, which is set to thecutoffof the previous page next- The next
page, i.e. the page of records beginning after thecutoff. Also thecutoffvalue retured by thenextmethod.
The cutoff of a page is the value that identifies where the page has ended, and the next one begins.
Let's consider an example of a simple set of 29 records, with an id column populated by character keys, and its order is: order(:id).
Assuming a LIMIT of 10, the "first page" will just include the first 10 records in the set: no cutoff value is known so far...
│ first page (10) >│ rest (19) >│
beginning of set >[· · · · · · · · · ·]· · · · · · · · · · · · · · · · · · ·]< end of set
At this point, it's the exact same first page pulled with OFFSET pagination, however, we don't want to use OFFSET to get the records after the first 10: that would be slow in a big table, so we need a way to identify the beginning of the next page without COUNTing the records (i.e., the whole point we want to avoid for performance).
So we read the id of the last one, which is the value X in our example... and that is the cutoff value of the first page. It can be described as: "the point up to the value X in the id column".
Notice that this is not like saying "up to the record X". It's important to understand that a cutoff refers just to a cutoff value of a column (or the values of multiple column, in case of multi-columns keysets).
Indeed, that very record could be deleted right after we read it, and our cutoff-X will still be the valid truth that we paginated the set up to the "X" value, cutting any further record off the page...
│ first page (10) >│ second page (10) >│ rest (9) >│
beginning of set >[· · · · · · · · · X]· · · · · · · · · ·]· · · · · · · · ·]< end of set
â–²
cutoff-X
For getting the next page of records (i.e. the "second page") we pull the next 10 records AFTER the cutoff-X. Again, we read the id of the last one, which is Y: so we have our new cutoff-Y, which is the end of the current page, and the next will go AFTER it...
│ first page (10) >│ second page (10) >│ last page (9) >│
beginning of set >[· · · · · · · · · X]· · · · · · · · · Y]· · · · · · · · ·]< end of set
â–² â–²
cutoff-X cutoff-Y
When we pull the next page from the cutoff-Y, we find only the remaining 9 records, which means that it's the "last page", which naturally ends with the end of the set, so it doesn't have any cutoff value to separate it from further records.
Keynotes
- A
cutoffidentifies a "cutoff value", for apagein theset. It is not a record, nor a reference to it. - Its value is extracted from the
keyset attributes valuesarray of the last record of thepage, converted to JSON, and encoded as a Base64 URL-safe string, for easy use in URLs.- The
:keysetpaginator embeds it in the request URL; the:keynav_jspaginator caches it on the clientsessionStorage.
- The
- All the
pages but the last, end with thecutoff. - All the
pages but the first, begin AFTER thecutoffof the previouspage.
Not unique combination...
Product.order(:name, :production_date)
The :id is usually the primary key...
Product.order(:name, :production_date, :id)
The generic to_json method used to encode the page may lose some information when decoded
- Check the actual executed DB query and the actual stored value
- Identify the column that has a format that doesn't match with the keyset
- Override the encoding with the
:pre_serialize option
The index has the wrong order, or it's the wrong type
- Ensure that the index reflects exactly the columns sequence and order of your keyset
- Use a B-tree or B+ Tree index. Use SQL
EXPLAIN ANALYZEor similar tool to confirm no table scan is performed. - With a same-direction order keysets, enabling the
:tuple_comparisonoption may help if your DB supports it.