In this post I will show you how to implement cursor-based pagination for Laravel applications using Eloquent.
In a typical web application that uses a database you might have more records than you can fit on a page or in a single result set from a query. To retrieve more data, there are basically two options: Offset-based and cursor-based pagination.
Offset-based pagination
Offset pagination is the most common form of pagination, e.g. for blog posts. You see a list of records and on the bottom there is a pagination bar you can use to navigate to a specific part of the dataset by clicking on a page number. Offset-based pagination uses a LIMIT and an OFFSET clause to control the records to load.
OFFSET is the number of records to skip before a specified and limited amount of rows is retrieved thereafter. Implementing offset-based pagination for Eloquent results in Laravel is pretty easy, as there is an official integration you can check out in the docs.
Cursor-based pagination
Sometimes, offset pagination leads to problems. One of them is performance. If the OFFSET gets really large, the database has to scan through lots of records just to find the right starting point for retrieving a relatively small amount of rows for the next page.
If you are dealing with data that is likely to change, another problem occurs. It may happen that offset pagination yields partially the same results multiple times and you have to deal with the duplication in application code, which sometimes is not obvious.
This is where cursor pagination comes into play. A cursor is a unique identifier for a specific record, from where further querying has to start to get the next page of results. Cursor pagination on the backend is the perfect counterpart to an infinite scrollable list of results on the frontend and makes its implementation really easy.
The idea of cursor pagination is that there is a clear unique order on the dataset. As a result of this, we can remove records we don't want to have in our results using a WHERE clause. This is very efficient, since our database can use indexes in this case. Additionally, we use LIMIT to restrict the amount of results per page and calculate the next cursor from the last result row, which is passed in on the next query as a starting point to get further results.
Cursor-based pagination with Eloquent
So let's see how to implement cursor based pagination with Laravel's ORM Eloquent. Most of this code will be living in a cursorPaginate($limit, $columns) macro on the Illuminate\Database\Eloquent\Builder at the end of the day, so I just pick out and explain the most interesting parts before I provide the full source code. $this always references to the instance of the Builder in the following code snippets.
First, we get the current cursor from the URL by calling $cursor = CursorPaginator::currentCursor();. A cursor is basically just an array that contains all relevant column values that identify a row. The names of those columns are passed by the user as keys of the $columns array, but later more about that.
Cursor-based pagination using multiple columns
As I've said before, a clear unique order has to be defined on the dataset to prevent fetching the same results twice and to work with a stable cursor. Of course, an obvious choice would be the table's primary key, e.g. the id column. But what happens if you want to order the results based on the created_at column, a user-specified column, or even multiple columns with different sort directions? Or what if you can't just order by id to mimic ordering by created_at (which is an ugly hack anyways), because your primary keys are UUIDs that are not sequential? Stay tuned, I've got you covered.
The created_at column is obviously not guaranteed to be unique, since multiple records could be created at the same time, so we would miss results. Therefore, our cursor has to combine created_at and at least one truly unique column to clearly identify a specific record of our dataset, like id.
As a result of that, we have to use created_at and id for the ORDER BY clause of the query to get a stable order in addition to a WHERE clause, that removes all results before the values specified by the cursor, including the cursor's own values.
Let's have a look at the mathematical background to find out how to create this WHERE clause. We have to use tuple comparison to remove all tuples from the results that are less than or equals the cursor's value.
So let a, b, c, d be columns. (a, b), (c, d) represent rows of a table. (a, b) > (c, d) is basically the same as a > c or (a = c and (b > d)). So let's expand this to use a third column: (a, b, c) > (d, e, f) is basically the same as a > d or (a = d and (b > e or (b = e and (c > f)))). You see the recursive pattern?
A recursive apply function for tuple comparison
We are not using tuple comparison in SQL directly (which would be possible), because basic tuple comparison is too inflexible for our use case. So we construct our WHERE clause using a recursive function. This allows us to specify different sort directions for different columns, e.g. created_at could be sorted descending and id could be sorted ascending at the same time, so we have the most flexibility.
$apply = function ($query, $columns, $cursor) use (&$apply) {
$query->where(function ($query) use ($columns, $cursor, $apply) {
$column = key($columns);
$direction = array_shift($columns);
$value = array_shift($cursor);
$query->where($column, $direction === 'asc' ? '>' : '<', $value);
if (! empty($columns)) {
$query->orWhere($column, $value);
$apply($query, $columns, $cursor);
}
});
};
$apply($this, $columns, $cursor);
$apply basically takes an Eloquent Query, an array of columns that specify the sort order, e.g. $columns = ['created_at' => 'desc', 'id' => 'asc'], and an array containing the values stored in the cursor, e.g. ['2019-06-22 17:01:14', 42], where the values contain the last created_at and id values of the previous query. Using this data as input, apply constructs the WHERE clause recursively based on the concepts of tuple comparison shown in the previous section.
Get the results in the right order
We have enforce a stable order for getting stable results. Therefore, we simply add an ORDER BY clause that appends the sort directions from the $columns array to the query.
foreach ($columns as $column => $direction) {
$this->orderBy($column, $direction);
}
Limit the results and create the next cursor
Finally, we append a LIMIT clause that is equals to the amount of records per page the user wants to retrieve.
But why the $limit + 1? That's a little trick to find out if there is another page of results available. We fetch one row more than required, which is later removed from the results. So we know we have to create another cursor, because there is at least one more result available that lives on the next page.
$items = $this->limit($limit + 1)->get();
if ($items->count() <= $limit) {
return new CursorPaginator($items);
}
$items->pop();
return new CursorPaginator($items, array_map(function ($column) use ($items) {
return $items->last()->{$column};
}, array_keys($columns)));
If a next page is available, we create the next cursor by specifying the second argument for CursorPaginator, which is an array of the relevant values of the last row of the current results to include in the next cursor.
The CursorPaginator helper class
We are returning an instance of CursorPaginator to the class that calls our cursorPaginate macro, e.g. a controller. It makes working with cursors and results simpler, since it takes care of encoding the cursor properly for its use in URLs, and exposes the items to respond, the next cursor URL, as well as the current cursor. In addition, you can specify additional query string parameters that are appended to the next cursor URL using appends.
class CursorPaginator
{
protected $items;
protected $nextCursor;
protected $params = [];
public function __construct($items, $nextCursor = null)
{
$this->items = $items;
$this->nextCursor = $nextCursor;
}
public static function currentCursor()
{
return json_decode(base64_decode(request('cursor')));
}
public function appends($params)
{
$this->params = $params;
return $this;
}
public function items()
{
return $this->items;
}
public function nextCursorUrl()
{
return $this->nextCursor ? url()->current().'?'.http_build_query(array_merge([
'cursor' => base64_encode(json_encode($this->nextCursor)),
], $this->params)) : null;
}
}
Extend the Eloquent Builder with a cursorPaginate macro
Let's put all this stuff together I've shown you before and create a cursorPaginate macro on the Eloquent Builder.
use Illuminate\Database\Eloquent\Builder;
Builder::macro('cursorPaginate', function ($limit, $columns) {
$cursor = CursorPaginator::currentCursor();
if ($cursor) {
$apply = function ($query, $columns, $cursor) use (&$apply) {
$query->where(function ($query) use ($columns, $cursor, $apply) {
$column = key($columns);
$direction = array_shift($columns);
$value = array_shift($cursor);
$query->where($column, $direction === 'asc' ? '>' : '<', $value);
if (! empty($columns)) {
$query->orWhere($column, $value);
$apply($query, $columns, $cursor);
}
});
};
$apply($this, $columns, $cursor);
}
foreach ($columns as $column => $direction) {
$this->orderBy($column, $direction);
}
$items = $this->limit($limit + 1)->get();
if ($items->count() <= $limit) {
return new CursorPaginator($items);
}
$items->pop();
return new CursorPaginator($items, array_map(function ($column) use ($items) {
return $items->last()->{$column};
}, array_keys($columns)));
});
Example: Cursor pagination with a JSON API
Finally, let me show you an example how to use our cursor paginator on a Laravel controller that manages a list of posts. This example should be pretty self-explanatory now. Of course you can use URL parameters to control limit and sort columns, just pass them to the macro and append them to the next cursor URL, that's up to you. But don't forget: A truly unique column has always to be specified so things don't break!
class PostsController
{
public function index()
{
$paginator = Post::cursorPaginate(request('limit', 25), [
'created_at' => 'desc',
'id' => 'desc',
])->appends(request()->only('limit'));
return [
'data' => $paginator->items(),
'links' => [
'next' => $paginator->nextCursorUrl(),
],
];
}
}
How does the generated SQL look like?
select *
from "posts"
where ("created_at" < ? or "created_at" = ? and ("id" < ?))
order by "purchased_at" desc, "id" desc
limit 26
Conclusion
Huh, this post got a lot longer than I've expected. Congratulations if you did it until here! :)
I've shown a lot of code and concepts in this post, so I'm sure you have to copy it and play around with it to fully understand everything. And you should!
I'm sure my implementation is not perfect, but it is at least more flexible than most implementations out there. Especially, the ability to sort by multiple columns in multiple directions at the same time.
I hope you can use some ideas of this post in your next projects and I'd like to hear from you what you think about it. I'm @skollro on Twitter.
]]>
