Vapor RestKit 2.1 - Backward Cursor Pagination

I've released a small update for Vapor RestKit, recently.
This version includes backward cursor pagination.

What the Heck Is Pagination

Imagine you've got a client-server application, let's say it's a messenger app. Then imagine you have a large collection of any kind of items. Let's say, messages are stored on a server.

You cannot just request all of them at once as it will create a load on the server instance and db, would hit the network bandwidth, and make your mobile device feel sad to proceed all at once.

To avoid this performance hit, wise people invented pagination - a way to fetch the data in portions or pages, as the naming says.

The most straightforward way to do it is to define the pagesize and query the desired page by offset. It's easy to implement, and it's already done in Vapor:

app.get("planets") { req in 
	try await Planet.query(on: req.db).paginate(for: req) 
}

Then you can make a request:

GET /planets?page=2&per=5

And get something like this in response:

{
	"items": [...], 
	"metadata": { 
		"page": 2,
		"per": 5, 
		"total": 8
	 } 

}

There are two downsides of offset-based pagination:

• Extra count query
• It's offset-based

If you want to return a total number of pages, you inevitably have to make an extra count query on db, every time you query a page, which is not perfect.

Offset-based pagination works crappy for frequently updated collections of data.
It happens because items that get into the page are defined by the offset from the beginning of the data collection, while offset is defined by the page number * per page limit.

When new items are inserted on top of the collection, again and again, items in the pages get shifted every time and eventually result in the client getting the wrong portion of data.

That means that offset-based pagination is not suitable for apps with actively updated collections of content like messengers, social network apps with posts and comments, etc.

Here Is Where the Cursor Pagination Kicks In

With cursor pagination server returns a cursor that points to the next portion of data, allowing clients to reliably get the next portion every time.

Magic!

The downside of cursor pagination is that the implementation becomes tricky if the collection is dynamically sorted by the client's request.

Folks from Slack have described the whole thing here in detail.

What's the Backward Cursor Pagination

Backward cursor pagination introduced in Vapor 2.1 makes the thing even more powerful.

It allows one to query not only the next portion of data but also request the previous one.

Its main use case is frequently updated collections with elements inserted on top. Backward pagination allows one to go back to the beginning of the collection and fetch newly added items.

Cursor Pagination in Vapor RestKit

Not a big deal.

Allow backward pagination in the config:

let config = CursorPaginationConfig(
	limitMax: 25, 
	defaultLimit: 10, 
	allowBackwardPagination: true
)

struct StarsController {
      
    func index(req: Request) throws -> EventLoopFuture<CursorPage<Star.Output>> {
        try ResourceController<Star.Output>().getCursorPage(
            req: req,
            config: config
        )
    }
}

This will make server return both previous and next cursor:

{
   "items": [
      //your data collection here
   ],
   
   "metadata": {
     "next_cursor": "W3siZmllbGRLZXkiOiJhc3NldHNfW3RpY2tlcl0iLCJ2",
     "prev_cursor": "dW3siZmllbGRLZXkiOiJhc3NldHNf2312RpY2tlcl0i3"
      
   }
}

And will allow to query not only the next portion:

https://api.yourservice.com/v1/stars?after=W3siZmllbGRLZXkiOiJhc3NldHNfW3RpY2tlcl0iLCJ2YWx1Z

But also a previous one:

https://api.yourservice.com/v1/stars?before=W3siZmllbGRLZXkiOiJhc3NldHNfW3RpY2t1234iLCJ2YWx1Z

More details can be found in Vapor RestKit