Paginating query and scan results in DynamoDB

DynamoDB is a fully managed NoSQL document database provided by AWS

2020-12-01

Metadata
title
Paginating query and scan results in DynamoDB
description
DynamoDB is a fully managed NoSQL document database provided by AWS
date
2020-12-01
image
./paginate.jpg
categories
awsdynamodbjavascriptdatabases

Most of the methods in DynamoDB take an object as their first parameter, which provides the necessary information to perform the requested operation. These options include things like the table name to operate against, the record you want to write to the database, or the index name that you want to query against. I often have to look up the names of the key names used for each of the parameters, and the one scenario I have to look-up more times than not is how to paginate the query and scan methods.

query and scan

Both the query and scan operations allow you to specify a limit on the number of results to return in a single operation. If the total number of results exceeds this limit or the response exceeds 1MB, the operation will return a cursor. The cursor will be an object, which is the last key the operation returned. When this cursor is given back to DynamoDB, it will pick up the next results where it left off last time. While you can page through all the results at once, the more likely scenario is that you would return the cursor to your front-end application to use in your next request to fetch the next page of results.

For more information about how to use query and scan, please refer to the official api docs.
query
scan

LastEvaluatedKey

When you use query or scan, included in the returned object is a property called LastEvaluatedKey.

const {
  Items,
  LastEvaluatedKey, // this is the cursor
} = await dynamoClient.query(params).promise();

ExclusiveStartKey

As mentioned above, LastEvaluatedKey will be an object, which is serializable, so it can be turned into JSON and sent to a client application. To use the cursor to get the next page of results you pass it to your query or scan via the property ExclusiveStartKey in the params object.

const { Items } = await dynamoClient.query({
  ...otherParams,
  ExclusiveStartKey: cursor,
}).promise();

Fetch all at once

A pattern I have used many times to page through all the results in one go is to use a do { ... } while() loop.

const params = {
  TableName: ‘my-table’,
};

const results = [];
let lastKey = null;

do {
  // copy the last key into the params
  // for the first time through the loop it will be null to tell
  // the operation to start at the beginning
  params.ExclusiveStartKey = lastKey;
  const response = await dynamoClient.scan(params).promise();
  // capture the results
  results.push(...response.Items);
  // capture the LastEvaluatedKey
  lastKey = response.LastEvaluatedKey;
} while (lastKey != null);

As always, just remember that fetching all results for the scan or query can consume a lot of throughput, so you will want to plan accordingly.

Paginating batchGetItem and batchWriteItem in DynamoDB

This post is one part of 2 blog posts all about paginating results in DynamoDB. To see how to paginate results for the batchGetItem and batchWriteItem operations, click here

Follow me