What Are Global Secondary Indexes And How To Use Them

Welcome to the 9th edition of Excelling With DynamoDB!

In this issue, I’ll discuss what Global Secondary Indexes are, why and when we need them and how to use them in practice.

What Are GSIs

To understand what Global Secondary Indexes are, we must first understand how DynamoDB works as a key-value database.

A key-value store is a NoSQL purpose-built database that acts like a dictionary data structure. Like SQL, we have tables that store items, but instead of storing them in a row structure, items in a key-value store are stored as documents like in NoSQL.

The structure of an item in a key-value store is also fundamentally different than both SQL and traditional document based NoSQL databases.

In a key-value database like DynamoDB, items consist of only two elements: a key and a value, just like in a dictionary.

The key is how you identify the item and can make up of a partition key only or a primary key - which is the partition key and sort key. A partition key identifies the item and a sort key will determine how multiple items are sorted. Together they can make up a composite key.

The value is the data which the item consists of and can be made up of various data types like any primitive data type, arrays and objects as well.

The reason why key-value stores are designed this way is for performance and latency reasons. You can read my previous editions of the newsletter to learn more about the whys of a key-value store.

So what are GSIs?

In a DynamoDB table, you can only query for items by their primary keys (partition and/or sort key). But that primary key can never be changed so what can you do if you need to query table items by another primary key?

This is where GSIs come in.

Why GSIs

GSIs allow us to query our table items in DynamoDB by using a different primary key.

This allows us to satisfy additional data access patterns in our application.

For example, say we wanted to get all posts that a user has written in our app. We could store all posts under the user’s userID as the primary key. To get all posts a user has written, we could simply query the “posts” table with the user’s userID.

But say another feature requires us to fetch the user’s favorited posts. We can’t change our primary key to satisfy this new access pattern.

GSIs allow us to maintain a perfectly synced replica of our base table with all items in it but define a different primary key to access the data (as shown in the image above).

For each access pattern we have in our application, we can create a new GSI.

However we must be cautious. Each GSI incurs additional costs in DynamoDB and costs as much as an additional table.

This brings us to the next question, when to use GSIs.

When To Use GSIs

In an ideal world, we would use a GSI for every access pattern our application requires. But as mentioned earlier, each GSI comes with extra costs.

What I recommend is to always identify your data access patterns before designing your database. Your primary access pattern is satisfied with your base table’s primary keys. Then for the other use cases, identify a few of the most important to couple with GSIs.

For the rest you can make use of Index Overloading, this will help you minimize the use of GSIs, reducing costs and GSI management.

You can read more about Index Overloading here.

How To Use GSIs

Let’s take a concrete example of how we can create and use a GSI for our users table.

To add an access pattern that allows us to queries for favorite posts we can define the following GSI with some Node JS code:

const { DynamoDBClient, UpdateTableCommand } = require("@aws-sdk/client-dynamodb");

const REGION = "us-east-1"; 
const TABLE_NAME = "users"; 
const GSI_NAME = "user-favorite-posts"; 

const client = new DynamoDBClient({ region: REGION });

const params = {
  TableName: TABLE_NAME,
  AttributeDefinitions: [
    { AttributeName: "GSI_Attribute1", AttributeType: "S" }, // Replace with your GSI attributes
  ],
  GlobalSecondaryIndexUpdates: [
    {
      Create: {
        IndexName: GSI_NAME,
        KeySchema: [
          { AttributeName: "isFavorite", KeyType: "HASH" }, // Partition key
          { AttributeName: "postDate", KeyType: "RANGE" }, // Sort key (optional)
        ],
        Projection: {
          ProjectionType: "ALL", // Change to 'KEYS_ONLY' or 'INCLUDE' as needed
        },
        ProvisionedThroughput: {
          ReadCapacityUnits: 5, // Adjust as needed
          WriteCapacityUnits: 5, // Adjust as needed
        },
      },
    },
  ],
};

const createGSI = async () => {
  try {
    const data = await client.send(new UpdateTableCommand(params));
    console.log("GSI created successfully:", data);
  } catch (err) {
    console.error("Error creating GSI:", err);
  }
};

createGSI();        

In the code above, we define an update to our base table by adding a GSI with an index name of “user-favorite-posts” and a new partition key of “isFavorite”.

We can also define new throughput values for reads and writes to our GSI.

Finally, we run the update command to our base table “users” to update the base table with our GSI.

Conclusion

Global Secondary Indexes are essential in DynamoDB for enabling queries on non-primary key attributes, providing flexible and efficient data access patterns.

While GSIs allow for additional query capabilities and improve application performance, they also come with additional costs, often requiring careful planning of data access patterns and prudent use of GSIs.

By understanding when and how to use GSIs, you can greatly optimize your DynamoDB tables to meet various application requirements effectively.

If you enjoyed this post, please consider subscribing and sharing the newsletter with your network: https://www.dhirubhai.net/newsletters/7181386750357893121/

?? My name is Uriel Bitton and I hope you learned something in this edition of Excelling With DynamoDB.

?? You can share the article with your network to help others learn as well.

?? You can also explore my social media links here:

• Medium
• X
• Substack
• Threads
• Instagram
• Facebook

?? *my blog website is coming soon, so stay tuned for that.

?? I hope to see you in the next week's edition!

Uriel