Redis for high performance
Mohan Sagar Killamsetty
Senior Software Engineer at Invesco India Pvt. Ltd. | Full Stack Development/ DSML, Large Language Models (LLM)
As a developer, I constantly struggle to improve the performance of my application, make it load faster and give a butter smooth performance to the end user. Of all the techniques that are available to make an application faster usage of Redis is my favourite with its wide range of use cases that can be achieved successfully with various Redis capabilities.
I found Redis quite useful in my development to increase the performance of my application and hence decided to write a post so that others also can find the vast set of use cases that could be improved with Redis capabilities. Not just the applications of Redis but also the installation steps are pretty simple for Redis.
Redis can be installed on any server. In this case, we’ll install it locally for testing.
On macOS, installation is simpler using Homebrew. Run:
brew install redis
Then run
brew services start redis
to make Redis start automatically, and re-start when the computer reboots.
You can also start it manually using:
redis-server /usr/local/etc/redis.conf
On Linux Ubuntu, you’ll need to run
sudo apt-get install redis-server
and Redis will automatically be up and running.
Once it’s started, Redis listens on?port 6379.
On a local server, it’s fine to keep it running without a password. But when Redis is exposed to the Internet, make sure you configure a password in the redis.conf a configuration file, whose location depends on your Operating System.
?
Redis Installation on Ubuntu/Debian
You can install recent stable versions of Redis from the official?packages.redis.io?APT repository.
Prerequisites
If you're running a very minimal distribution (such as a Docker container) you may need to install?lsb-release?first:
?
sudo apt install lsb-release
Add the repository to the?apt?index, update it, and then install:
?
curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/redis.list
sudo apt-get update
sudo apt-get install redis
Install from Snapcraft
?
The?Snapcraft store?provides?Redis packages?that can be installed on platforms that support snap.
To install via snap, run:
?
sudo snap install redis
?
Install Redis from the Source
Compile and install Redis from the source
You can compile and install Redis from source on a variety of platforms and operating systems including Linux and macOS. Redis has no dependencies other than a C compiler and libc.
Downloading the source files
The Redis source files are available on [this site's Download page]. You can verify the integrity of these downloads by checking them against the digests in the?Redis-hashes git repository.
To obtain the source files for the latest stable version of Redis from the Redis downloads site, run:
wget https://download.redis.io/redis-stable.tar.gz
Compiling Redis
To compile Redis, first, the tarball, change to the root directory, and then run make:
?
tar -xzvf redis-stable.tar.gz
cd redis-stable
make
If the compile succeeds, you'll find several Redis binaries in the?src?directory, including:
To install these binaries in?/usr/local/bin, run:
make install
Starting and stopping Redis in the foreground
Once installed, you can start Redis by running
redis-server
If successful, you'll see the startup logs for Redis, and Redis will be running in the foreground.
To stop Redis, enter?Ctrl-C.
?
Install Redis on Windows
Use Redis on Windows for development
Redis is not officially supported on Windows. However, you can install Redis on Windows for development by following the instructions below.
To install Redis on Windows, you'll first need to enable?WSL2? (Windows Subsystem for Linux). WSL2 lets you run Linux binaries natively on Windows. For this method to work, you'll need to be running Windows 10 version 2004 and higher or Windows 11.
Install or enable WSL2
Microsoft provides?detailed instructions for installing WSL. Follow these instructions, and take note of the default Linux distribution it installs. This guide assumes Ubuntu.
Install Redis
Once you're running Ubuntu on Windows, you can follow the steps detailed at?Install on Ubuntu/Debian?to install recent stable versions of Redis from the official?packages.redis.io?APT repository. Add the repository to the?apt?index, update it, and then install:
curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/redis.list
sudo apt-get update
sudo apt-get install redis
Lastly, start the Redis server like so:
领英推荐
sudo service redis-server start
Connect to Redis
You can test that your Redis server is running by connecting with the Redis CLI:
redis-cli
127.0.0.1:6379> ping
PONG
?
RediSearch
?
RediSearch, a real-time secondary index with full-text search capabilities for Redis, is one of the most mature and feature-rich Redis modules. It is also becoming even more popular every day—in the past few months RediSearch Docker pulls have jumped 500%! That soaring popularity has led customers to come up with a wide variety of interesting use cases ranging from?real-time inventory management?to?ephemeral search.
To extend that momentum, we’re now introducing the public preview of RediSearch 2.0, designed to?improve the developer experience?and be?the most scalable version of Redisearch. RediSearch 2.0 supports Redis’?Active-Active geo-distribution?technology, is?scalable?without downtime, and includes?Redis on Flash?support (currently in private preview). To meet those goals without negatively impacting performance, we created a brand new architecture for RediSearch 2.0—and it worked:?RediSearch 2.0 is 2.4x faster?than RediSearch 1.6.?
Inside RediSearch 2.0’s new architecture?
Having a rich query-and-aggregation engine in your Redis database enables a wide variety of new use cases that extend well beyond caching. RediSearch lets you use Redis as your primary database in situations where you need to access data using complex queries. Even better, it preserves Redis’ world-class speed, reliability, and scalability, and doesn’t require you to add complexity to the code to let you update and index data.?
For RediSearch 2.0 we re-architected the way indices are kept in sync with the data. Instead of having to write data through the index (using the FT.ADD command), RediSearch now follows the data written in hashes and synchronously indexes it. This re-architecture comes with several changes in the API, which we discussed in a previous post when?RediSearch 2.0 Hit Its First Milestone.
This new architecture brings two main benefits. First, it’s now easier than ever to create a secondary index on top of your existing data. You can just?add RediSearch to your existing Redis database, create an index, and start querying it, without having to migrate your data or use new commands for adding data to the index. This drastically lowers the learning curve for new RediSearch users and lets you create indexes on your existing Redis databases—without even having to restart them.
In addition to implementing a new way to index data, we also took the index out of the keyspace. This enables Redis Enterprise’s?Active-Active technology, which is based on?conflict-free replicated data types (CRDTs).?Merging two inverted indices conflict-free is difficult, but Redis already has?a proven?CRDTs implementation?of Hashes. So the second big benefit of?this new architecture is making RediSearch 2.0 even more scalable. Because RediSearch now follows Hashes and the index was moved out of the keyspace, you can now run RediSearch in an Active-Active geo-distributed database.
A document will be replicated to all databases in the replication set in a?strongly eventual consistent manner. In each replica, RediSearch will simply follow all the updates on the Hashes, which means all indices are strongly eventual consistent as well.
OSS cluster support for open source Redis
We didn’t want to limit increasing the scalability capabilities to only Redis Enterprise users, so we added support for scaling a single index over multiple shards with the open-source Redis cluster API. Previously, a single RediSearch index, and its documents, had to reside on a single shard. This meant that dataset size and throughput for OSS Redis were bound to what a single Redis process could handle. Redis Enterprise offered the ability to distribute documents in a clustered database and aggregate the results at query time. This fan-out and aggregation are handled by a component called the “coordinator” that is now also publicly available?under the?Redis Source Available License?so it will work with open source Redis clusters as well as Redis Enterprise. The result is the most scalable version of RediSearch yet.?
Show me the numbers!
To assess RediSearch 2.0’s ingestion performance, we extended our full-text search benchmark (FTSB) suite with?the publicly available NYC Taxi dataset. This dataset is used across the industry due to its rich set of data types (text, tag, geographic, and numeric), and a large number of documents.?
This benchmark focuses on write performance, using trip-record data of rides in yellow cabs in New York City. Specifically for this benchmark, we used the January 2015 dataset, which loads more than 12 million documents with an average size of 500 bytes per document. For the full benchmark specification please refer to the FTSB on GitHub.
All benchmark variations were run on Amazon Web Services instances, provisioned through our benchmark-testing infrastructure. The tests were executed on a 3-node cluster with 15 shards, with RediSearch Enterprise versions 1.6 and 2.0. Both the benchmarking client and the 3 nodes comprising the database with RediSearch enabled were running on separate c5.9xlarge instances.
Given that RediSearch 2.0 comes with the ability to follow changes in Hashes in Redis and automatically index them, we’ve added variants for the?FT.ADD?and?HSET commands. To make upgrades easier, we remapped the now deprecated FT.ADD command to the HSET commands in RediSearch 2.0. The two charts below display the overall ingestion rate and latency for both RediSearch 1.6 and RediSearch 2.0 while retaining sub-millisecond latencies.
RediSearch has always been fast, but with this architectural change, we’ve moved from indexing 96K documents per second to 132K docs/sec at an overall p50 ingestion latency of 0.4ms, drastically improving write scaling.?
Not only will you benefit from the boost in the throughput, but each ingestion also becomes faster. Apart from the overall ingestion improvement due to the changes in architecture, you can now also rely on the OSS Redis Cluster API capabilities to linearly scale the ingestion of your search database.?
Combining throughput and latency improvements,?RediSearch 2.0 delivers up to a 2.4X speedup?compared to the RediSearch 1.6.
What’s next for RediSearch 2.0
To sum up,?RediSearch 2.0 is the fastest and most scalable version for all Redis users that we have ever released.?In addition, RediSearch 2.0’s new architecture improves the developer experience of creating indices for existing data within Redis seamlessly and removes the need to migrate your Redis data to another RediSearch-enabled database. This new architecture allows RediSearch to follow and auto-index other data structures, such as Streams or Strings. In upcoming releases, it will let you work with additional data structures such as the nested data structure in RedisJSON.?
We plan to keep on adding more features to further enhance the developer experience. Coming next, look for a new command that allows you to profile your search queries to better understand where performance bottlenecks occur during query execution.
Ready to get started? Check out Tug Grall’s blog on …?Getting Started with RediSearch 2.0! Then follow the steps in this?tutorial on GitHub?or create a free database in?Redis Enterprise Cloud Essentials. (Note that the public preview of RediSearch 2.0 is available in two Redis Enterprise Cloud Essentials regions: Mumbai and Oregon.)
?Redis JSON
?JavaScript Object Notation, or JSON, is a super fast and lightweight data exchange format widely adopted. It is human-readable while still providing flexibility for both machines and users.
Learning to work with JSON is a requirement when building any non-trivial application. It is there helpful to know how to store JSON in Redis.
Requirements:
Method 1 – Pre-Serialized Data
The simplest method to store JSON is to take pre-serialized data and store it as a native Redis key.
For example:
127.0.0.1:6379>?SET user_info?'{"id":3,"first_name":"Valida","last_name":"Lindop","email":"[email protected]","ip_address":"140.207.199.111"}'
OK
Here, we use a Redis string to hold JSON data. However, it works for simple use cases; a few issues arise from using this method in extensive use cases.
There is no direct way to manipulate the values. Therefore, if you need to update any value, you will have to set the key from scratch.
During read/write, serialization uses more resources.
Since Redis does not compress the data you add to it, JSON will add a heavy memory overhead.
Third, the output from JSON as a serialized object is not very readable.
For example:
127.0.0.1:6379>?GET user_info
"{"id":3,"first_name":"Valida","last_name":"Lindop","email":"[email protected]","ip_address":"140.207.199.111"}"
Method 2 – RedisJSON Module
The second and most viable method to use JSON in your Redis database is the RedisJSON module.
We will not cover the installation in this article. Instead, you can check the resource provided below:
To add a new JSON document using the JSON module, run the command:
127.0.0.1:6379> JSON.SET user_info.?'{"id":1}'
OK
The above should add a new key holding a JSON document.
We can fetch the JSON using GET as:
127.0.0.1:6379>?JSON.GET user_info
"{"id":1}
To get the type of a JSON entity, we can run:
127.0.0.1:6379>?JSON.TYPE user_info .id
"integer"
True, id is holding an integer type.
To delete a JSON document, run:
127.0.0.1:6379>?JSON.DEL user_info
(integer)?1
??
This post is in collaboration with Redis.
Learn more: