Redis is a handy place to keep data. With all the commands Redis supports, you can solve a ton of really common problems.

For example, do you need a queue you can safely use from a bunch of different processes? LPUSH and BRPOP have you covered. That's actually how Sidekiq works! (Resque pushes and pops in the other direction).

In fact, Redis is such an easy place to stuff data, that it could become your first choice for storing miscellaneous things. You'll find the command that does what you want, call it, and everything will be great! That is, until you store more and more data under certain keys.

Maybe you want to see if a job has already been queued, so you don't queue it again:

index = 0
while (payload = Resque.redis.lindex("queue:#{queue_name}", index)) do
  # ... see if payload matches the job we're looking for ... 

And all of a sudden, you'll notice that all of your communication with Redis just got slower. You might even start seeing Redis::TimeoutErrors. What happened?

In order to understand how this code broke, you need to understand a little bit about Redis, and a little fundamental Computer Science.


Take another look at Redis' command documentation, and you'll notice something on each page:


Yep, it's Big-O notation -- that thing you studied to prepare for your last interview. In your day-to-day development, you probably don't think about it too much. In Redis, though, slower algorithms can destroy your app's overall performance.

Most Redis commands are pretty fast: O(1) or O(log(N)). But a few, like that LINDEX from up above, are O(N). (Some, like ZUNIONSTORE, are even worse. Don't ask me how I know that).

That means that if you add twice the elements to your queue, that call will probably run (roughly) twice as slow.

And because Redis is single-threaded, a slow Redis command can keep other commands from running. When that happens, you'll start to see your error tracker fill up with Redis errors from totally unrelated places.

How do you detect and fix it?

Redis has SLOWLOG, which can show you which queries are taking the longest:

~ jweiss$ redis-cli> slowlog get 3
1) 1) (integer) 26
   2) (integer) 1436856612
   3) (integer) 13286
   4) 1) "get"
      2) "key2"
2) 1) (integer) 25
   2) (integer) 1436856610
   3) (integer) 41114
   4) 1) "get"
      2) "key2"
3) 1) (integer) 24
   2) (integer) 1436856609
   3) (integer) 10891
   4) 1) "get"
      2) "key2"

That's a lot of numbers. The first one is automatically generated -- you don't need to worry about it. The second is a timestamp. The third and fourth are the most important -- the amount of time (in microseconds) it took for the command to run, and the command + arguments you sent.

The slowlog can be noisy, and won't always point you to exactly the right place. But it's a good first place to look for performance problems.

After you've identified some slow queries, though, how do you fix them?

Unfortunately, there's no approach that works in every situation. But here are a few that have helped me:

  • Scan through the other Redis commands that work with the data structure you're using. If any of them are an almost as good a fit, but faster, you can try to find a way to use those instead.

  • Store an extra copy of the data in a way that makes it fast to look up later. This works especially well for commands like LINDEX. It's like having data in an array to make it easy to iterate, with a lookup table in a hash to make specific elements easy to find.

    While this can work well, you do have to be careful. It's easy to create weird situations where the duplicate copy wasn't added right, or removed right, or got into an inconsistent state. MULTI and EXEC can help, but still take some care to use correctly.

  • Don't use Redis for that data at all. A traditional database might be a better solution, or there might be a more clever way of solving the problem.

In your day-to-day work, you probably don't think too much about algorithmic complexity and O() notation.

But Redis is a single-threaded server where speed is incredibly important. The change in performance as your data grows makes a huge difference. And if you're not paying enough attention, it can hurt your entire app.

So, watch for it. Understand how to compare different algorithms, so you can pick the right one. And build intuition about how fast good algorithms should be, and the tradeoffs between them. That intuition will help you more often than you'd expect.