Redis 內存相關

Can I backup a Redis DB while the server is working?

RDB and AOF files are compatible between 32 bit and 64 bit instances (and between little and big endian of course,文件格式Redis會自己統一) so you can switch from 32 to 64 bit, or the contrary, without problems.

When Redis saves the DB it actually creates a temp file, then rename(2) that temp file name to the destination file name. So even while the server is working it is safe to save the database file just with the cp UNIX command.

Background saving is failing with a fork() error under Linux even if I've a lot of free RAM!

Short answer: echo 1 > /proc/sys/vm/overcommit_memory :)

And now the long one:

Redis background saving schema relies on the copy-on-write semantic of fork in modern operating systems: Redis forks (creates a child process) that is an exact copy of the parent. The child process dumps the DB on disk and finally exits. In theory the child should use as much memory as the parent being a copy, but actually thanks to the copy-on-write semantic implemented by most modern operating systems the parent and child process will share the common memory pages. A page will be duplicated only when it changes in the child or in the parent. Since in theory all the pages may change while the child process is saving, Linux can't tell in advance how much memory the child will take, so if the overcommit_memory setting is set to zero fork will fail unless there is as much free RAM as required to really duplicate all the parent memory pages, with the result that if you have a Redis dataset of 3 GB and just 2 GB of free memory it will fail.

Setting overcommit_memory to 1 says Linux to relax and perform the fork in a more optimistic allocation fashion, and this is indeed what you want for Redis.

A good source to understand how Linux Virtual Memory work and other alternatives for overcommit_memory and overcommit_ratio is this classic from Red Hat Magazine, "Understanding Virtual Memory".

What is the maximum number of keys a single Redis instance can hold? and what the max number of elements in a List, Set, Ordered Set?

In theory Redis can handle up to 2³² keys, and was tested in practice to handle at least 150 million of keys per instance. 

Every list, set, and ordered set, can hold 2³² elements.

Actually Redis internals are ready to allow up to 2⁶⁴ elements but the current disk dump format don't support this, and there is a lot time to fix this issues in the future as currently even with 128 GB of RAM it's impossible to reach 2³² elements.???可能是一個物理文件存不了那麼多,這是什麼意思

How much time it takes to load a big database at server startup?

Just an example on normal hardware: It takes about 45 seconds to restore a 2 GB database on a fairly standard system, no RAID. 

What's the Redis memory footprint(佔用量)?

Worst case scenario: 1 Million keys with the key being the natural numbers from 0 to 999999 and the string "Hello World" as value use 100MB on my Intel MacBook (32bit). Note that the same data stored linearlyin an unique string takes something like 16MB,

With large keys/values the ratio is much better of course.

64 bit systems will use much more memory than 32 bit systems to store the same keys, especially if the keys and values are small,this is because pointers takes 8 bytes in 64 bit systems. But of course the advantage is that you can have a lot of memory in 64 bit systems, so to run large Redis servers a 64 bit system is more or less required. (64位系統允許更大的虛擬內存)

dataset 超過RAM怎麼辦？

使用一致性Hash 進行分佈式存儲。

Redis官方不建議單個Server上,dataset大於RAM,會引出很多問題,將來很可能不支持大於RAM的dataset

In the future of Redis we want to simply provide the best in-memory database (but persistent on disk as usually) ever, without considering at least for now the support for databases bigger than RAM. Our future efforts are focused into providing scripting, cluster, and better persistence.

Why Redis takes the whole dataset in RAM? 

If the data is larger then memory, and this data is stored on disk, 此時會受到磁盤I/O的瓶頸影響。32位有4GB虛擬內存可用(OS來完成交換)

With modern operating systems malloc() returning NULL is not common, usually the server will start swapping and Redis performances will be disastrous so you'll know it's time to use more Redis servers or get more RAM.

理論上來說未來64位的內存 將沒有RAM空間限制。

The INFO command (work in progress in this days) will report the amount of memory Redis is using so you can write scripts that monitor your Redis servers checking for critical conditions.

You can also use the "maxmemory" option in the config file to put a limit to the memory Redis can use. If this limit is reached Redis will start to reply with an error to write commands (but will continue to accept read-only commands).

I have an empty Redis server but INFO and logs are reporting megabytes of memory in use!

This may happen and it's perfectly okay. Redis objects are small C structures allocated and freed a lot of times. This costs a lot of CPU so instead of being freed, released objects are taken into a free list and reused when needed. This memory is taken exactly by this free objects ready to be reused.

Is there something I can do to lower the Redis memory usage?

Yes, try to compile it with 32 bit target if you are using a 64 bit box.

使用Hash type 會節約空間

If you are using hashes or any other type with values bigger than 128 bytes try also this to lower the RSS usage (Resident Set Size): EXPORT MMAP_THRESHOLD=4096

???

Does Redis use more memory running in 64 bit boxes? Can I use 32 bit Redis in 64 bit systems?

Redis uses a lot more memory when compiled for 64 bit target, especially if the dataset is composed of many small keys and values. Such a database will, for instance, consume 50 MB of RAM when compiled for the 32 bit target, and 80 MB for 64 bit! That's a big difference.

You can run 32 bit Redis binaries in a 64 bit Linux and Mac OS X system without problems. (只是兼容性可以運行，但是實際上最大還是4GB內存，因爲編譯後，程序的尋址範圍還是4GB）For OS X just use make 32bit. For Linux instead, make sure you have libc6-dev-i386 installed, then use make 32bit if you are using the latest Git version. 

If your application is already able to perform application-level sharding, it is very advisable to run N instances of Redis 32bit against a big 64 bit Redis box (with more than 4GB of RAM) instead than a single 64 bit instance,as this is much more memory efficient.

Background saving is failing with a fork() error under Linux even if I've a lot of free RAM!

Short answer: echo 1 > /proc/sys/vm/overcommit_memory :)

And now the long one:

Redis background saving schema relies on the copy-on-write semantic of fork in modern operating systems: Redis forks (creates a child process) that is an exact copy of the parent. The child process dumps the DB on disk and finally exits. In theory the child should use as much memory as the parent being a copy, but actually thanks to the copy-on-write semantic implemented by most modern operating systems the parent and child process will share the common memory pages. A page will be duplicated only when it changes in the child or in the parent. Since in theory all the pages may change while the child process is saving, Linux can't tell in advance how much memory the child will take, so if the overcommit_memory setting is set to zero fork will fail unless there is as much free RAM as required to really duplicate all the parent memory pages, with the result that if you have a Redis dataset of 3 GB and just 2 GB of free memory it will fail.

Setting overcommit_memory to 1 says Linux to relax and perform the fork in a more optimistic allocation fashion, and this is indeed what you want for Redis.

A good source to understand how Linux Virtual Memory work and other alternatives for overcommit_memory and overcommit_ratio is this classic from Red Hat Magazine, "Understanding Virtual Memory".

Are Redis on disk snapshots atomic????

Yes, redis background saving process is always fork(2)ed when the server is outside of the execution of a command, so every command reported to be atomic in RAM is also atomic from the point of view of the disk snapshot.

Redis is single threaded, how can I exploit multiple CPU / cores?

Simply start multiple instances of Redis in different ports in the same box and treat them as different servers!Given that Redis is a distributed database anyway in order to scale you need to think in terms of multiple computational units. At some point a single box may not be enough anyway.

In general key-value databases are very scalable because of the property that different keys can stay on different servers independently.

In Redis there are client libraries such Redis-rb (the Ruby client) that are able to handle multiple servers automatically using consistent hashing. We are going to implement consistent hashing in all the other major client libraries. If you use a different language you can implement it yourself otherwise just hash the key before to SET / GET it from a given server. For example imagine to have N Redis servers, server-0, server-1, ..., server-N. You want to store the key "foo", what's the right server where to put "foo" in order to distribute keys evenly among different servers? Just perform the crc = CRC32("foo"), then servernum = crc % N (the rest of the division for N). This will give a number between 0 and N-1 for every key. Connect to this server and store the key. The same for gets.

This is a basic way of performing key partitioning, consistent hashing is much better and this is why after Redis 1.0 will be released we'll try to implement this in every widely used client library starting from Python and PHP (Ruby already implements this support).

I'm using some form of key hashing for partitioning, but what about SORT BY?

With [SortCommand SORT] BY you need that all the weight keys are in the same Redis instance of the list/set you are trying to sort. In order to make this possible we developed a concept called key tags. A key tag is a special pattern inside a key that, if preset, is the only part of the key hashed in order to select the server for this key. For example in order to hash the key "foo" I simply perform the CRC32 checksum of the whole string, but if this key has a pattern in the form of the characters {...} I only hash this substring. So for example for the key "foo{bared}" the key hashing code will simply perform the CRC32 of "bared". This way using key tags you can ensure that related keys will be stored on the same Redis instance just using the same key tag for all this keys. Redis-rb already implements key tags.

減少空間使用的方法

bit and byte level operations

Redis 2.2 introduced new bit and byte level operations: GETRANGE, SETRANGE(Byte級), GETBIT and SETBIT(Bit級,可用於位圖). 節省空間

Special encoding of small aggregate data types (對數據進行壓縮,對用戶透明)

hash-max-zipmap-entries 512   //超過一定數量時，可能對hash進行壓縮
hash-max-zipmap-value 64   // 最大元素不超過512時，纔對hash進行壓縮,   512是數據的二進制表示，string表示的數據不一定得是512
list-max-ziplist-entries 512  
list-max-ziplist-value 64
set-max-intset-entries 512  //Set 只要數據大小超過512時,且String的表示的整數不超過64位integer,就可壓縮

This operation is very fast for small values, but if you change the setting in order to use specially encoded values for much larger aggregate types the suggestion is to run some benchmark and test to check the conversion time.(改變時需要做測試,檢查時間空間耗費比)

Use hashes when possible (小的hash很剩空間)

Small hashes are encoded in a very small space, so you should try representing your data using hashes every time it is possible.

小hash被處理爲小數組(有CPU的cache優化,並且剩空間)，大hash被處理爲hashtable,

(you can configure the limit in redis.conf).

hash中的key-value,在Redis中被定義爲field,value

,以和key-value相區分，因爲hash不允許嵌套,field只能爲string，沒有結構。 simplicity more than features

Using hashes to abstract a very memory efficient plain key-value store on top of Redis

Let's start with some fact: a few keys use a lot more memory than a single key containing an hash with a few fields. 

Now let's assume the objects we want to cache are numbered, like:

• object:102393
• object:1234
• object:5

 For instance the object named *object:1234" is actually split into:

• a Key named object:12
• a Field named 34

HSET object:12 34 somevalue

As you can see every hash will end containing 100 fields, that is an optimal compromise between CPU and memory saved.(剩空間，必然有CPU耗費)

 object:2 and object:10 will both end inside the key "object:", but one as field name "2" and one as "10".

How much memory we save this way (10times)