Redis 持久化

Redis 持久化

1.Redis RDB

---

1.1 RDB

• 英譯：Redis DataBase
• 含義：時間段內==>快照SnapShot==>寫入到磁盤
• 作用：rdb 保存的是dump.rdb 文件

 

1.2 Fork

• 作用：fork的作用是複製一個與當前進程一樣的進程。
  • 新進程的所有數據（變量、環境變量、程序計數器等）數值都和原進程一致，但是是一個全新的進程，並作爲原進程的子進程.

1.3 RDB 配置文件

################################ SNAPSHOTTING  ################################

#

# Save the DB on disk:

#

#   save <seconds> <changes>

#

#   Will save the DB if both the given number of seconds and the given

#   number of write operations against the DB occurred.

#

#   In the example below the behaviour will be to save:

#   after 900 sec (15 min) if at least 1 key changed

#   after 300 sec (5 min) if at least 10 keys changed

#   after 60 sec if at least 10000 keys changed

#

#   Note: you can disable saving completely by commenting out all "save" lines.

#

#   It is also possible to remove all the previously configured save

#   points by adding a save directive with a single empty string argument

#   like in the following example:

#

#   save ""

save 900 1

save 300 10

save 60 10000

# By default Redis will stop accepting writes if RDB snapshots are enabled

# (at least one save point) and the latest background save failed.

# This will make the user aware (in a hard way) that data is not persisting

# on disk properly, otherwise chances are that no one will notice and some

# disaster will happen.

#

# If the background saving process will start working again Redis will

# automatically allow writes again.

#

# However if you have setup your proper monitoring of the Redis server

# and persistence, you may want to disable this feature so that Redis will

# continue to work as usual even if there are problems with disk,

# permissions, and so forth.

stop-writes-on-bgsave-error yes

# Compress string objects using LZF when dump .rdb databases?

# For default that's set to 'yes' as it's almost always a win.

# If you want to save some CPU in the saving child set it to 'no' but

# the dataset will likely be bigger if you have compressible values or keys.

rdbcompression yes

# Since version 5 of RDB a CRC64 checksum is placed at the end of the file.

# This makes the format more resistant to corruption but there is a performance

# hit to pay (around 10%) when saving and loading RDB files, so you can disable it

# for maximum performances.

#

# RDB files created with checksum disabled have a checksum of zero that will

# tell the loading code to skip the check.

rdbchecksum yes

# The filename where to dump the DB

dbfilename dump.rdb

# The working directory.

#

# The DB will be written inside this directory, with the filename specified

# above using the 'dbfilename' configuration directive.

#

# The Append Only File will also be created inside this directory.

#

# Note that you must specify a directory here, not a file name.

dir ./

---

1.4 RDB 如何觸發

• 在配置文件中修改

# Save the DB on disk:

#

#   save <seconds> <changes>

#

#   Will save the DB if both the given number of seconds and the given

#   number of write operations against the DB occurred.

#

#   In the example below the behaviour will be to save:

#   after 900 sec (15 min) if at least 1 key changed

#   after 300 sec (5 min) if at least 10 keys changed

#   after 60 sec if at least 10000 keys changed

#

#   Note: you can disable saving completely by commenting out all "save" lines.

#

#   It is also possible to remove all the previously configured save

#   points by adding a save directive with a single empty string argument

#   like in the following example:

#

#   save ""

save 900 1

save 300 10

save 60 10000

• 命令 SAVE 或 BGSAVE

1. SAVE : 只管保存，其他不管，全部阻塞。
2. BGSAVE: Redis 會在後臺異步進行快照操作，快照同時可以響應客戶端請求 。

• 可以通過lastsave 命令，獲取最後一次成功執行快照的時間。 

1.5 RDB 如何恢復 

1.  將備份文件（dump.rdb） ==> 移動到 redis 安裝目錄 ==>並啓動服務
2.  config get dir獲取目錄

1.6 RDB 優勢

• 適合大規模的數據恢復
• 對數據完整性和一致性要求不高 

1.7 RDB 缺點

• 在一定時間間隔做一次備份，所以如果redis 意外shutdown 的話，就會丟失最後一次快照後的所有修改
• fork的時候，內存中的數據被克隆了一份，大致2倍膨脹性需要考慮

1.8 RDB 如何停止

• 停止rdb保存規則：

redis-cli config set save ""

2.Redis AOF 

---

2.1 AOF

• 英譯：Append Only File (只允許追加文件，不允許修改文件)
• 含義：以日誌的形式記錄每個寫的操作。

2.2 AOF 配置文件    

############################## APPEND ONLY MODE ###############################

# By default Redis asynchronously dumps the dataset on disk. This mode is

# good enough in many applications, but an issue with the Redis process or

# a power outage may result into a few minutes of writes lost (depending on

# the configured save points).

#

# The Append Only File is an alternative persistence mode that provides

# much better durability. For instance using the default data fsync policy

# (see later in the config file) Redis can lose just one second of writes in a

# dramatic event like a server power outage, or a single write if something

# wrong with the Redis process itself happens, but the operating system is

# still running correctly.

#

# AOF and RDB persistence can be enabled at the same time without problems.

# If the AOF is enabled on startup Redis will load the AOF, that is the file

# with the better durability guarantees.

#

# Please check http://redis.io/topics/persistence for more information.

appendonly no

# The name of the append only file (default: "appendonly.aof")

appendfilename "appendonly.aof"

# The fsync() call tells the Operating System to actually write data on disk

# instead of waiting for more data in the output buffer. Some OS will really flush

# data on disk, some other OS will just try to do it ASAP.

#

# Redis supports three different modes:

#

# no: don't fsync, just let the OS flush the data when it wants. Faster.

# always: fsync after every write to the append only log. Slow, Safest.

# everysec: fsync only one time every second. Compromise.

#

# The default is "everysec", as that's usually the right compromise between

# speed and data safety. It's up to you to understand if you can relax this to

# "no" that will let the operating system flush the output buffer when

# it wants, for better performances (but if you can live with the idea of

# some data loss consider the default persistence mode that's snapshotting),

# or on the contrary, use "always" that's very slow but a bit safer than

# everysec.

#

# More details please check the following article:

# http://antirez.com/post/redis-persistence-demystified.html

#

# If unsure, use "everysec".

# appendfsync always

appendfsync everysec

# appendfsync no

# When the AOF fsync policy is set to always or everysec, and a background

# saving process (a background save or AOF log background rewriting) is

# performing a lot of I/O against the disk, in some Linux configurations

# Redis may block too long on the fsync() call. Note that there is no fix for

# this currently, as even performing fsync in a different thread will block

# our synchronous write(2) call.

#

# In order to mitigate this problem it's possible to use the following option

# that will prevent fsync() from being called in the main process while a

# BGSAVE or BGREWRITEAOF is in progress.

#

# This means that while another child is saving, the durability of Redis is

# the same as "appendfsync none". In practical terms, this means that it is

# possible to lose up to 30 seconds of log in the worst scenario (with the

# default Linux settings).

#

# If you have latency problems turn this to "yes". Otherwise leave it as

# "no" that is the safest pick from the point of view of durability.

no-appendfsync-on-rewrite no

# Automatic rewrite of the append only file.

# Redis is able to automatically rewrite the log file implicitly calling

# BGREWRITEAOF when the AOF log size grows by the specified percentage.

#

# This is how it works: Redis remembers the size of the AOF file after the

# latest rewrite (if no rewrite has happened since the restart, the size of

# the AOF at startup is used).

#

# This base size is compared to the current size. If the current size is

# bigger than the specified percentage, the rewrite is triggered. Also

# you need to specify a minimal size for the AOF file to be rewritten, this

# is useful to avoid rewriting the AOF file even if the percentage increase

# is reached but it is still pretty small.

#

# Specify a percentage of zero in order to disable the automatic AOF

# rewrite feature.

auto-aof-rewrite-percentage 100

auto-aof-rewrite-min-size 64mb

# An AOF file may be found to be truncated at the end during the Redis

# startup process, when the AOF data gets loaded back into memory.

# This may happen when the system where Redis is running

# crashes, especially when an ext4 filesystem is mounted without the

# data=ordered option (however this can't happen when Redis itself

# crashes or aborts but the operating system still works correctly).

#

# Redis can either exit with an error when this happens, or load as much

# data as possible (the default now) and start if the AOF file is found

# to be truncated at the end. The following option controls this behavior.

#

# If aof-load-truncated is set to yes, a truncated AOF file is loaded and

# the Redis server starts emitting a log to inform the user of the event.

# Otherwise if the option is set to no, the server aborts with an error

# and refuses to start. When the option is set to no, the user requires

# to fix the AOF file using the "redis-check-aof" utility before to restart

# the server.

#

# Note that if the AOF file will be found to be corrupted in the middle

# the server will still exit with an error. This option only applies when

# Redis will try to read more data from the AOF file but not enough bytes

# will be found.

aof-load-truncated yes

# When rewriting the AOF file, Redis is able to use an RDB preamble in the

# AOF file for faster rewrites and recoveries. When this option is turned

# on the rewritten AOF file is composed of two different stanzas:

#

#   [RDB file][AOF tail]

#

# When loading Redis recognizes that the AOF file starts with the "REDIS"

# string and loads the prefixed RDB file, and continues loading the AOF

# tail.

#

# This is currently turned off by default in order to avoid the surprise

# of a format change, but will at some point be used as the default.

aof-use-rdb-preamble no

2.3 AOF 啓用 

• 在redis.config 修改

appendonly no ==>> appendonly yes  

• 關機==>退出 ==> 重啓

shutdown nosave &&  exit  &&  redis-Server  /media/lenovo/軟件盤/redis/redis.conf

• 生成 AOF 文件名爲：appendonly.aof

2.4 AOF 文件損壞-修復

• 在/usr/local/bin/redis-check-aof  命令 ，修復損壞文件appendonly.aof 
  

redis-check-aof --fix appendonly.aof

• 恢復後，然後重啓redis 重新加載

redis-server /media/lenovo/軟件盤/redis/redis.conf

2.5 AOF--rewirte

• 含義：
  • AOF採用文件追加方式，文件會越來越大爲避免出現此種情況，新增了重寫機制。
  • 當AOF文件的大小超過所設定的閾值時，Redis就會啓動AOF文件的內容壓縮,,只保留可以恢復數據的最小指令集.

• 原理：
  • AOF文件持續增長而過大時，會fork出一條新進程來將文件重寫，
    • 遍歷新進程的內存中數據，每條記錄有一條的Set語句。
  • 重寫aof文件的操作，並沒有讀取舊的aof文件，而是將整個內存中的數據庫內容用命令的方式重寫了一個新的aof文件，這點和快照有點類似

2.6 AOF--rewirte 觸發機制

# Automatic rewrite of the append only file.

# Redis is able to automatically rewrite the log file implicitly calling

# BGREWRITEAOF when the AOF log size grows by the specified percentage.

#

# This is how it works: Redis remembers the size of the AOF file after the

# latest rewrite (if no rewrite has happened since the restart, the size of

# the AOF at startup is used).

#

# This base size is compared to the current size. If the current size is

# bigger than the specified percentage, the rewrite is triggered. Also

# you need to specify a minimal size for the AOF file to be rewritten, this

# is useful to avoid rewriting the AOF file even if the percentage increase

# is reached but it is still pretty small.

#

# Specify a percentage of zero in order to disable the automatic AOF

# rewrite feature.

auto-aof-rewrite-percentage 100   達到100

auto-aof-rewrite-min-size 64mb    最小64mb

• Redis會記錄上次重寫時的AOF大小，默認配置是當AOF文件大小是上次rewrite後大小的一倍且文件大於64M時觸發

2.7 AOF 優勢

# The fsync() call tells the Operating System to actually write data on disk

# instead of waiting for more data in the output buffer. Some OS will really flush

# data on disk, some other OS will just try to do it ASAP.

#

# Redis supports three different modes:

#

# no: don't fsync, just let the OS flush the data when it wants. Faster.

# always: fsync after every write to the append only log. Slow, Safest.

# everysec: fsync only one time every second. Compromise.

#

# The default is "everysec", as that's usually the right compromise between

# speed and data safety. It's up to you to understand if you can relax this to

# "no" that will let the operating system flush the output buffer when

# it wants, for better performances (but if you can live with the idea of

# some data loss consider the default persistence mode that's snapshotting),

# or on the contrary, use "always" that's very slow but a bit safer than

# everysec.

#

# More details please check the following article:

# http://antirez.com/post/redis-persistence-demystified.html

#

# If unsure, use "everysec".

# appendfsync always      同步持久化 每次發生數據變更會被立即記錄到磁盤  性能較差但數據完整性比較好

   appendfsync everysec  (默認)異步操作，每秒記錄   如果一秒內宕機，有數據丟失

# appendfsync no             從不同步

• ​ appendfsync always     同步持久化 每次發生數據變更會被立即記錄到磁盤  性能較差但數據完整性比較好
•  appendfsync everysec  (默認)異步操作，每秒記錄   如果一秒內宕機，有數據丟失
•  appendfsync no   不同步：appendfsync no   從不同步
  • 配置文件如下：
  • 操作： 將默認的appendfsync everysec ==> appendfsync everysec

2.8 AOF 缺點

• 相同數據集的數據而言aof文件要遠大於rdb文件，恢復速度慢於rdb
  
• aof運行效率要慢於rdb, 每秒同步策略效率較好，不同步效率和rdb相同
  

---

3.總結

---

3.1 RDB 和 AOF 持久化方式

• RDB持久化方式:能夠在指定的時間間隔能對你的數據進行快照存儲
  
• AOF持久化方式:記錄每次對服務器寫的操作,
  • 當服務器重啓的時候:
    • 會重新執行這些命令來恢復原始的數據
    • AOF命令以redis協議追加保存每次寫的操作到文件末尾.
  • Redis還能對AOF文件進行後臺重寫,使得AOF文件的體積不至於過大

3.2 RDB 和 AOF 區別

• 相同數據集的數據而言
  • aof 文件要遠大於 rdb文件
  • aof 恢復速度慢於 rdb
  • aof 每秒同步策略效率較好
  • aof 不同步效率和 rdb相同

3.3 判斷是不是數據量大還是隻做緩存？

 只做緩存：如果你只希望你的數據在服務器運行的時候存在,你也可以不使用任何持久化方式.

3.4 同時開啓兩種持久化方式

• 在這種情況下,當redis重啓的時候會優先載入AOF文件來恢復原始的數據,因爲在通常情況下AOF文件保存的數據集要比RDB文件保存的數據集要完整.

• RDB的數據不實時，同時使用兩者時:
  • 服務器重啓也只會找AOF文件。那要不要只使用AOF呢？
    • 作者建議不要，因爲RDB更適合用於備份數據庫(AOF在不斷變化不好備份)，快速重啓，而且不會有AOF可能潛在的bug，
  • 留着作爲一個萬一的手段。
    

 

3.5 性能建議

• 因爲RDB文件只用作後備用途，建議只在Slave上持久化RDB文件，而且只要15分鐘備份一次就夠了，只保留save 900 1這條規則。
  

• 如果Enalbe AOF，
  • 好處：
    • 是在最惡劣情況下也只會丟失不超過兩秒數據，啓動腳本較簡單隻load自己的AOF文件就可以了。
  • 壞處:
    • 一是帶來了持續的IO，
    • 二是AOF rewrite的最後將rewrite過程中產生的新數據寫到新文件造成的阻塞幾乎是不可避免的。
      • 只要硬盤許可，應該儘量減少AOF rewrite的頻率，AOF重寫的基礎大小默認值64M太小了，可以設到5G以上。默認超過原大小100%大小時重寫可以改到適當的數值。

• 如果不Enable AOF ，僅靠Master-Slave Replication 實現高可用性也可以。
  • 好處：
    • 能省掉一大筆IO也減少了rewrite時帶來的系統波動。
  • 缺點：
    • 如果Master/Slave同時倒掉，會丟失十幾分鐘的數據，啓動腳本也要比較兩個Master/Slave中的RDB文件，載入較新的那個。（新浪微博就選用了這種架構）