pass CONFIG SET maxmemory 100mb or in redis.conf Profile settings maxmemory 100mb Redis memory usage limit. When the maximum memory value is reached, the memory elimination policy will be triggered to delete data.

In addition, when the key reaches the expiration time, Redis will have the following two strategies for deleting expired data:

• The background timing task selects part of the data to delete;
• Lazy deletion.

For details, please refer to “Redis Expired Data Deletion Things”.

Assuming that the Redis instance saves 5GB of data and now deletes 2GB of data, will the memory occupied by the Redis process be reduced? (Also known as RSS, the number of memory pages consumed by the process).

The answer is: it may still take up about 5GB of memory, even though the Redis data only takes up about 3GB.

Everyone must setmaxmemoryotherwise Redis will continue to allocate memory for the newly written data, and failure to allocate will cause the application to report an error, and of course it will not cause a downtime.

Where does the freed memory go?

Obviously delete the data, use the top command to view, why is it still occupying so much memory?

Where did all the memory go?use info memory Command to get Redis memory related indicators, I listed several important data:

127.0.0.1:6379> info memory
# Memory
used_memory:1132832  // Redis 存储数据占用的内存量
used_memory_human:1.08M  // 人类可读形式返回内存总量
used_memory_rss:2977792  // 操作系统角度，进程占用的物理总内存
used_memory_rss_human:2.84M // used_memory_rss 可读性模式展示
used_memory_peak:1183808 // 内存使用的最大值，表示 used_memory 的峰值

used_memory_peak_human:1.13M  // 以可读的格式返回 used_memory_peak的值
used_memory_lua:37888   // Lua 引擎所消耗的内存大小。
used_memory_lua_human:37.00K
maxmemory:2147483648    // 能使用的最大内存值，字节为单位。
maxmemory_human:2.00G  // 可读形式
maxmemory_policy:noeviction // 内存淘汰策略

// used_memory_rss / used_memory 的比值，代表内存碎片率
mem_fragmentation_ratio:2.79 

Redis process memory consumption mainly consists of the following parts:

• The memory occupied by Redis’s own startup;
• memory for storing object data;
• Buffer memory: mainly by client-output-buffer-limit client output buffer, copy backlog buffer, AOF buffer.
• Memory fragmentation.

The memory occupied by Redis’s own empty process is very small and negligible, and the object memory is a piece of proportion, which stores all the data.

It is easy to get out of control if there is a large traffic scene in the buffer, causing the Redis memory to be unstable, which needs to be paid attention to.

Excessive memory fragmentation leads to the fact that there is space available, but data cannot be stored.

Fragmentation = used_memory_rss actually used physical memory (RSS value) divided by used_memory actually stored data memory.

What is memory fragmentation

Memory fragmentation will cause the memory space to be free but unable to store data. For example, if you go to the cinema to watch a movie with a pretty lady, you definitely want to sit together.

Suppose there are now 8 seats, 4 tickets have been sold, and 4 more can be bought. But as a coincidence, the people who bought the tickets were very strange, and they bought tickets one seat apart.

Even if there are still 4 seats available, you can’t buy a ticket with two seats connected together, big gift crab!

Causes of memory fragmentation

What causes memory fragmentation?

There are two main reasons:

• The allocation strategy of the memory allocator.
• The size of the key-value pair is different and the deletion operation: Redis frequently performs update operations, deletes a large number of expired data, and the released space (not continuous enough) cannot be reused, resulting in an increase in the fragmentation rate.

Next, I will discuss the reasons for what actually happened…

The allocation strategy of the memory allocator

The default memory allocator of Redis uses jemalloc, and the optional allocators are: glibc, tcmalloc.

The memory allocator cannot allocate on demand, but uses a fixed range of memory blocks for allocation.

For example, 8 bytes, 16 bytes…, 2 KB, 4KB, when the requested memory is closest to a fixed value, jemalloc will allocate the space closest to the fixed value.

In this way, memory fragmentation will occur. For example, the program only needs 1.5 KB, and the memory allocator will allocate 2KB space, then this 0.5KB is fragmentation.

The purpose of this is to reduce the number of memory allocationsFor example, if you apply for 22 bytes of space to save data, jemalloc will allocate 32 bytes. If you want to write 10 bytes later, you don’t need to apply for space from the operating system, and you can use the previously requested 32 bytes.

When deleting a key, Redis will not immediately return the memory to the operating systemthis situation is caused by the management of the underlying memory allocator, for example, most deleted keys are still allocated in the same memory page as other valid keys.

In addition, in order to reuse free memory blocks, the allocator deletes 2 GB of the original 5 GB of data. When data is added to the instance again, the RSS of Redis will remain stable and will not increase too much.

becauseThe memory allocator basically reuses the 2GB of memory freed by the previous delete.

Different sizes of key-value pairs and deletion and modification operations

Since the memory allocator allocates memory according to a fixed size, the allocated memory space is usually larger than the size occupied by the actual data, which will cause fragmentation and reduce the storage efficiency of the memory.

In addition, the frequent modification and deletion of key-value pairs leads to the expansion and release of memory space. For example, if a string that originally occupied 32 bytes is now modified to a string that occupies 20 bytes, then the released 12 bytes are free space .

If the next data storage request needs to apply for a 13-byte string, the 12-byte space just released cannot be used, resulting in fragmentation.

The biggest problem with fragmentation: the total amount of space is large enough, but these memories are not continuous, and may not be able to store data.

Solution to memory fragmentation

So how to solve it?

First of all, it is necessary to determine whether memory fragmentation has occurred, focusing on the front INFO memory command prompt mem_fragmentation_ratio Index, indicating the memory fragmentation rate:

mem_fragmentation_ratio = used_memory_rss/ used_memory

If 1 < Fragmentation Rate < 1.5, it can be considered reasonable, and greater than 1.5 indicates that the fragmentation rate has exceeded 50%, and we need to take some measures to solve the problem of excessive fragmentation rate.

Restart Dafa

The simplest and rude way is to restart. If persistence is not enabled, the data will be lost.

If you enable persistence, you need to use RDB or AOF to restore data. If there is only one instance, the large data will lead to the failure to provide services for a long time during the recovery phase, and the high availability will be greatly reduced.

What should I do?code brother pretty boy

Automatically clean up memory fragments

Now that you call me pretty boy, let me tell you the ultimate killer move: After Redis version 4.0, it provides a memory fragmentation cleaning mechanism.

How to clean it?

It’s very simple, still the above example, I want to buy two movie tickets connected together. Communicate with others and switch positions, and it will be realized.

For Redis, when a continuous memory space is divided into several discontinuous spaces, the operating system first moves and stitches the data together one by one, and releases the space occupied by the original data to form a continuous free memory space . .

As shown below:

The cost of automatically cleaning up memory fragmentation

Although automatic cleaning is good, don’t do it recklessly. The operating system needs to consume resources to move data to a new location and then release the original space.

The instructions for Redis to operate data are single-threaded, so when data is copied and moved, the request can only be processed after the debris is cleaned up, resulting in performance loss.

How to avoid the impact of cleaning debris on performance and realize automatic cleaning?

Good question, use the following two parameters to control memory fragmentation cleanup and end timing, avoid taking up too much CPU, and reduce the performance impact of cleaning fragmentation on Redis processing requests.

Enable automatic memory defragmentation

CONFIG SET activedefrag yes

This is just to enable automatic cleaning. When the cleaning needs to meet the following two conditions at the same time, the cleaning operation will be triggered.

cleaning conditions

active-defrag-ignore-bytes 200mb: The memory occupied by memory fragmentation reaches 200MB, start to clean up;

active-defrag-threshold-lower 20: The space of memory fragmentation accounts for 20% of the space allocated by the system to Redis, and it starts to be cleaned up.

avoid performance impact

Cleanup time is available, and the impact of cleanup on performance needs to be controlled. One or two settings are used to allocate the CPU resources occupied by cleaning debris first, so as to ensure that debris can be cleaned normally, and the performance impact on Redis processing requests can be avoided.

active-defrag-cycle-min 20: During the automatic cleaning process, the proportion of CPU time occupied is not less than 20%, so as to ensure that the cleaning task can be carried out normally.

active-defrag-cycle-max 50: The proportion of CPU time occupied by the automatic cleaning process cannot be higher than 75%. If it exceeds, the cleaning will be stopped immediately to avoid blocking Redis and causing high latency.

Summarize

If you find that the memory occupied by Redis to store data is much smaller than the memory allocated to Redis by the operating system, but the data cannot be saved, there may be a lot of memory fragmentation.

pass info memory Command to see memory fragmentationmem_fragmentation_ratio Whether the indicator is normal.

Then we enable automatic cleanup and set the cleanup timing and CPU resource usage reasonably. This mechanism involves memory copying, which will pose potential risks to Redis performance.

If the performance of Redis slows down, check whether it is caused by cleaning up debris, if so, then adjust it down active-defrag-cycle-max value.

Ma Ge created a technology group Moyu group, if you want to fish, fart and chat with programmers from various cities, join the group.

Finally, can I call me handsome? Do you have any questions you want to tell Brother Ma? Leave a message in the message area, know everything.

Like, bookmark, share and let’s go!

refer to

[1].Redis core technology and practice

[2].https://juejin.cn/post/6844903967298682893#heading-4

[3].https://redis.io/docs/reference/optimization/memory-optimization/#memory-allocation