Could you provide a benchmark between the disk & the memory cache?

Hi @spacewander

I think these two strategies are suitable for different scenarios. It may be difficult to get a reasonable test result.

For example:

These are suitable for disk strategy,

1. Large files (> 1MB)
2. Lots of large files
3. Small files (< 1MB) with hotspots
4. Persistence

For memory strategy,

1. Small files (< 1MB)
2. The amount of the cached data doesn't exceed the host memory

As a result, these two scenarios can compare is small files with hotspots. But Nginx's proxy-cache send cache file requires 2 disk io (One for header part, another for body part), here ignores pagecache because it's uncertain. Maybe the result is already known.

In addition, sorry for the limited environment, I cannot finish the benchmark. If someone can help to accomplish this test, I can provide the test steps.

Configure proxied server

1. generate test files in /home/nginx/data/

dd if=/dev/sda of=4k bs=4 count=1024
dd if=/dev/sda of=64k bs=64 count=1024
dd if=/dev/sda of=512k bs=512 count=1024
dd if=/dev/sda of=1024k bs=1024 count=1024

2. configure location in proxied server

   server {
       listen 1996;
       server_name localhost;

   	location / {
   	    if ($arg_size) {
   	        rewrite .* /${arg_size} break;
   	    }
   	
   	    root /home/nginx/data/;
   	}

   }

Configure the APISIX

need to download master branch: https://github.com/agile6v/apisix

When testing memory strategy, we need to use this configuration.

curl http://127.0.0.1:9080/apisix/admin/routes/1  -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "plugins": {
        "proxy-cache": {
            "cache_strategy": "memory",
            "cache_key":  ["$uri", "-cache-id"],
            "cache_bypass": ["$arg_bypass"],
            "cache_zone": "memory_cache",
            "cache_method": ["GET"],
            "cache_http_status": [200],
            "hide_cache_headers": true,
            "no_cache": ["$arg_nocache"]
        }
    },
    "upstream": {
        "nodes": {
            "127.0.0.1:1996": 1
        },
        "type": "roundrobin"
    },
    "uri": "/*"
}'

This for disk strategy.

curl http://127.0.0.1:9080/apisix/admin/routes/1  -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "plugins": {
        "proxy-cache": {
            "cache_strategy": "disk",
            "cache_key":  ["$uri", "-cache-id"],
            "cache_bypass": ["$arg_bypass"],
            "cache_zone": "disk_cache_one",
            "cache_method": ["GET"],
            "cache_http_status": [200],
            "hide_cache_headers": true,
            "no_cache": ["$arg_nocache"]
        }
    },
    "upstream": {
        "nodes": {
            "127.0.0.1:1996": 1
        },
        "type": "roundrobin"
    },
    "uri": "/*"
}'

Verify the configuration

curl -v -o /dev/null http://127.0.0.1:9080/hello?size=4k

Benckmark tool use wrk

./wrk http://127.0.0.1:9080 -s scripts/random_url.lua

random_url.lua contains the following code.

math.randomseed(os.time())

function Random(n)
    local t = {
        "a","b","c","d","e","f","g","h","i","j","k","l","m","n","o","p","q","r","s","t","u","v","w","x","y","z",
    }
    local s = ""
    for i =1, n do
        s = s .. t[math.random(#t)]
    end;
    return s
end

local arg_size = "?size=4k"

request = function()
   return wrk.format("GET", "/" .. Random(3) .. arg_size)
end{}

Note:
Random(3) will generate 17000+ unique random url
Random(4) will generate 456976+ unique random url

There is still CI failure: https://github.com/apache/apisix/runs/3719358180

@shuaijinchao
Please take a look.

good feature. I benchmarked this PR. In most scenarios, disk cache is better than shared memory cache.
@agile6v can look my test results: Apache APISIX Proxy Cache Plugin Benchmark

good feature. I benchmarked this PR. In most scenarios, disk cache is better than shared memory cache. @agile6v can look my test results: Apache APISIX Proxy Cache Plugin Benchmark

Good work. Is WRK and APISIX on the same machine? If the machine memory is too small, SWAP will be used and pls take a look at the MISS ratio. Could you use the larger memory machine to test? Thanks.

Would you provide your own benchmark? It would satisfy your idea better.

Would you provide your own benchmark? It would satisfy your idea better.

Hi @spacewander Sorry for the limited environment reason, there is no resources to do this test now.

Would you provide your own benchmark? It would satisfy your idea better.

Hi @spacewander Sorry for the limited environment reason, there is no resources to do this test now.

OK, let me look at the implementation first.

t/plugin/proxy-cache-memory.t                 (Wstat: 2048 Tests: 123 Failed: 8)
  Failed tests:  12, 31, 60, 75, 83, 90, 94, 98
  Non-zero exit status: 8
Files=262, Tests=10124, 2074 wallclock secs ( 2.54 usr  0.46 sys + 322.91 cusr 110.99 csys = 436.90 CPU)
Result: FAIL

Look like some tests are not passed?