Since you explicitly asked in the other PR, these are my particular needs:

• Most frequently I run benchmarks to test one git revision vs. another. I usually put the git revisions in separate columns right next to each other with an extra column for the speedup since I think that this is the easiest way to compare the results. I also add some basic hardware info like the GPU. My current workflow is that I just type the numbers from the console into an Emacs org mode table and then replace + with | to get a markdown table.
• I want to start collecting, curating, and publishing GPU data for my blog soon. This will include performance but also some perplexity benchmarks. I would want to periodically re-run those benchmarks in order to keep the numbers up-to-date. So my goal is to completely automate the process of running the benchmarks and creating the tables and plots.
• I will use Slurm to schedule the compute jobs associated with the previous point on my server. I therefore want an easy way to collect the data from the various runs for my analysis scripts. A database would be ideal, separate self-contained files that contain all relevant information also work.

I think we could to add sqlite as an output option if it is isolated to the example. We could copy the sqlite3.h/c files in the example directory to avoid having to install or download it separately to build llama.cpp, or just make it optional at compile time. The hardware info seems a bit more complicated to do in a multiplatform way, but maybe we could add it only for Linux initially and then expand it later to other platforms, if anyone is interested in that. Other than that, I think that this should work as is.

I think we could to add sqlite as an output option if it is isolated to the example.

Or we could just print SQL to stdout. SQLite comes with a binary that accepts SQL on stdin so something like llama-bench -o sql | sqlite3 llama.sqlite would work assuming that the user has SQLite installed (and the documentation has the command to initialize the database).

In the third row of the sample results table in OP, what is the meaning of the t/s column? Is it averaged across PP and TG?
If so, I don't think it is very informative - separate numbers would be much better.

Yes, it is the average t/s of a prompt of 512 tokens followed by a generation of 128 tokens. The way this works currently is that n_prompt and n_gen are part of the grid, by default have the values 0,512 and 0,128, and all combinations are tested (except 0,0 which would do nothing). I agree that it is probably not be very useful to do it this way, and may be better to consider each value of n_prompt and n_gen as a different test instead.

I just remembered: I think it would be useful to print a warning when the benchmark is being run with debug settings. I previously wasted time trying to figure out why the performance is suddenly terrible when I had just forgotten to remove LLAMA_DEBUG=1 from the compile args (which had disabled all optimizations).

I will add a warning if NDEBUG is not defined, but it is not completely reliable. I am not sure if there is a better way.

I think that would be good enough. The biggest factor for performance is compiling without optimizations (because otherwise the compiler may optimize out local variables) and to my knowledge this always implies that NDEBUG is not defined.

How about this for the performance columns: one column for prompt t/s, one for generation t/s, and one for the total time.

Unfortunately, that doesn't fit very well with the design. There may be any number of tests with any number of different parameters, tests don't need to include both prompt and generation, it is possible to disable either one by setting it to zero, or there may be multiple values for the number of tokens of prompt and generation.

A few more examples of the markdown output with CUDA:

./llama-bench

./llama-bench -m models/3B/ggml-model-q4_0.bin -m models/7B/ggml-model-q4_0.bin

./llama-bench -m models/3B/ggml-model-q4_0.bin -m models/7B/ggml-model-q4_0.bin -mmq 0,1 -n 0

./llama-bench -m models/3B/ggml-model-q4_0.bin -b 128,256,512 -n 0

./llama-bench -m models/3B/ggml-model-q4_0.bin -b 128,256,512 -n 0 -mmq 0,1

./llama-bench -ngl 20,30,99

When building without a GPU backend, the number of threads is always shown instead of the number of GPU layers:

./llama-bench -m models/3B/ggml-model-q4_0.bin

./llama-bench -m models/3B/ggml-model-q4_0.bin -p 0 -n 32 -t 8,16,32

./llama-bench -m models/3B/ggml-model-q4_0.bin -p 32 -n 0 -t 8,16,32

The current defaults are these:

static cmd_params cmd_params_defaults = {
    /* model         */ {"models/7B/ggml-model-q4_0.bin"},
    /* n_prompt      */ {512},
    /* n_gen         */ {128},
    /* n_batch       */ {512},
    /* f32_kv        */ {false},
    /* n_threads     */ {get_num_physical_cores()},
    /* n_gpu_layers  */ {99},
    /* main_gpu      */ {0},
    /* mul_mat_q     */ {true},
    /* low_vram      */ {false},
    /* tensor_split  */ {{}},
    /* reps          */ 5,
    /* verbose       */ false,
    /* output_format */ MARKDOWN
};

The default should serve as a good standard test, so we may want to change some of the values.

I will add a warning if NDEBUG is not defined, but it is not completely reliable. I am not sure if there is a better way.

You can check whether optimizations are enabled in gcc and clang by checking if __OPTIMIZE__ is defined—LLAMA_DEBUG uses -O0. I personally build without NDEBUG in case I encounter a bug during normal use.

Nice, thanks. Checking __OPTIMIZE__ for gcc and clang and _DEBUG for msvc should do it. This still doesn't stop anyone from compiling ggml with different flags, but it should be better than checking for NDEBUG only.

I noticed a weird bug when testing with prompt sizes 128 and 1024 in the same run with CUDA enabled. The test with size 128 will work, but the test with 1024 will fail with an out of memory in the scratch buffer error:

ggml_new_tensor_impl: not enough space in the scratch memory pool (needed 117440512, available 113246208)

When testing only with prompt size 1024 this doesn't happen. A new llama_model and llama_context is created for every test, so this shouldn't happen unless there is some global state in llama.cpp, and I don't think there is. There is global state in ggml-cuda, but I am not sure how that could affect the ggml scratch buffer.

I found the problem. Turns out, there is global state in llama.cpp:
https://github.com/ggerganov/llama.cpp/blob/0919a0f73d95cfb93a1646a1d1741a0615fe2c5e/llama.cpp#L118-L129
The scratch buffer size map is initialized the first time MEM_REQ_SCRATCH0 is called based on the value of n_ctx passed. Subsequent calls, will reuse the same map with the previous value of n_ctx.

I have added basic CPU info (Linux only) and GPU info (CUDA only), and SQL output (probably only works with sqlite). Is there anything else that could be added here? Otherwise, I consider this ready to review.