Add a new benchmark ENAMEL for evaluating the efficiency of LLM-generated code

[q-rz]

• Motivation first:

Code efficiency is an important measure of code quality and has been overlooked in existing evaluations. Not until very recently have some sporadic attempts been made to evaluate the efficiency of LLM-generated code, and a number of fundamental challenges remain uncharted and open, including how to rigorously handle right-censored execution time, sample size, algorithm/implementation optimization, correctness, and worst-case efficiency. To address these challenges, we develop ENAMEL (EfficeNcy AutoMatic EvaLuator), a rigorous and high-standard benchmark for evaluating the capability of LLMs in generating expert-level efficient code.

• Write a full paragraph describing the feature:

Firstly, we propose a new efficiency metric called eff@k, which generalizes the pass@k metric from correctness to efficiency and appropriately handles right-censored execution time. Furthermore, we derive an unbiased and variance-reduced estimator of eff@k via Rao--Blackwellization; we also provide a numerically stable implementation for the new estimator. Secondly, to set a high-standard for efficiency evaluation, we employ a human expert to design best algorithms and implementations as our reference solutions of efficiency, many of which are much more efficient than existing canonical solutions in HumanEval and HumanEval+. Moreover, to ensure a rigorous evaluation, we employ a human expert to curate strong test case generators to filter out wrong code and differentiate suboptimal algorithms. An extensive study across 30 popular LLMs using our benchmark ENAMEL shows that state-of-the-art LLMs still fall short of generating expert-level efficient code. This is our official implementation of ENAMEL and follows exactly the same approach in the paper.

• Provide a code snippet that demonstrates its future use:

accelerate launch main.py \
  --model <MODEL_NAME> \
  --max_length_generation 2048 \
  --tasks enamel \
  --temperature 0.8 \
  --top_p 0.95 \
  --do_sample True \
  --n_samples 10 \
  --batch_size 10 \
  --allow_code_execution

• In case this is related to a paper, please attach a link:

Our paper: https://arxiv.org/abs/2406.06647