ACL 2026 · LONG PAPER · VOLUME 1ACL 2026 · LONG PAPER · VOLUME 1

EVM-QuestBench: An Execution-Grounded Benchmark for Natural-Language Transaction Code Generation

Pei Yang¹* · Wanyi Chen²* · Ke Wang¹ · Lynn Ai¹ · Eric Yang¹ · Tianyu Shi¹†

¹ Gradient  ·  ² Soochow University
*共同贡献（同等贡献） ·  †通讯作者*Equal contribution ·  †Corresponding author

把自然语言的链上意图，放进真实 EVM 执行中验证。

Putting natural-language blockchain intent into real EVM execution.

📄 ACL 正式论文ACL Anthology ⌁ arXivarXiv ⌘ 代码与题库Code 🤗 数据集Dataset ✎ 引用 / BibTeXCite / BibTeX

Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics · 2026 · pp. 35513–35529Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics · 2026 · pp. 35513–35529

ACL 2026主会 Long Paper
Volume 1Main-conference
Long Paper · Vol. 1
107固定任务定义
62 Atomic + 45 Compositetask definitions
62 Atomic + 45 Composite
10,700执行实例
20 模型 × 5 rounds × 107execution instances
20 models × 5 rounds × 107
8,236最高 aggregate score
Claude-Sonnet-4.5best aggregate score
Claude-Sonnet-4.5

摘要式定位Abstract

EVM-QuestBench 研究一个具体而容易被高估的问题：语言模型能否把用户的自然语言区块链意图，编译成可执行的交易代码，并真正改变 EVM 状态。与只比较生成代码字符串的评测不同，本基准在 BSC 主网 fork 上执行模型返回的 unsigned transaction，再根据 receipt、调用目标、calldata、余额、allowance、NFT owner、LP token 等状态证据评分。基准把能力拆成单步 Atomic 与多步 Composite，Composite 还显式记录规划、逐步执行、反馈和效率衰减。

EVM-QuestBench asks whether language models can compile natural-language blockchain intent into executable transaction code that produces the intended EVM state. Instead of comparing strings, it executes unsigned transactions on a BSC mainnet fork and scores receipts, targets, calldata, balances, allowances, NFT ownership, LP tokens, and other state evidence. The benchmark separates single-operation Atomic tasks from multi-step Composite workflows, with explicit planning, feedback, and efficiency decay for the latter.

数据集与复现资源Dataset and reproducibility resources

EVM-QuestBench 的 107 个任务定义已发布到 Hugging Face，包含 62 个 Atomic 任务和 45 个 Composite 工作流，可在线浏览并通过 datasets 直接加载。GitHub 仓库提供完整 Runner、Validator、Anvil/BSC 执行环境和复现实验说明。

All 107 EVM-QuestBench task definitions are available on Hugging Face as 62 Atomic tasks and 45 Composite workflows. They can be explored in the Dataset Viewer or loaded directly with datasets. The GitHub repository provides the full runner, validators, Anvil/BSC execution environment, and reproduction instructions.

from datasets import load_dataset

dataset = load_dataset("berryccc1/EVM-QuestBench")

Hugging Face DatasetGitHub ImplementationHugging Face Paper

论文 Figure 1 · 方法总览Paper Figure 1 · Method overview

从任务定义到可验证的 EVM 状态From task definition to a verifiable EVM state

这不是一个“模型回答完就结束”的代码题。题目 JSON 负责定义任务，动态实例化负责生成本轮自然语言，模型负责生成代码或动作，Runner 在隔离 fork 上执行，Validator 最后对 receipt 与状态变化打分。

This is not a code question that ends when the model responds. JSON defines the task, dynamic instantiation creates the current natural-language instance, the model generates code or an action, the runner executes it on an isolated fork, and validators score the receipt and state change.

62 Atomic45 CompositeBSC forkTask-specific validators
EVM-QuestBench layered evaluation architecture from task definition to environment and validation
Figure 1. EVM-QuestBench evaluation architecture and end-to-end pipeline.

为什么普通代码评测不够？Why code similarity is not enough

01

字符串正确 ≠ 交易正确String match ≠ valid transaction

selector、ABI 编码、to、value 或 decimals 只错一个字符，代码仍可能“看起来合理”，但上链会 revert 或产生错误状态。

A plausible-looking snippet can still revert or encode the wrong recipient, selector, value, or decimals.

02

查询与执行是两件事Reading and acting are different

模型可以读取余额、allowance 和 reserves，但真正的能力是把这些观测转成正确的下一笔交易。

The model may read balances, allowances, or reserves; the hard part is turning observations into the next correct transaction.

03

多步工作流会暴露依赖Workflows expose dependencies

approve → swap → query 的顺序、参数传播和额外轮次，无法用一段参考代码充分表达。

Ordering, parameter propagation, and unnecessary rounds in approve → swap → query cannot be captured by one reference string.

评测执行契约The execution contract

题目定义、模型调用、执行环境和验证器是不同组件；页面把它们明确拆开。

Task definitions, model calls, execution infrastructure, and validators are separate components.

01 · Question JSON固定 id、模板、参数、前后置条件id, templates, parameters, predicates
02 · Instantiate采样金额、比例、地址并解析自然语言 Tasksample values and resolve the natural-language Task
03 · Model APIAtomic 一次；Composite 规划后逐 subtask 多轮one-shot Atomic; multi-round Composite after planning
04 · Runner + Anvil签名、广播、执行交易并记录 receipt / statesign, broadcast, execute, and record receipt / state
05 · Validator把执行证据映射为分数与 pass 标记map execution evidence to score and pass
Model只生成代码或动作 JSON，不直接拥有链上执行权。Generates code or action JSON; it does not directly control the chain.
ControllerComposite 的规划器与循环器，保存每轮结果并构造下一轮输入。Orchestrates planning, rounds, feedback, and the next prompt.
Runner运行 TypeScript、签名并提交 unsigned transaction。Runs TypeScript, signs, and submits the unsigned transaction.
Validator检查 receipt、调用细节和最终 EVM 状态。Checks receipts, call details, and final EVM state.

两个 split，两个能力面Two splits, two capability profiles

Atomic · 62

一次模型 API 调用。输入为通用 atomic_role_prompt、通用 environment_description 和当前解析出的 Task；输出是一个 TypeScript 模块，导出 executeSkill，返回一笔 unsigned transaction object。

One model API call. The input combines the atomic role prompt, environment description, and a resolved task. The output is a TypeScript module exporting executeSkill and returning one unsigned transaction object.

  • BNB / ERC-20 / NFT
  • PancakeSwap / staking / contract calls
  • receipt + state-delta validation

Composite · 45

模型先给出 JSON planning；Controller 按 subtask 多轮调用模型。每轮的 action_result、tx_hash、query result 或 error 会进入下一轮上下文。

The model first emits a JSON plan; the Controller calls it per subtask. Each round feeds action results, transaction hashes, query results, or errors into the next context.

  • planning phase + execution rounds
  • optimal steps are defined per task
  • extra steps receive efficiency decay

真实题目案例：从 JSON 到状态验证Concrete cases: JSON → execution → state checks

ATOMIC · bnb_transfer_percentage.json

Transfer {percentage}% of my BNB balance to {to_address}

Question JSON
{
  "id": "bnb_transfer_percentage",
  "parameters": {"percentage": {"min": 10, "max": 90, "step": 5},
                 "to_address": {"type": "address", "method": "random"}},
  "natural_language_templates": [
    "Transfer {percentage}% of my BNB balance to {to_address}"
  ]
}
Resolved Task

Transfer 15% of my BNB balance to 0x1111...1111

Model output contract
export async function executeSkill(...): Promise<TransactionRequest> {
  const balance = await provider.getBalance(agentAddress);
  return { to: "0x1111...1111", value: balance * 15n / 100n, gasLimit: 21000n };
}
receipt.successto = recipientvalue ≈ 15%sender balance delta
COMPOSITE · composite_approve_swap_query_result.json

Approve USDT, swap for BUSD, then verify the received balance

Question JSON
{
  "composite_structure": {
    "optimal_steps": 3,
    "atomic_operations": ["erc20_approve",
      "swap_exact_tokens_for_tokens", "query_erc20_balance"]
  },
  "interaction_config": {
    "score_decay_formula": "base_score * min(1, optimal_steps / actual_steps)"
  }
}
Resolved Task

Approve 4.50 USDT for PancakeSwap, swap it for BUSD, and check how much BUSD you received.

Controller loop
  1. plan: approve → swap → query
  2. execute one action per round
  3. feed receipt / query result into next prompt
  4. submit when done; penalize unnecessary rounds
router = PancakeSwapselector = 0x38ed1739USDT decreasesBUSD increases

分数来自执行证据Scores come from execution evidence

Atomic

每个题目按其 validator 的后置条件评分。以 BNB 百分比转账为例，validator 会分别检查交易成功、目标地址、转账金额和发送方余额变化；因此可能得到部分分，但不是过程分。

Each task is scored against its validator post-conditions. For the BNB percentage transfer, checks cover success, recipient, amount, and sender balance delta. Partial credit reflects satisfied outcome checks, not hidden reasoning steps.

successtovaluestate

Composite

Composite 复用关键 atomic validator，同时比较实际执行轮次和题目定义的 optimal steps。额外轮次会按题目中的衰减公式降低分数。

Composite reuses the key atomic validator and compares actual rounds with the task's optimal steps. Unnecessary rounds reduce the score through the task-defined decay formula.

3 optimal steps4 steps → 75%6 steps → 50%

最终评测同时保留原始执行证据：交易 receipt、tx hash、查询结果、错误信息和 before/after state。代码“像答案”不等于链上“达到答案”。

The evaluation retains receipts, transaction hashes, query results, errors, and before/after state. Code that looks like an answer is not enough unless the chain reaches the intended state.

ACL 论文主结果Main ACL results

论文在 20 个模型上运行 5 个独立 rounds，共 10,700 个执行实例。下表列出论文报告的最高六个 aggregate scores。

The paper evaluates 20 models across 5 independent rounds, yielding 10,700 execution instances. The table shows the six highest reported aggregate scores.

ModelAtomicCompositeTotalSDCV
Claude-Sonnet-4.54,2783,9588,2361742.1%
Gemini-3-Pro4,3023,5617,8632673.4%
GPT-54,1283,6467,7744075.2%
GPT-5.13,7153,6177,3321442.0%
Kimi-K2-Thinking3,6053,6337,2382072.9%
DeepSeek-V3.23,0713,6386,7093845.7%
Atomic ↔ Composite asymmetry单步高分不保证多步规划同样稳健。Strong single-step execution does not guarantee robust workflow planning.
Variance mattersSD/CV 显示不同轮次的稳定性。SD/CV expose run-to-run stability.
Cost is measurable完整 107 题约 334K tokens；标准模型单轮约 $0.06–$0.29，thinking-heavy 模型最高约 $14.16。A full 107-task run uses about 334K tokens; standard-model rounds cost roughly $0.06–$0.29, while thinking-heavy models reach about $14.16.
Paper Figure 3 showing total benchmark score versus token usage and API cost
Figure 3. Total score versus total token usage and API cost.

论文观察Takeaway

执行正确性与工作流能力并不等价Execution precision and workflow competence are not equivalent

论文报告的模型分布显示，Atomic 与 Composite 能力存在明显不对称：有些模型擅长单步交易，却在多步依赖、接口格式和效率控制上失分。

The reported model distribution shows a clear Atomic–Composite asymmetry: some models are strong at single transactions but lose points on multi-step dependencies, interface format, and execution efficiency.

Paper Figure 2 showing the end-to-end benchmark pipeline
Figure 2. End-to-end evaluation pipeline.

闭环执行，而不是静态生成A closed execution loop, not static generation

每个分数都能追溯到一条链上证据Every score traces back to on-chain evidence

一次评测会记录参数采样、Prompt、模型响应、执行前快照、交易 receipt、tx hash、查询结果、错误信息和执行后状态。这样别人不仅能看到分数，还能理解模型到底在哪一层失败。

A run records parameter sampling, prompts, model responses, pre-state snapshots, receipts, transaction hashes, query results, errors, and post-state. The score is therefore tied to an auditable failure surface.

复现与资料Reproduce the benchmark

仓库包含题库 JSON、参数生成器、Controller、Anvil fork 环境、TypeScript runner 和 validators。安装依赖后可按 split 运行：

The repository contains JSON task definitions, parameter generation, Controller, Anvil-fork environment, TypeScript runner, and validators. Run either split after installing dependencies:

pip install -r requirements.txt
cd bsc_quest_bench/skill_runner && bun install && cd ../..
python run_quest_bench.py --model MODEL --type atomic
python run_quest_bench.py --model MODEL --type composite

RepositoryHugging Face DatasetHugging Face PaperACL AnthologyarXivDOI

引用这项工作Citation

BibTeX（点击右侧按钮复制）BibTeX (copy with the button)
@inproceedings{yang-etal-2026-evm-questbench,
  title = {EVM-QuestBench: An Execution-Grounded Benchmark for Natural-Language Transaction Code Generation},
  author = {Yang, Pei and Chen, Wanyi and Wang, Ke and Ai, Lynn and Yang, Eric and Shi, Tianyu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  pages = {35513--35529},
  publisher = {Association for Computational Linguistics},
  url = {https://aclanthology.org/2026.acl-long.1642/}
}