The task
How could a benchmark measure AI capabilities for specific research and software tasks without mistaking a public benchmark score for real-world readiness?
Projected solution
I propose a task-family benchmark with executable checks, evidence-based rubrics, separate human adjudication, and reserved evaluation items.
The plan starts from concrete use cases and observable success criteria. Task variants separate superficial wording from the underlying capability. Capability, refusal behavior, cost, and failure severity would be reported separately, with contamination checks and uncertainty estimates.
Proposed workflow
Proposed: executable tasks · rubric scoring · reproducible assessment- 01
Specify
Define the intended use, construct, observable behavior, and scoring rubric.
- 02
Construct
Author task families and reserve evaluation items before model comparison.
- 03
Assess
Combine executable checks and independent adjudication with uncertainty reporting.
Implementation roadmap
Proposed milestones / Not started- Define and freeze. Specify the task, data rights, baselines, evaluation criteria, and reserved test material.
- Build the smallest end-to-end study. Implement the proposed workflow with traceable inputs and reproducible outputs.
- Compare and stress-test. Run controlled baseline comparisons and ablations; inspect failure modes and resource use.
- Reproduce and report. Repeat the study from a clean environment and publish measured findings with supporting artifacts.
The research proposal expands these milestones into project-specific methods, experimental controls, and deliverables.
Methods / evidence / reproduction
Read the research.
The proposed evaluation examines inter-rater agreement, task-family generalization, repeatability, and model ranking stability. A small research-assistance and software-maintenance pilot would establish scoring behavior before expanding scope.
The proposal develops the research question, proposed method, evaluation design, and phased roadmap. Its GitHub directory contains the planning documents. No implementation or experimental results are claimed.