Quick Start

Setup, Rendering, Generation, and Judging

This page mirrors the benchmark workflow in the repository: create the environment, render PNG references, run a smoke test, run full generation, and then run the DeepEval judge.

Setup Back to Overview

Environment Setup

Clone the repository, install Python and Node dependencies, then create a local credential file. Scripts auto-load .env.local and .env.

git clone https://github.com/YuvrajSingh-mist/ASCIITermDraw-Benchmark.git
cd ASCIITermDraw-Benchmark
uv sync
npm install
npx playwright install chromium
cp .env.example .env.local
TOGETHER_API_KEY=...
OPENAI_API_KEY=...
ANTHROPIC_API_KEY=...

The Together AI key is for generation (below); you only need one of OPENAI_API_KEY/ANTHROPIC_API_KEY depending on which judge provider you use in the Scoring section.

Smoke Test

Run a synchronous shared-module spot check before spending time on full-benchmark batch inference.

--tasks tasks requires the private task set (see the Tasks page). Don't have it yet? Download the public dataset locally first, then swap in --tasks public_dataset to run this exact command against the 12 public example tasks instead: 

huggingface-cli download YuvrajSingh9886/asciitermdraw-bench-public \
  --repo-type dataset --local-dir public_dataset
uv run smoke \
  --model Qwen/Qwen3.7-Plus \
  --tasks tasks \
  --outputs outputs/smoke-qwen3.7-plus \
  --sample-count 5 \
  --seed 7 \
  --reasoning-effort none \
  --network-retries 5

Full Generation

Run the full benchmark via Together AI's synchronous chat-completions transport (one request per task). Category 3 tasks automatically attach source images.

uv run run-model \
  --model Qwen/Qwen3.7-Plus \
  --tasks tasks \
  --outputs outputs/your-run \
  --reasoning-effort none \
  --network-retries 5

Scoring

Judging goes through DeepEval BaseMetric against OpenAI or Anthropic, following the shared judge prompt, with structured JSON output where the main score is geval_structural_score (see structural score) plus geval_semantics_score (see semantic score). This is the only scoring path; there is no separate deterministic/text-heuristic step. 

uv run judge-geval \
  --provider openai \
  --model your-judge-model \
  --tasks tasks \
  --outputs outputs/your-run \
  --results results_your_run.csv \
  --num-judgments 10

If a run dies partway through (rate limit, quota, crash), rerun the exact same command with --resume added. It skips any task that already has a gval/result.json under --outputs and only judges what's left: 

uv run judge-geval \
  --provider openai \
  --model your-judge-model \
  --tasks tasks \
  --outputs outputs/your-run \
  --results results_your_run.csv \
  --num-judgments 10 \
  --resume