ASCIITermDraw-Bench

ASCIITermDraw-Bench is a benchmark for structured ASCII diagram generation and editing. Instead of grading models on prose quality alone, it checks whether they can place labels, preserve spacing, route arrows, maintain overall structure, and make constrained edits to an existing diagram.

The benchmark covers four families of tasks:

• Box Layout Basics: single boxes, rows, columns, arrows, and spacing-sensitive layout
• Network Topology Diagrams: clusters, buses, data flows, and connection topology
• Diagram Editing: image-conditioned edits applied to an existing ASCII diagram
• Software Architecture Diagrams: real-world gateways, pipelines, feeds, and storage layers

Every task ships with a prompt, a reference ASCII artifact (a PNG to be rendered), and structured assertions.

Every candidate diagram is judged through an LLM-as-a-judge pipeline: a DeepEval BaseMetric judge (OpenAI or Anthropic), following the shared judge prompt, scores each candidate against the reference on two axes, averaged over multiple judge rounds and combined into the final geval_score:

• Structural score: do the required labels, entities, and edges actually appear, checked against structured assertions.
• Semantic score: does the layout, spacing, and connection routing actually match the prompt.

Category 3 is multimodal: the model receives a rendered source.png and must apply a textual edit instruction while still returning plain ASCII. The remaining categories are text-to-ASCII generation tasks.

Every candidate is judged num_judgments times independently (not cached repeats) so the reported score is a mean across rounds with a population stdev, not a single noisy judge call. See the Leaderboard for published results and metrics.json for the exact methodology notes.

• Box Layout Basics - simple boxes, chains, stacks, fan-in, fan-out, diamonds, and spacing-sensitive arrangements.
• Network Topology Diagrams - brokers, parameter servers, service meshes, CDNs, DNS flows, and cluster communication patterns.
• Diagram Editing - image-conditioned edit tasks that require preserving the original structure while applying the requested change.
• Software Architecture Diagrams - 3-tier systems, event pipelines, API gateways, queues, ETL layouts, and other canonical engineering sketches.

Generation follows the flow shown in Benchmark Anatomy: a task's prompt.txt (plus source.png for editing tasks) goes to Together AI for model inference, producing raw ASCII output, which is then rendered to a diagram-only PNG before being handed to the judge.

• Playwright PNG rendering: reference ASCII artifacts are rendered as diagram-only images for editing tasks and visual inspection; see Technical Details.
• Together AI generation: model responses and rendered PNGs only; Together is not used for judging.

Judging follows the flow shown in Judge / GEval Process:

1. Candidate and reference PNGs, plus the judge prompt and assertions.json, go to the judge.
2. The judge round repeats num_judgments times (default 5).
3. Structural score: harness-computed, not judged directly. The judge reports what it observes; the harness checks each observation against assertions.json as an independent, all-or-nothing component (no partial credit within a component), then averages whichever of these apply to the task:
   • required_labels: every required label appears somewhere in the diagram.
   • entity_count: the observed entity count matches the expected count exactly.
   • required_edges: every required connection is present.
   • required_edge_labels: editing tasks only, every required edge label is present.
   • preserved_elements: editing tasks only, elements that should survive the edit still do.
4. Semantic score: the judge's own direct 0.0-1.0 rubric judgment, informed by its own checks on:
   • connections_correct: are the connections actually made where they should be.
   • text_inside_nodes_correct: is the text inside each node/box correct.
   • text_centered_in_nodes: is that text centered within its node/box.
   • labels_spelled_correct: are all labels spelled correctly.
   • arrows_cleanly_aligned: are arrows cleanly aligned to their source and target.
   • layout_matches_prompt: does the overall layout follow the prompt.
5. Both scores are averaged across rounds and summed into geval_score.

• Repeated judging: each task can be judged multiple times (default 5) with mean/stdev reported, to separate real quality differences from judge-scoring noise.
• Repository-first organization: task directories, scripts, outputs, and website metadata are all arranged for extension and public contribution.

One hard-difficulty task per category, loaded live from the public Hugging Face dataset (the same format as the private benchmark). Click a diagram to view it full size, or see every public example on the Tasks page.

Task assets live in named category directories, and the website data is built directly from the repository tree (see website/tools/build_site_data.mjs) rather than from hand-maintained content.

Rendering

Every ASCII artifact (reference or candidate) is rendered to a diagram-only PNG by scripts/rendered/render_ascii.mjs: headless Playwright/Chromium loads the raw text into a monospace <pre> element on a minimal page, resizes the viewport to exactly fit that content's bounding box, and screenshots only that region at 2x device scale. There is no surrounding page chrome in the output, just the diagram itself.

Inference and Scoring

• Smoke generation: synchronous shared-module Together AI calls for fast spot checks on a random subset of tasks.
• Full generation: the same synchronous Together AI transport, run across all tasks.
• DeepEval judge: a custom BaseMetric against OpenAI or Anthropic, following the shared judge prompt. Structural score verifies visual evidence against assertions.json; semantic score comes from a task-specific rubric prompt.

A compact view of the benchmark coverage before diving into the full detailed task listing.