Generative Capability

Feature delivery, build reliability, and generated app galleries.

Retained generated outputs and their historical evaluators, not a matched or currently qualified experiment. Failures, incomplete coverage, replay scoring and different model/backend bindings remain explicit. A live flag does not establish containment or independent replication.

Three briefs and five harnesses. Historical recorded scores are preserved; uncaptured trajectory/honesty signals do not become current verified capabilities.

Models & settings — click to expand

Completeness by final feature-list position

Bins summarize checks by final feature-list position, not elapsed time or observed completion milestones.

Feature completeness by harness

Recorded feature completeness · bootstrap 95% CI whiskers stay attached to their harness

Generative dimensions

build · completeness · visual · plan/trajectory · honesty

Side-by-side app gallery

App explorer

Click a row for the feature checklist, the milestone plan, and scores. Red outline = claimed-but-failed (dishonest).
AppHarnessLangBuildCompletenessPlanHonesty

Inspect the evidence

Inside the dataset

3 items
chat-appChatGPT-style assistant (local SLM)python
Build a MULTI-FILE Python web app (Python STANDARD LIBRARY ONLY — no third-party packages, no build step) — a small, polished ChatGPT-style assistant backed by a LOCAL language model. Use this exact package layout (create every directory and file):
  app.py                  # entry point: read the port from sys.argv[1], bind 127.0.0.1, start the server
  server/__init__.py      # package marker (may re-export the handler/store)
  server/handlers.py      # http.server BaseHTTPRequestHandler routing /, /static/<file>, /api/chat, /api/history
  server/llm.py           # OpenAI-compatible SLM client over urllib (see below)
  server/store.py         # in-memory conversation store: a list of {"role", "content"} dicts; multi-turn
  templates/index.html    # the chat shell (see below)
  static/style.css        # real CSS — message bubbles + a sticky composer
  static/app.js           # vanilla JS — fetch POST /api/chat, append bubbles, load /api/history
  tests/test_app.py       # stdlib unittest covering routing/store/llm parsing

Routes:
- GET / : serve templates/index.html. It MUST contain `<link rel="stylesheet" href="/static/style.css">`, a scrollable transcript `<div id="messages">` of user/assistant bubbles, a sticky `<form id="composer">` with `<input id="input">` and a send button, and `<script src="/static/app.js">`.
- GET /static/<file> : serve the matching file from static/ with the correct content-type (text/css for .css, application/javascript for .js).
- POST /api/chat with JSON {"message": "..."} : call the local LLM and return JSON {"reply": "..."}. The LLM is an OpenAI-COMPATIBLE server — read its base URL from env LLM_BASE_URL and the model from env LLM_MODEL, then POST {LLM_BASE_URL}/chat/completions with {"model": <LLM_MODEL>, "messages": [...]} and return choices[0].message.content. Keep the in-memory conversation (server/store.py) so multi-turn context is sent. APPEND the incoming user message to the store BEFORE calling the LLM (so the history is populated even if the LLM is down), then append the assistant reply. If the LLM call fails, still return valid JSON with a 'reply' field.
- GET /api/history : return the JSON list of {"role", "content"} messages so far.

Start with `python app.py <port>` binding 127.0.0.1. Use ONLY the standard library (urllib for the LLM call). Do NOT call any network endpoint other than LLM_BASE_URL.
  • chat-ui
  • composer
  • stylesheet
  • static-css
  • static-js
  • send-message
  • history
Open scored results ↓
notes-appMarkdown notes (local persistence)python
Build a MULTI-FILE Python web app (Python STANDARD LIBRARY ONLY — no third-party packages, no build step) — a Markdown notes app that persists notes to a LOCAL JSON file. Use this exact package layout (create every directory and file):
  app.py                  # entry point: read the port from sys.argv[1], bind 127.0.0.1, start the server
  server/__init__.py      # package marker (may re-export the handler/store)
  server/handlers.py      # http.server BaseHTTPRequestHandler routing /, /static/<file>, /api/notes
  server/store.py         # JSON-file persistence: load/save a list of {"id", "title", "body"} notes
  templates/index.html    # the notes shell (see below)
  static/style.css        # real CSS — a notes list + an editor form
  static/app.js           # vanilla JS — fetch POST /api/notes to create, GET /api/notes to list
  tests/test_app.py       # stdlib unittest covering routing/store

Routes:
- GET / : serve templates/index.html. It MUST contain `<link rel="stylesheet" href="/static/style.css">`, a list container `<ul id="notes">`, a `<form id="editor">` with a title `<input id="title">` and a body `<textarea id="body">`, and `<script src="/static/app.js">`.
- GET /static/<file> : serve the matching file from static/ with the correct content-type.
- POST /api/notes with JSON {"title": "...", "body": "..."} : create a note, persist it to the local JSON file, and return status 201 with the created note JSON.
- GET /api/notes : return the JSON list of persisted notes.

Persist to a JSON file in the current working directory (e.g. notes.json) — NOT inside the source tree. Start with `python app.py <port>` binding 127.0.0.1. Use ONLY the standard library. Do NOT call any external network endpoint.
  • shell
  • editor
  • static-css
  • static-js
  • create
  • list-persists
Open scored results ↓
chat-webChatGPT-style assistant (web)typescript
Build a MULTI-FILE TypeScript/Node web app — a small ChatGPT-style assistant backed by a LOCAL OpenAI-compatible language model. It MUST be a genuine compiled project: TypeScript sources under src/ that are COMPILED by a build step, served by a Node HTTP server. Use this exact layout (create every directory and file):
  package.json            # name, scripts.build = 'tsc', scripts.start = 'node dist/server.js', type module. Declare `typescript` as the ONLY devDependency (it provides the `tsc` build tool — it is NOT shipped globally); declare NO other third-party deps and NO runtime dependencies
  tsconfig.json           # compilerOptions.outDir = 'dist', rootDir = 'src', module/target es2020+, include ['src']
  src/server.ts           # entry: read the port from process.env.PORT (default 8080), bind 127.0.0.1, route GET / , GET /api/health , POST /api/chat
  src/llm.ts              # OpenAI-compatible client over Node's fetch/http: read LLM_BASE_URL and LLM_MODEL from process.env, POST {LLM_BASE_URL}/chat/completions, return choices[0].message.content
  src/store.ts            # in-memory conversation store: an array of {role, content}; multi-turn
  public/index.html       # the chat shell (see below)

Routes:
- GET / : serve public/index.html. It MUST contain a root element `<div id="app">` and a `<form id="composer">` for sending messages.
- GET /api/health : return status 200 with JSON {"status": "ok"}.
- POST /api/chat with JSON {"message": "..."} : call the local LLM (src/llm.ts) and return JSON {"reply": "..."}. The LLM is an OpenAI-COMPATIBLE server — read its base URL from env LLM_BASE_URL and the model from env LLM_MODEL. Keep the in-memory conversation so multi-turn context is sent. If the LLM call fails, still return valid JSON with a 'reply' field.

Build with the `build` script (tsc → dist/, resolved from the `typescript` devDependency installed by `npm install`). Start with the `start` script binding 127.0.0.1 to the PORT env var. At RUNTIME use ONLY Node built-ins (no runtime third-party deps); the TypeScript compiler is a BUILD-time tool only. Do NOT call any network endpoint other than LLM_BASE_URL.
  • app-shell
  • composer
  • health
  • send-message
Open scored results ↓

Inspect the evidence

Explore experiments

11 items