`PROMPTFOO_PASS_RATE_THRESHOLD` takes a **percentage, not a fraction** — so writing `0.9` when you meant 90% gives you a **0.9%** threshold, and CI goes green with half your tests failing. Nothing warns you. This episode builds the eval gate that makes swapping a model a check instead of a vibe: one YAML file, one `echo` provider, and an exit code you can read. Plus three finds, two tips, and a ten-minute thing to do today. No git this episode — episodes 3, 4 and 5 lived in worktrees and hooks, and this one deliberately leaves them. ## In this episode - **Kimi K3 (Moonshot AI)** — 2.8T-parameter MoE, 104B activated per token, open weights, 1M context, natively multimodal (text, images and video). Weights went public **Monday 27 July 2026**. **The licence is the story.** Plenty of coverage carried "Modified MIT" forward from K2. It isn't. K2's file is titled *Modified MIT License*; K3's is titled *Kimi K3 License*, and **MIT is never named in it**. **Section 2** (MaaS revenue over **$20M across any consecutive 12 months** → separate agreement with Moonshot; and MaaS is defined narrowly, excluding models embedded in a specific product feature and excluding mere relaying to models hosted by others) and **Section 3** (**$20M per month** in revenue or 100M MAU → display "Kimi K3" in your UI) are **different units and must not be conflated**. **Section 4** exempts internal use and use via Moonshot's own products or certified inference partners — but "internal use" is defined as **not making the software, its outputs, or its underlying capabilities available to third parties**, which is narrower than "inside your own company". Weights are free; Moonshot **names no price** for the separate agreement, which is not the same as free. - **DeepSeek-V4-Flash-0731** — the official release, superseding the preview. `0731` is a **build date, not a version**. **304B parameters, read from the repo's own safetensors index**, not from press coverage. Active parameter count and context length are **not stated** on the model card page fetched, so the episode doesn't state them. **Licence: MIT** — verified from the repo's licence tag and the README. Weights free; **DeepSeek API pricing was not checked this run**. - **DwarfStar / `antirez/ds4` (Salvatore Sanfilippo, creator of Redis)** — **name mismatch: the repo and binary are `ds4`, the README calls it "DwarfStar"**. GitHub *repository* search for "DwarfStar" won't surface it (the name is in neither the repo name nor the description) — you'll get other people's wrappers instead. Search the path, `antirez/ds4`. A small native local inference engine tuned first for DeepSeek V4 Flash — and specifically the **0731 build**: the README names it repeatedly and ships `DeepSeek-V4-Flash-DSpark-support-0731.gguf`. Also GLM 5.2 and, on very-high-memory machines, DeepSeek V4 PRO. Metal / CUDA / ROCm. **MIT**, free, no paid tier named anywhere in the README or LICENSE. README credits **Georgi Gerganov** for the llama.cpp / GGML work. **Caveats in the maintainer's own words: "beta quality", DSpark "experimental" and opt-in, and — on 128 GB MacBooks — PRO q2 streaming "experimental but usable for inspection".** Metal's primary target is **Macs with 96 GB+**; smaller machines get `--ssd-streaming`. **Created back in May — ~20k stars and climbing, so trending, not new.** Its README also credits GPT 5.5, 5.6 and Claude Fable with "strong assistance", humans leading ideas, testing and debugging. - **The Recipe — put a pass/fail gate on your prompts before you change models.** promptfoo 0.122.0, **MIT — verified from the installed `package.json` and npm's registry entry.** (Neither OpenAI's post nor promptfoo's names a licence at all; OpenAI's says they'll "continue building the open-source project", which is a claim about openness, not a second confirmation of MIT.) **OpenAI announced its intent to acquire Promptfoo on Monday 9 March 2026 — announced, not closed: the post is titled "OpenAI *to acquire* Promptfoo" and says closing is subject to customary conditions.** Founders **Ian Webster** and **Michael D'Angelo** — worth knowing that an OpenAI-bound tool grades every vendor's models. **Price: Community is "Free Forever" including red teaming capped at 10,000 probes/month; Enterprise and On-Premise both say "Custom", which means promptfoo names no price — not the same as free.** Node floor is **`>=22.22.0`**, read off the installed package; the GitHub Action states the same floor and recommends Node 24 LTS. - **The steps** — `npx promptfoo@latest init --no-interactive` (**plain `init` is a wizard** — with nothing attached to answer it, it exits **130** and writes no files) → get a **zero-cost green run first** using the `echo` provider (seven deterministic assertion types passed, $0 spent) → then swap in the provider you actually ship → gate CI on the exit code. - **The exit-code contract, measured rather than quoted** — all pass: **0**. At least one assertion fails: **100**. Typo'd provider id, missing config, missing API key: **1**. So "promptfoo returns non-zero" is true and useless — a shell check for `$? -eq 1` catches the broken setup and **silently ignores every failing test**. (The 100 is the default; `PROMPTFOO_FAILED_TEST_EXIT_CODE` can override it, so check inherited CI before assuming.) - **Tip 1 — the threshold unit trap.** `PROMPTFOO_PASS_RATE_THRESHOLD` is a **percentage**, and it's inclusive (`>=`). Against a rigged 50% pass rate: `60` → exit 100, `50` → exit 0, **`0.9` → exit 0**, because it means 0.9%. **Write `90`, not `0.9`.** Precisely what's silent: the terminal *does* still print `✗ 1 failed (50.00%)` — what you don't get is any threshold message (that line only prints when the gate trips) and, critically, the **exit code is 0**. A human sees the failure; the machine gating your deploy sees success. The counter-intuitive bonus: **misspelling the variable name is safe** — `PROMPTFOO_PASSRATE_THRESHOLD` is ignored, the default applies, and the gate still fires. The name is forgiving; the value is not. - **Tip 2 — a typo'd assertion `type:` counts as an "error", not a "failure".** It still exits 100, so CI catches it, but the summary reads `0 passed, 0 failed, 1 error` — so a dashboard counting *failures* shows a clean green bar while nothing ran. Count errors too, or trust only the exit code. - **The honest counter-argument** — Hamel Husain and Shreya Shankar's evals FAQ, aired rather than softened. Note the attribution: the sharpest line ("all you get from prefab evals is you don't know what they actually do… at worst they create an illusion of confidence that is unjustified") is the FAQ **quoting one of their instructors, Eleanor Berger** — the authors' own version is "generic evaluations waste time and create false confidence". Theirs directly: "if you're passing 100% of your evals, you're likely not challenging your system enough", and you should be "extremely skeptical of this process being delegated". The harness is the easy part; the assertions are not. - **Stated on air** — none of this was run against a live frontier model this week. Everything was verified on Windows / Node 24.18.0 with the `echo` provider, so latency, cost accounting and `llm-rubric` model-graded behaviour are **not verified**. Also disclosed: **Claude Code requires a paid Claude plan (Pro, Max, Team, Enterprise, or API billing) — nothing on the free tier.** - **Do This Today** — one folder, one `promptfooconfig.yaml`, two test cases (one passing, one failing), then check the exit code. **You're looking for 100, not 1.** ## Sources **Fresh Finds** - [moonshotai/Kimi-K3 — Hugging Face](https://huggingface.co/moonshotai/Kimi-K3) - [Kimi K3 LICENSE](https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE) - [Kimi-K2-Instruct LICENSE (for the "Modified MIT" contrast)](https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/main/LICENSE) - [Simon Willison — Kimi K3](https://simonwillison.net/2026/Jul/27/kimi-k3/) - [deepseek-ai/DeepSeek-V4-Flash-0731 — Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) - [DeepSeek-V4-Flash-0731 — HF model API (parameter count)](https://huggingface.co/api/models/deepseek-ai/DeepSeek-V4-Flash-0731) - [antirez/ds4 — "DwarfStar"](https://github.com/antirez/ds4) - [ds4 README](https://github.com/antirez/ds4/blob/main/README.md) - [ds4 LICENSE](https://github.com/antirez/ds4/blob/main/LICENSE) **The Recipe** - [promptfoo — getting started](https://www.promptfoo.dev/docs/getting-started/) - [promptfoo — command line reference (exit codes)](https://www.promptfoo.dev/docs/usage/command-line/) - [promptfoo — the `echo` provider](https://www.promptfoo.dev/docs/providers/echo/) - [promptfoo — GitHub Action](https://www.promptfoo.dev/docs/integrations/github-action/) - [promptfoo — pricing](https://www.promptfoo.dev/pricing/) - [OpenAI to acquire Promptfoo](https://openai.com/index/openai-to-acquire-promptfoo/) - [Promptfoo — joining OpenAI](https://www.promptfoo.dev/blog/promptfoo-joining-openai/) - [Hamel Husain and Shreya Shankar — Evals FAQ](https://hamel.dev/blog/posts/evals-faq/) --- Sister show: **The AI Daily Wire** — the daily AI news brief from the same network. Different host, same standards. ## Built by - **Culmen Games** — a solo-built indie game studio. studiosculmen@gmail.com - **Handled** — done-for-you content and AI agent workflows for creators and time-poor professionals. handled.anmol@gmail.com