Achhina's Digital Garden

2026-07-16 Superpowers Plugin Evaluation Research

4 min read

2026-07-16 Superpowers Plugin Evaluation Research

What I set out to do

Figure out whether the superpowers plugin (installed via superpowers-marketplace, v6.1.1) offers any user-level configuration, and whether anyone has evaluated it against other plugins or bare model defaults.

What I actually did

  • Audited the installed plugin: no settings-file config surface at all. Only knobs are env vars (SUPERPOWERS_DISABLE_TELEMETRY, semi-internal BRAINSTORM_* server vars). The SessionStart <EXTREMELY_IMPORTANT> injection is unconditional; the sanctioned customization path is CLAUDE.md precedence over skills.
  • Sibling plugins have real env-var config: episodic-memory (EPISODIC_MEMORY_API_*, EPISODIC_MEMORY_CONFIG_DIR) and double-shot-latte (DOUBLE_SHOT_LATTE_MODEL/TIMEOUT).
  • Surveyed effectiveness evals:
    • AgentStackBench study (June 2026, 500 SWE-bench-style tasks): correctness 45.6% → 47.8% (not significant), +625k tokens and +74s per task.
    • MindStudio n=12 Claude Code test (May 2026): 9% cheaper, 14% fewer tokens, better quality on non-trivial tasks. Anecdote-tier sample.
    • Upstream’s own “drill” eval harness measures skill compliance, not effectiveness vs baseline; issue #1462 asking for effectiveness benchmarks was closed as too vague.
    • Broader area: SkillsBench shows curated domain-knowledge skills give +16.6pp average and that focused sets (≤3 modules) beat exhaustive bundles; LangChain’s eval showed 9% → 82% completion from domain skills.
  • Checked the already-installed alternatives (feature-dev, tdd-workflows): equally gate-heavy (“DO NOT START WITHOUT USER APPROVAL”), but opt-in per slash command rather than auto-triggered every session.

What was striking

  • The evidence splits cleanly by skill type: domain-knowledge skills benchmark strongly positive with token savings; process/methodology skills (superpowers’ category) benchmark neutral-to-negative on correctness with real token cost, since frontier models increasingly do the discipline by default.
  • The human-approval gates I find annoying are hardcoded prose in the SKILL.md files (“you MUST present it and get approval”, even for a config change); nothing toggles them.
  • SkillsBench’s focused-beats-bundle finding cuts directly against superpowers’ 20+ skill always-injected design.

Open decision

How to keep the parts I like (TDD, spec-then-implement) without the forced human checkpoints. Options on the table, deliberately not decided yet:

  1. Fork the 2-3 valued skills into nix-managed user skills with gates stripped; uninstall superpowers.
  2. Rely on already-installed opt-in plugins (/tdd-cycle, /feature-dev) when ceremony is wanted; drop superpowers.
  3. Distill everything into a few CLAUDE.md lines + maybe one short personal skill (CLAUDE.md already says “prefer TDD”).

Candidate experiment: disable superpowers for a week, run on defaults + existing CLAUDE.md + opt-in commands, and fork only what turns out to be missed.

Deep research results (same day, two parallel agents)

  • Upstream will not add an autonomy toggle: gate-skipping triaged as a bug (#1441); extensibility issue #1566 open with no commitment, filer concludes “maintain a fork”. No credible gate-stripped fork exists (closest: superpowers-optimized, stale; super-ralph, 9 stars, machine gates).
  • Gates are concentrated in ~3 skills (brainstorming 76% ceremony, writing-plans handoff, executing-plans). The liked skills (TDD, systematic-debugging, verification, subagent-driven-development) are already gate-free; SDD explicitly forbids pausing between tasks.
  • Distillation: of ~3,100 skill lines, ~1,200 are durable methodology. Fork verdicts: TDD (~300), subagent-driven-development (~420), systematic-debugging (~380), writing-plans (~120). Brainstorming and verification distill to CLAUDE.md lines; executing-plans and using-superpowers drop.
  • Pure CLAUDE.md insufficient for TDD: the failure mode is in-context temptation; the rationalization table and anti-pattern gate functions act at the point of temptation and do not compress (HumanLayer, TDD enforcement).
  • Alternatives ruled out as replacements: cc-sdd matches the target model (approve spec once, autonomous TDD execution) but small and slowing; compounding-engineering has autonomous /lfg but not strict red-green; ralph-loop (installed) is the unattended wrapper regardless.

Recommendation (pending my decision): fork-and-replace hybrid. ~24-line CLAUDE.md block (spec-then-implement autonomous, TDD sharpeners, verification, debugging 3-strike) + fork the 4 skills into nix-managed personal skills + uninstall superpowers (kills the SessionStart injection and the unshadowable gated brainstorming front door). ~95% of durable value at ~35% of lines, zero approval gates.

2026-07-09 Claude Settings Hardening and checkLinkTargets Root Cause

Graph View

Backlinks