Code Quality & Reliability
App: Python Course · Date: September 1, 2026
Scope: the built-in Python interpreter, the authored curriculum content (2194+ exercises, 48 lessons, 17 modules, a new «Lifehacks» section — 15 cards, 6 languages), the spaced-repetition scheduling algorithm, and — new in this report — whether the test suite itself actually catches regressions (mutation testing) across the 5 most critical logic modules
In short: two separate questions, two answers. Is the code correct? 9,710 candidates checked against CPython and other oracles — 0 discrepancies. Do the tests actually protect that? 3,694 deliberately injected defects (mutation testing — coverage grew along with the codebase) — all 3,694 caught. The second question is new to this report — it's what makes the first one trustworthy rather than just asserted.
Mutation Testing — Do the Tests Actually Catch Bugs
A coverage percentage only answers "did this line execute at least once," not "would a test notice if this line broke." Mutation testing answers the second question directly: a tool deliberately injects thousands of small, individual defects into the code (flips < to <=, && to ||, removes a bounds check, and so on) and checks whether at least one test then fails. If no test catches the injected defect, that's a real gap — even when line coverage itself reads 100%.
| Module | Mutants | Caught | Score |
|---|---|---|---|
Lesson-unlock gate logic (gate-engine) |
208 | 208 | 100.00% |
Achievements, XP, streak, hearts (gamification-engine) |
267 | 267 | 100.00% |
Spaced-repetition scheduler (srs-engine) |
47 | 47 | 100.00% |
| Python interpreter (tokenizer, parser, evaluator — including module 17's import/file code) | 3,123 | 3,123 | 100.00% |
«Lifehacks» section (lifehacks-engine + manage-lifehacks) |
49 | 49 | 100.00% |
| Combined, 5 most critical modules | 3,694 | 3,694 | 100.00% |
"Caught" here means a mutant either turned at least one test red, or caused an infinite loop/hang (also counted — the defect was still unambiguously flagged, just by a different signal; the Python interpreter row above counts exactly one such case this way). Every individual case where a mutant initially survived was reviewed by hand: either a new test was written that documents the real behavior, or it was mathematically proven that the specific code change can never affect the outcome at all (and in that case, the line itself carries a comment explaining why, not a silent pass), or — where that turned out to be the sharpest answer — the code itself was simplified so the unsupported branch of logic simply stopped existing. One such case is walked through in detail below.
Methodology — Code Correctness (CPython Cross-Check)
Three independent tracks, sharing no code and with no dependency between results. Interpreter: the app runs a custom, from-scratch Python interpreter (not CPython, not a third-party library) — every reference solution, template-generated instance, and randomly generated program was run simultaneously through the app's interpreter and a system-installed CPython 3.12.3, then compared for an exact result or error-consistency match. Curriculum structure: a full, non-sampled pass over every content file. Spaced-repetition scheduler: its output checked against a separately written implementation of the same formula. Validation ran through a temporary, isolated tool kept outside the app's own code, in several independent rounds run without referencing each other's results in advance.
Results Checked Against an Oracle
| Validation track | Checked | Matched | Discrepancies |
|---|---|---|---|
Interpreter — curriculum content and extended practice mode (reference + alternative solutions, fill-the-blank value-mode, hardcore static + parameterized templates, module 17's real import/open() execution) and the «Lifehacks» section's before/after code snapshots |
393 | 393 | 0 |
| Interpreter — randomized fuzz testing (20 construct categories) | 400 | 400 | 0 |
| Spaced-repetition scheduler (1,020 scenarios, 8,917 review steps) | 8,917 | 8,917 | 0 |
| Combined | 9,710 | 9,710 | 0 |
Structural & Content Checks
| Check | Coverage | Issues found |
|---|---|---|
| Curriculum structure (solution validity, prerequisite-graph integrity, id uniqueness) | 2,194 exercise files, 48 lessons, 17 modules | 0 |
| Extended practice-mode pool structure | 64 pool entries (53 static + 11 templates) | 0 |
| «Lifehacks» section (every card references a real curriculum concept; before/after code produces identical output on the real interpreter) | 15 cards, 30 code snapshots | 0 |
| Localization parity (content string keys, no untranslated placeholders) | 12,214 keys × EN, ES, FR, DE, PT (61,070 comparisons) | 0 |
Tools
strict, noUncheckedIndexedAccess, exactOptionalPropertyTypes.How These Numbers Were Actually Reached — No Gloss
These figures aren't where things started. The first mutation-testing pass on the largest, hardest module (the Python interpreter, ~2,500 lines) came back at 75.65% — roughly a quarter of the deliberately injected defects went completely unnoticed. Closing that gap to 100% took seven consecutive passes, each one closing part of the remainder while surfacing new edge cases — including methodology mistakes made along the way, during the checking process itself.
Real application bugs found and fixed (not just test gaps) — surfaced specifically by this thorough process: lesson scores were being wrongly deflated after a corrected mistake (one wrong answer capped the maximum possible score, even if the learner immediately got the retry right); the placement test was wrongly granting the "Flawless Run" and "No Mercy" achievements for lessons that were skipped, not genuinely completed; and three narrow interpreter bugs (integer division, remainder-operator rounding, bool-vs-int comparison).
One methodology mistake was caught and corrected mid-process: an intermediate verification approach (replacing an entire line of text instead of mutating at Stryker's exact coordinates) overcounted how many mutants were truly "equivalent" (i.e., unfixable) — of 35 such assumptions, only 1 held up. Fixed by switching to precise, coordinate-level verification for every individual case.
The last 5 mutants (string comparison used by list sorting) resisted both a proof of equivalence and a test written through the normal public interface — JavaScript's own sort algorithm makes redundant comparisons that mask a single wrong pairwise result. Resolved by testing the internal comparison function directly, bypassing the third-party sort algorithm's unpredictable behavior entirely.
Bottom line: no number in this report is painted on. Every one is the result of iterative, verified work — including admitting and fixing our own mistakes along the way. That's exactly why these figures can be trusted: the app doesn't just behave correctly on the examples that were checked — the test safety net around it genuinely catches future regressions, rather than just formally existing.
Why "2,797 of 2,797, 100%" briefly became "3,373 of 3,374, 99.97%" before returning to 100%: after first reaching 100% (above), the interpreter gained new curriculum topics — direct iteration (for x in iterable: without range()), __str__/__eq__, @staticmethod/@classmethod, dicts/sets/comprehensions/generators — hundreds of new lines that had never been through mutation testing at all. Rather than leave that growth unchecked (and quietly keep showing a stale "100%"), it was put through Stryker just as rigorously as everything else: 32 new regression tests, each written against one specific surviving mutant (first reading the code around it, confirming logically why the existing tests missed it, only then writing the test) — not coverage padding. Everything else that changed since the previous mutation-testing pass (August 17) was checked too: the parser (new grammar for those same new topics) and the spaced-repetition scheduler (a new pre-interleaving threshold) — both already at 100% with no further work needed. That left exactly one mutant unresolved, and this is where the report briefly read "3,373 of 3,374, 99.97%" with an explanation of why that's more honest than a manufactured "100%": a local-variable-lookup function (`Scope.tryGet`) had a check that, in theory, could matter in two different scenarios, but one turned out to be mathematically impossible given every actual call site in the app — and the other, genuinely reachable scenario, was already covered by a dedicated test. Marking both mutants on that line "equivalent" with a single comment would have been the fastest path to "100%" — which is exactly why it wasn't done at the time: the tool can only disable mutation-checking for a whole line, not one specific mutant, so that shortcut would have swept away the coverage for the scenario that does matter too.
What changed after that — and why this is genuine 100%, not a workaround: instead of settling for 99.97% or quietly suppressing the remainder, the next question was whether that line was dead code in the first place. A closer trace of every call to `tryGet()` in the app (there is exactly one) and every place the value it was guarding could ever originate showed: yes — the line was genuinely unreachable in any real program state, not merely hard to test. It was removed outright. This isn't a shortcut: deleting truly dead code loses no real coverage (unlike suppressing the mutant with a comment, which would have thrown away the sibling mutant's real, already-verified coverage too) — the mutant that line used to generate simply stopped existing along with the line itself. The full test suite (1,035 tests at the time) confirmed behavior didn't change by a hair. The next Stryker run came back at a genuine **100.00%**. The lesson from this small detour in the report itself: being willing to show an honest number just under 100% turned out to be the faster route to a real 100%, not a slower one than papering over the gap from the start.
Where the growth from 3,371 to 3,694 and from 9,655 to 9,710 came from (September 1, 2026): the course grew — a new 17th module, "Import, Modules, and File Handling," added real (not just syntax-recognition) execution of import/open()/with against a simulated virtual filesystem, and a new "Lifehacks" section (15 "before → after" cards explaining why the shorter form works the same). Both additions went through the same discipline as the rest of the app: interpreter.ts and parser.ts were re-run through mutation testing (+238 and +36 mutants respectively, all caught), the new lifehacks-engine and manage-lifehacks were brought from scratch to 100% (2 real test gaps closed, not masked), all 15 cards (30 code snapshots) were checked against CPython, and module 17's 2 new loop-based exercises were too. Separately, the same day closed the last translation backlog: the Lifehacks section and all 434 new module-17 keys were translated into Spanish, French, German, and Portuguese right away, so all 6 course languages are now at 100% content parity (12,214 keys each) — which is why the localization-comparison count grew from 58,635 to 61,070.
Notes
During iterative validation, three narrow interpreter bugs were found and fixed — none ever affected real curriculum content, and all three are now locked in place by permanent tests that make sure they never resurface:
- whole-number division printing as an integer instead of Python's actual decimal-typed result (
11instead of11.0); - a rare floating-point rounding artifact in the remainder (
%) operator on fractional operands; boolnot comparing as numerically equal tointin==/!=/in/list.remove().
The spaced-repetition scheduler's maximum-interval cap correctly engaged in several hundred steps designed to reach it, with no exceptions. The results above are from the final, most recent pass, run with no known open issues.