What 786 passing tests actually tell you.

At 10XU, I migrated more than 100 Cloud Functions from TypeScript to Dart. A differential test suite matched Firestore state in all 786 cases. Cold starts became six times faster.

Those are two different results. The speedup tells you about performance. The state comparisons tell you whether the new code preserved the behavior covered by the tests. You need both before calling a migration an improvement.

Compare what the code leaves behind.

A function can return the expected response and still write the wrong data. It might skip a field, create an extra document or increment a value twice. Checking only the response would miss that.

For this migration, the comparison was Firestore state. The implementation language changed; the stored result was the thing that needed to match.

This is the basic idea behind differential testing: give two implementations equivalent inputs, then compare an output that matters. The difficult part is choosing that output and collecting cases that exercise the behavior you rely on.

786 passing cases are evidence about 786 cases.

They do not prove every possible execution is correct. A test suite can miss an input, an interaction between operations, or a condition that only appears in production. A passing result is useful precisely when you can say what it covers.

State equality also has a blind spot: two implementations can agree on the wrong result. A differential suite catches changes in behavior. Tests against the product requirements are still needed to check that the shared behavior is right.

Try a failing case.

The interactive model on this site demonstrates the comparison with 786 illustrative JavaScript fixtures. These are sample cases, not the original migration tests.

Enable “Introduce an off-by-one bug” and run the comparison. Case 418 writes revision 3 where the reference writes revision 4. Select “Inspect the mismatch” to isolate those writes in 3D. Remove the bug and rerun: all 786 cases match again.