Check Paired Coverage Before Comparing Image Model Trials in Node.js

A folder of successful images can hide an uneven experiment: one model may have a result for every planned case while another is missing several. Before reviewing image quality, build a coverage report that keeps successful, failed and absent observations separate. This Node.js example checks paired trial records; it does not score images.

1. Define the comparison before collecting results

Use two model labels, an ordered list of case IDs and a repeat count. Each case-and-repeat slot expects one terminal observation from each model. The labels demo-A and demo-B below are fictional; they do not identify or rank real providers.

Keep each case definition in a versioned brief outside this small schema: prompt text, input assets, output target and the controls actually available on each route. A case ID is only a reference to that brief. Matching IDs cannot prove the prompts, references or generation settings were equivalent.

Assign repeat numbers before running a batch. Repeat 1 is a planned slot, not a claim that the same random seed means the same thing across models. Keep unsuccessful attempts visible so a review does not silently compare only the easiest surviving results.

2. Index terminal observations without overwriting duplicates

Save the following as paired-coverage.mjs. JSON.stringify builds a tuple key from case ID, repeat and model label; Map holds one terminal status per key. Rejecting duplicates prevents a later rerun from silently replacing the earlier observation. Record reruns in a new declared trial batch.

const text = value => typeof value === 'string' && value.trim().length > 0;
const key = (caseId, repeat, model) => JSON.stringify([caseId, repeat, model]);

export function pairedCoverage(plan, rows) {
  if (!plan || !Array.isArray(plan.models) || plan.models.length !== 2 ||
      !Array.from(plan.models).every(text) || new Set(plan.models).size !== 2 ||
      !Array.isArray(plan.cases) || plan.cases.length === 0 ||
      !Array.from(plan.cases).every(text) ||
      new Set(plan.cases).size !== plan.cases.length ||
      !Number.isSafeInteger(plan.repeats) || plan.repeats < 1 ||
      plan.repeats > 20 || plan.cases.length * plan.repeats > 2000 ||
      !Array.isArray(rows)) throw new Error('Invalid plan or rows');
  const models = new Set(plan.models);
  const cases = new Set(plan.cases);
  const index = new Map();
  for (const row of Array.from(rows)) {
    if (!row || !models.has(row.model) || !cases.has(row.caseId) ||
        !Number.isSafeInteger(row.repeat) || row.repeat < 1 ||
        row.repeat > plan.repeats || !['ok', 'failed'].includes(row.status)) {
      throw new Error('Invalid observation');
    }
    const id = key(row.caseId, row.repeat, row.model);
    if (index.has(id)) throw new Error('Duplicate observation');
    index.set(id, row.status);
  }
  const pairs = [];
  for (const caseId of plan.cases) {
    for (let repeat = 1; repeat <= plan.repeats; repeat++) {
      const [left, right] = plan.models.map(model =>
        index.get(key(caseId, repeat, model)) ?? 'missing');
      pairs.push({ caseId, repeat, left, right,
        comparable: left === 'ok' && right === 'ok' });
    }
  }
  const pairedSuccess = pairs.filter(pair => pair.comparable).length;
  return { models: [...plan.models], plannedPairs: pairs.length,
    pairedSuccess, blockedPairs: pairs.length - pairedSuccess, pairs };
}

The example accepts only ok and failed as recorded statuses. An omitted observation becomes missing in the report. A pending job is not a terminal result and is rejected. The limits of 20 repeats and 2,000 planned pairs are local application limits for this example, not model or platform limits.

Array.from makes sparse arrays explicit during validation, so holes are rejected instead of being skipped. Identifiers are matched exactly. Whitespace-only identifiers are invalid; otherwise the code preserves spelling rather than normalizing different IDs into one.

3. Make the missing slot visible in a small fixture

Save this synthetic batch as trials.json. There are two cases and two repeats: four planned pairs, requiring eight observations for a fully recorded batch. Seven observations are present.

{
  "plan": {
    "models": [
      "demo-A",
      "demo-B"
    ],
    "cases": [
      "bottle",
      "poster"
    ],
    "repeats": 2
  },
  "rows": [
    {
      "model": "demo-A",
      "caseId": "bottle",
      "repeat": 1,
      "status": "ok"
    },
    {
      "model": "demo-B",
      "caseId": "bottle",
      "repeat": 1,
      "status": "ok"
    },
    {
      "model": "demo-A",
      "caseId": "bottle",
      "repeat": 2,
      "status": "ok"
    },
    {
      "model": "demo-B",
      "caseId": "bottle",
      "repeat": 2,
      "status": "ok"
    },
    {
      "model": "demo-A",
      "caseId": "poster",
      "repeat": 1,
      "status": "ok"
    },
    {
      "model": "demo-B",
      "caseId": "poster",
      "repeat": 1,
      "status": "failed"
    },
    {
      "model": "demo-A",
      "caseId": "poster",
      "repeat": 2,
      "status": "failed"
    }
  ]
}

Both bottle repeats have ok on both sides. For poster repeat 1, demo-A is ok and demo-B failed. For poster repeat 2, demo-A failed and demo-B has no observation. A missing result and a recorded failure remain distinct even though neither pair is ready for image comparison.

The report therefore has plannedPairs 4, pairedSuccess 2 and blockedPairs 2. Filtering the input to successful rows before building the report would erase the failure evidence. Always pass the complete terminal record set alongside the original plan.

4. Add a local command with useful exit codes

Save this as audit.mjs beside the module. It reads only the local JSON file. Exit 0 means every planned pair has two ok records; exit 1 means at least one pair is blocked; exit 2 means the input or command is invalid.

import { readFile } from 'node:fs/promises';
import { pairedCoverage } from './paired-coverage.mjs';

try {
  if (process.argv.length !== 3) throw new Error('Usage: node audit.mjs trials.json');
  const input = JSON.parse(await readFile(process.argv[2], 'utf8'));
  const report = pairedCoverage(input.plan, input.rows);
  console.log(JSON.stringify(report, null, 2));
  process.exitCode = report.blockedPairs === 0 ? 0 : 1;
} catch (error) {
  console.error(error.message);
  process.exitCode = 2;
}

Run the synthetic batch with this command. Its report lists the two bottle pairs first, then the two poster pairs, regardless of observation order. The expected exit code is 1 because two planned pairs are blocked.

node audit.mjs trials.json

For an all-ok fixture, add or replace terminal records in a separate copy so all eight planned observations are ok; the command should exit 0. A JSON object without a plan should exit 2. Preserve the original fixture so the missing-versus-failed example remains reproducible.

5. Exercise the cases that can distort coverage

The accompanying local verification ran 31 synthetic checks on Node.js 24.20.0. They covered the four planned slots, the two paired successes, the two blocked pairs, distinct failed and missing statuses, observation-order independence, input immutability, empty observations and an all-success batch.

Validation checks covered duplicate observations, undeclared models and cases, repeat zero, fractional or out-of-plan repeats, pending or literal missing statuses, invalid model lists, empty or duplicate case lists, local size limits, sparse arrays, null plans and non-array rows. A case with prototype-looking identifiers also passed because the lookup uses Map entries.

Independent CLI runs confirmed exit 0 for the complete batch, exit 1 for this incomplete fixture and exit 2 for invalid input. These checks exercised record accounting only. No provider was called, and no image was generated, downloaded, decoded or visually reviewed.

6. Use paired success as the start of review

The comparable field means only that both terminal records say ok for that planned slot. Before judging the images, verify that each recorded success points to the intended artifact and that the versioned case brief and controls match the experiment you meant to run. This module intentionally does not inspect artifacts or enforce those external facts.

Review the paired-success set with a declared task rubric, and report blocked pairs alongside it. Do not call the pairedSuccess fraction an image-quality score or infer a model winner from this fictional batch. The report answers which planned pairs have two successful records, while preserving the rest of the denominator.

AI assistance: This tutorial and code were drafted with AI assistance and checked with local synthetic tests. The model labels, cases and observations are invented examples. No real model benchmark or image-quality evaluation was performed.

Source: Adapted as an independent developer tutorial from PixMind’s Krea vs Midjourney vs Flux comparison. The original comparison prompted the experiment-accounting workflow; its dated prices, model rankings and capability claims are not reproduced here.

JavaScript reference: MDN Map.