Decode a Blind Style Review in Node.js

Suppose you are testing whether adding a reference image helps an image series keep a target style. Show reviewers the same pair of outputs as left and right, then keep the assignment key separate until voting closes. If the reference output sometimes appears on the left and sometimes on the right, counting left votes directly gives the wrong result.

This tutorial builds the decoder that runs after a review. It turns side choices into variant counts, preserves ties and skips, and tells you which share of the planned pairs received no vote.

Set up one ballot per pair

Keep the subject, prompt, and other settings aligned within each pair. Write down the style criterion before voting, such as whether the palette and texture match a shared target. Use opaque pair IDs in the review interface if labels could reveal the assignment.

For this small example, each planned pair has one designated reviewer and one recorded vote. The assignment table stores the variant on the left, and the variant on the right follows from that. Keep this table private until the review closes. A system with multiple reviewers would need a different key and an aggregation rule decided before voting.

The decoder trusts the recorded side assignment. Record it from the same data the interface uses to render each pair. Reconstructing sides later from filenames or display order makes it easy to swap them.

Keep three kinds of indecision separate

A tie means the reviewer inspected both outputs and found neither better on the chosen criterion. A skip means they explicitly did not score the pair, for example because a preview failed to load. A missing vote means the decoder never received a row for that pair.

Keep these buckets separate so the counts can be reconciled with the original plan. Treat a duplicate vote or an unknown pair as an input error. Decide what happened before rerunning the report.

Implement the decoder

Save this as decode-review.mjs. Use a Map to look up the side assignment and a Set to detect repeated votes. Validate the vote choice before incrementing the corresponding total.

export function decodeReview({ assignments, votes }) {
  if (!Array.isArray(assignments) || !Array.isArray(votes)) {
    throw new Error("assignments and votes must be arrays");
  }

  const leftByPair = new Map();
  for (const { pairId, left } of assignments) {
    if (typeof pairId !== "string" || !/^[a-z0-9-]+$/.test(pairId)) {
      throw new Error("Invalid pair ID");
    }
    if (left !== "baseline" && left !== "reference") {
      throw new Error(`Invalid assignment for ${pairId}`);
    }
    if (leftByPair.has(pairId)) {
      throw new Error(`Duplicate assignment for ${pairId}`);
    }
    leftByPair.set(pairId, left);
  }

  const outcomes = { baseline: 0, reference: 0, tie: 0, skip: 0, missing: 0 };
  const seen = new Set();
  for (const { pairId, choice } of votes) {
    if (!leftByPair.has(pairId)) throw new Error(`Unknown pair ${pairId}`);
    if (seen.has(pairId)) throw new Error(`Duplicate vote for ${pairId}`);
    seen.add(pairId);

    const left = leftByPair.get(pairId);
    const right = left === "baseline" ? "reference" : "baseline";
    if (choice === "left") outcomes[left]++;
    else if (choice === "right") outcomes[right]++;
    else if (choice === "tie") outcomes.tie++;
    else if (choice === "skip") outcomes.skip++;
    else throw new Error(`Invalid choice for ${pairId}`);
  }

  outcomes.missing = leftByPair.size - seen.size;
  return { planned: leftByPair.size, ...outcomes };
}

Run the example

Save this entry point as review.mjs in the same directory. It reads local JSON, writes the summary to standard output, and exits with status 1 on invalid input.

import { readFile } from "node:fs/promises";
import { decodeReview } from "./decode-review.mjs";

try {
  const [path, ...extra] = process.argv.slice(2);
  if (!path || extra.length) throw new Error("Usage: node review.mjs review.json");
  const manifest = JSON.parse(await readFile(path, "utf8"));
  console.log(JSON.stringify(decodeReview(manifest)));
} catch (error) {
  console.error(error.message);
  process.exitCode = 1;
}

Save the following synthetic fixture as review.json. The two portrait votes favor the same variant even though one says right and the other says left.

{
  "assignments": [
    {"pairId":"portrait-1","left":"baseline"},
    {"pairId":"portrait-2","left":"reference"},
    {"pairId":"product-1","left":"baseline"},
    {"pairId":"product-2","left":"reference"},
    {"pairId":"scene-1","left":"baseline"},
    {"pairId":"scene-2","left":"reference"}
  ],
  "votes": [
    {"pairId":"portrait-1","choice":"right"},
    {"pairId":"portrait-2","choice":"left"},
    {"pairId":"product-1","choice":"left"},
    {"pairId":"product-2","choice":"tie"},
    {"pairId":"scene-1","choice":"skip"}
  ]
}

Run the command below from that directory.

node review.mjs review.json
# {"planned":6,"baseline":1,"reference":2,"tie":1,"skip":1,"missing":1}

Report the counts with their denominators

In the example, the reference variant won two of the three decisive votes. Across the six planned pairs, the complete report is two reference wins, one baseline win, one tie, one skip, and one missing vote. A compact headline about decisive votes should carry the full count beside it.

Six synthetic pairs illustrate the bookkeeping. If results vary by subject or reviewer, show those splits before making a broader claim. A missing result may matter more when it comes from a subject that is difficult to judge.

Before applying the decoder to review data, check three cases against known answers. Reverse a pair's sides and the vote together, submit a duplicate row, and leave an assigned pair without a vote. Those cases exercise the mapping, rejection, and missing-vote rules.

Sources and disclosure

Workflow background comes from PixMind's repeatable style reference test, which compares reference and baseline variants across subjects and seeds. The ballot format in this tutorial is a local example.

See the Node.js readFile reference for the file-reading API.

AI disclosure. AI assisted with drafting this article. The Node.js example was run on synthetic review records. No images were generated or reviewed for this tutorial.