Self-Consistency Prompts: How Majority Voting Works

I use self-consistency to find the most common final answer - not the best-sounding explanation. If 6 out of 10 valid responses give the same answer, that’s a 60% majority. It still needs checking.

Here’s my process:

• Choose a clear-answer task and define which answers count as the same.
• Test 5–20 responses using one fixed prompt and sampling settings that allow variation.
• Extract and standardise final answers, then count valid votes. Report invalid responses, ties and cases with too few valid votes separately.
• Check the leading answer independently. Shared mistakes can produce agreement; decisions affecting people need human review.
• Compare against one response, measuring checked accuracy, cost in £ and total time.

The rule I keep in mind: agreement is not proof. I use voting only when testing shows that the accuracy gain is worth the extra cost and time - not for every task.

::: @figure {Self-Consistency Prompts: From Samples to Checked Answers} :::

Self-Consistency Prompting: Majority Vote Over Multiple Reasoning Paths in LLMs.

1. Choose tasks with clear answers

Use majority voting only when every response can be reduced to one comparable final answer.

Check which tasks suit voting

Majority voting needs final answers that can be put into the same form. A task may suit voting without becoming more accurate as a result.

Run a small pilot before increasing the number of samples. If several answers could be valid, clarify the task or define which answers count as equivalent first. Similar wording alone doesn't make two answers the same.

Define when answers count as the same

Set comparison rules before sampling. Specify decimal precision or numerical tolerance, require units, and allow conversions only if they preserve the value.

For text, decide which spelling variants count as equivalent. For structured labels, use an approved label list. Map variants only to an explicitly agreed label, and keep separate categories apart. Flag missing or unrecognised answers rather than guessing what they mean.

Score only the final answer; use the explanation only to extract it. Different reasoning paths can lead to the same result. Evaluation on maths tasks such as MATH and AIME24 focuses on answer accuracy, not whether the reasoning paths match [1]. Keep results with different meanings separate.

Once you've fixed the equivalence rules, sample several runs and count the final answers.

2. Sample responses and count answers

Set the prompt and answer format

Once you’ve fixed the answer-equivalence rules, lock the prompt and output format. Use the same task prompt, constraints, model version and sampling settings across samples. Request a single structured final field, such as Answer: ..., and parse ONLY that field.

Set sampling variation and limits

Use non-zero temperature, where supported, to introduce variation. Repeating deterministic outputs adds little. Treat 5–20 samples as a test range, not a default, and evaluate temperature, top-p and sample count for each task. Allow enough output tokens for the final-answer field.

Samples from one model are not independent: they can repeat the same error.

Measure total cost and elapsed time per task. Parallel requests can cut waiting time, but they won’t reduce token costs. Keep sampling settings fixed within each voting batch so you can compare configurations on equal terms.

Extract, normalise and count final answers

Parse the final field, apply your agreed comparison rules and count each valid answer once.

20, 20. and twenty count together only if your agreed numeric parser accepts all three as the same value; otherwise flag unsupported forms.

Mark missing, truncated or unparseable fields as invalid. They don’t count as votes.

counts = {}
records = []

repeat sample_count times:
    response = sample(prompt, fixed_config)
    parsed = parse_final_field(response)
    answer = normalise(parsed, agreed_rules) if parsed.valid else INVALID
    records.append(response, parsed, answer, fixed_config)
    if answer != INVALID:
        counts[answer] = counts.get(answer, 0) + 1

if counts is empty:
    result = NO_VALID_ANSWERS
else:
    leaders = answers_with_highest_count(counts)
    result = leaders[0] if length(leaders) == 1 else TIE

save(records, counts, result)

The most frequent answer is the mode, but it may have only a plurality - not a majority. It’s the leading answer, not proof that the answer is correct. Report ties explicitly rather than picking an answer arbitrarily. If every sample is invalid, return no answer.

Record valid and invalid counts, vote counts, raw responses, extracted answers and the configuration, including temperature, top-p and model version. Use the vote counts to assess agreement, then verify the winner separately.

3. Assess agreement and check correctness

Report majorities, pluralities and ties

After counting valid votes, classify the run as a majority, plurality, tie or inconclusive. Report invalid responses separately.

A majority means the leading answer has more than half the valid votes. A plurality means it leads without passing that threshold. If the top answers tie, report every tied leader - don’t force a single winner.

If too few valid votes remain to classify the result with confidence, mark it as inconclusive. Treat a majority as a confidence signal, not proof: the answer still needs an independent check.

Investigate disagreement and shared errors

When answers differ, check the prompt for missing constraints, ambiguity or a false premise. Several responses can agree because they share the same error.

Low agreement should prompt a review of the prompt or further checks - not automatic acceptance of the minority answer. If you revise the prompt, assess the new batch separately.

Verify high-stakes outputs independently

Once you have a leading answer, check it using methods suited to the task. Use deterministic checks for calculations, primary evidence for factual claims, schema validation for structured outputs and tests for code.

Schema compliance confirms format, not factual correctness. A second model can flag inconsistencies, but agreement between models is not independent evidence.

For decisions that affect people, require human review. Give reviewers the underlying evidence and check results, not just the winning answer.

Conclusion: Measure accuracy, cost and latency

Before deployment, check that tasks have clear answers, sampled responses vary, and equivalent answers normalise to the same final form.

Once the voting pipeline works, compare it with a single-response baseline on representative tasks. Measure verified accuracy, cost per task (£) and end-to-end latency, including verification.

Self-consistency is one control within AI evaluation and risk management - not proof of correctness. Deploy it where the measured accuracy gain justifies the extra cost and delay.

FAQs