Prompts — why the same question gets different answers (LLM nondeterminism and prompt sensitivity, with practical fixes)

The prompt that was perfect yesterday gives an odd answer today. The cause is usually one of two things, and their remedies are opposites. One is server-side randomness you can't touch; the other is a problem with the prompt you wrote.

---

Background / why this topic now

Use generative AI for just a few days and you will inevitably run into this. You ask the same question twice and get different answers. You paste in the prompt that worked yesterday and the tone of the output has changed. You and a teammate enter the exact same sentence and get different answers.

Many users shrug it off with 'that's just how AI is', but the story changes once you start attaching AI to real work. If the answer wobbles every time, you cannot set review criteria, you cannot trust an automated pipeline, and you cannot reproduce a problem when one occurs. If the earlier installment on hallucination was about 'AI being wrong', this one is about 'AI being inconsistent'.

Since 2025, the causes of this phenomenon have been substantially identified.

To state the conclusion first, the wobble has two distinct causes, and the user can control only one of them. This article covers how to tell them apart and how to respond.

---

Key data and current state

• Results diverge even at temperature 0: Running the same prompt 1,000 times produced 80 different outputs; the first 102 tokens were completely identical, then they diverged at the 103rd (Thinking Machines Lab, 2025)
• API models wobble more: At temperature 0.0, gpt-4o-mini produced differing outputs in about 24% of runs, while locally run llama3.1-8b did so in about 9% (arXiv 2601.19934, 2026)
• Changing only the format flips performance: Keeping the content the same and changing only the prompt's formatting (separators, line breaks, etc.) produced accuracy differences of up to 76 points(Sclar et al., ICLR 2024)
• Even model rankings flip: On the same task, varying only the format made per-model performance swing by about 10 points, and Qwen1.5-7B's accuracy moved between 22.9% and 51.2% depending on format (Hugging Face, 2024)

? Interpretation: The third and fourth figures matter most. That scores spread by as much as 76 points when only the format, not the content was changed means much of the basis on which we believe 'I wrote a good prompt' may in fact have been a lucky format. Put the other way, when an answer is bad, fixing the format before tearing apart the content may be the faster remedy.

---

In-depth analysis

1) Two kinds of 'different' to distinguish first

This is the most important distinction in the article.

Aspect	Randomness (Nondeterminism)	Sensitivity
Condition	The same prompt, run again	A slightly different prompt
Cause	Sampling plus server infrastructure	Prompt design and format
Controllability	Almost no user control	Mostly user-controllable
Response	Fix the format, run repeatedly, judge statistically	Revise and standardize the prompt

Most user frustration comes from mixing these two up. Blame server-side wobble on the prompt and you waste time wrestling with something that cannot be fixed; write off a prompt problem as 'that's just AI' and you neglect something that can be.

2) Randomness ① — a structure built on 'drawing' from the start

A language model does not settle on a single next word. It assigns probabilities to all the possible candidates and then draws one. The values that control the nature of this draw are temperature and top-p.

• The higher the temperature , the more room for low-probability candidates to be drawn → diverse but unstable
• The closer temperature is to 0 , the more only the highest-probability candidate is chosen → consistent but monotonous

Most chatbot services default to a non-zero value so answers don't become stiff. In other words, the environment ordinary users work in is set up from the start to give a different answer each time.

3) Randomness ② — the real reason it wobbles even at temperature 0

This is the part that was only recently uncovered. With temperature at 0, it always picks the top candidate, so the result should be fixed, but in practice it isn't.

According to the analysis Thinking Machines Lab published in September 2025, the main cause is not GPU randomness or floating point itself but 'the breakdown of batch invariance' . Servers bundle requests from many users and process them together, and the size of this bundle (batch) varies with the server load at that moment. But the compute kernels change the order of additionsdepending on batch size, and floating-point arithmetic gives slightly different results when the order changes.

Normally this tiny difference is negligible. The problem arises when the probabilities of the first- and second-ranked candidates are nearly tied. The moment a sub-decimal error flips the ranking, the entire rest of the sentence takes a different path. In an actual experiment, running the prompt 'Tell me about Richard Feynman' 1,000 times produced 'Queens, New York' 992 times and 'New York City' 8 times, and the answers diverged from that point.

Author's view: This is the most important point in practice. It means the reason my answer wobbled may be other people's traffic hitting the server at that moment. However much you polish the prompt, this part cannot be removed on the user's side. The research team achieved full reproducibility by rewriting the kernels to be batch-invariant, but had to accept a performance hit of about 2x. Speed and reproducibility are in a trade-off.

 

4) Sensitivity — a single comma changes the score

If randomness is the uncontrollable domain, sensitivity is exactly the opposite.

A study presented at ICLR 2024 measured performance while changing only the format of the prompt, leaving the meaning intact . The changes were at the level of switching a separator from a colon to a dash, moving a line break, or adding a space. The result was accuracy differences of up to 76 points on LLaMA-2-13B, and the researchers concluded that 'even comparing performance across models is not valid with a single fixed format' (Sclar et al., 2024).

A follow-up study measuring sensitivity when the question is rephrased with the same meaning (ProSA, EMNLP Findings 2024) goes a step further. Sensitivity varied greatly by task: largest on math and reasoning tasks and small on simple common-sense questions.

Adding just one example (zero-shot → one-shot) also sharply reduced sensitivity.

 

5) The invisible variables

Beyond the two above, there are things users believe are 'the same conditions' that have actually changed.

• Accumulated conversation context: If you have been chatting for a while in the same window, all of the earlier content is included in the input. It is not the same question.
• System prompt and model updates: Service providers sometimes quietly change internal instructions or model versions. There is no guarantee that yesterday's model and today's are the same.
• Whether search or tools were used: The evidence differs depending on whether web search was attached, and search results themselves change moment to moment.
• Personalization settings and memory: With personalization on, the input differs from person to person. This is usually why your answer differs from a teammate's.

---

Practical countermeasures — 7 ways to handle the wobble

① Diagnose the cause in 3 minutes first

Before blindly fixing the prompt, check in this order.

1. Open a new conversation windowand run the same prompt 3 times .
2. If all 3 go in a similar direction → a sensitivity problem. Fixing the prompt will solve it.
3. If the 3 go in different directions → a randomness problem. However much you fix the prompt, it will never be fully tamed. Manage it with fixed formats and repeated runs.

② Always validate in a new conversation

When testing a prompt, always use an empty conversation window. Success in an existing window may be thanks to that conversation's context, and neither other people nor tomorrow's you will be able to reproduce it.

③ Nail down the output format

This is the most cost-effective measure. Forcing the output structure noticeably reduces variance in results. In the Hugging Face experiment, too, applying structured output reduced the standard deviation across prompt variations (2024).

"Output the result only in the format below.

1. Conclusion (one sentence)
2. Evidence (3 items, one sentence each)
3. Uncertain points. Do not add any other explanation or introduction."

 

④ Include at least one example

In the sensitivity research, the most dramatic improvement came when the number of examples went from 0 to 1. Attaching just one sample of the desired output greatly reduces the wobble. One well-chosen example is often more effective than ten lines of instructions.

 

⑤ Manage prompts as 'assets'

Save prompts that worked in full in a notes file, along with the model name and date used. When a model is updated, the same prompt's performance can change, and without a record you will never know what changed. If you use the API, fix and record the temperature and seed values as well.

 

⑥ Use the wobble as a 'diagnostic tool'

The more important the question, the more deliberately you should run it 3 or more times. The parts that come out the same every time are areas the model is confident about; the parts that change every time are areas where the model is effectively guessing. In other words, where it wobbles is where you need to verify. It is the same principle as the cross-checking in the earlier hallucination installment.

 

⑦ Not every task needs reproducibility

For tasks like ideation, generating copy candidates, or producing several draft versions, the wobble is actually an asset. The realistic approach is to separate tasks that require reproducibility (calculation, classification, extraction, summarization) from tasks that require diversity (creation, ideation).

---

Implications

The biggest concern of companies adopting generative AI was surveyed as 'generation of incorrect information and lack of confidence in results' (61.3%) (Megazone Cloud and Foundry, 2025), and in practice this 'confidence' item appears to mix in a good deal of consistency problems alongside accuracy. The typical scene of an internal rollout running aground is often not 'it gave a wrong answer' but 'the answer changed every time, so it couldn't get through the approval chain'.

It is also worth noting that under the AI Basic Act that took effect in 2026, disclosure of AI use and putting management systems in order are emerging as tasks. Reproducibility of results and prompt history management may become foundational material for future internal controls and audit responses.

Author's view: The most effective measure an organization can take right now is not adopting a higher-performance model but building a 'prompt repository' . Simply accumulating prompts that worked in a team-shared document, with model name and date, rather than in personal notes, reduces variance in results for the same work and makes it possible to trace the cause of failures. The cost is effectively zero.

---

Outlook and variables to watch

• Improvement variable — reproducible inference: Technologies that guarantee determinism, like batch-invariant kernels, may be offered as an option in commercial services. Given the reported performance loss of about 2x, however, selective application to areas that need audit and regulatory responses seems more likely than blanket adoption.
• Risk variable — error accumulation in agents: In structures where AI executes multiple steps on its own, small variations at each step can be passed to the next step's input and amplified. A wobble that is negligible in a one-off conversation can lead to entirely different results in a 10-step task.
• Checkpoints: ① The extent to which major APIs offer seed and determinism options ② whether structured output (enforced JSON schema) becomes the default ③ whether Korean benchmarks on prompt sensitivity in Korean are published

---

Conclusion

1. There are two kinds of 'different', and diagnosis comes first. Running a prompt 3 times in a new conversation window mostly tells you whether it is server-side randomness or a problem with your prompt.
2. Temperature 0 does not guarantee reproducibility either. Changes in batch size due to server load change the results, and this is an area the user cannot control.
3. Focus on what you can control. Test in a new conversation, fix the output format, include at least one example, keep a prompt record: these four are most of the consistency an individual can get. And rather than trying to eliminate the remaining wobble, it is better to use it as a signal telling you where verification is needed.

References

• Defeating Nondeterminism in LLM Inference — Thinking Machines Lab (2025)
• Quantifying non-deterministic drift in large language models — arXiv 2601.19934 (2026, preprint)
• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design — Sclar et al., ICLR 2024
• ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs — EMNLP Findings 2024
• Improving Prompt Consistency with Structured Generations — Hugging Face (2024)
• The enforcement of the AI Basic Act and its implications — Law Times