When an LLM Context Window Dropped Instructions: Why Your App Trimmed Its Own Rules
The answer looked like a model failure. The request builder caused it.
A user pastes a long document into an AI product and asks for an analysis. The reply engages with the document, but ignores the product's required output format and crosses a boundary the product is meant to enforce.
When an LLM context window dropped instructions in this manner, the team first checks the instructions in the codebase. They are still there. Short requests return the expected format. At first, it looks as though the prompt was too long and the model ignored rules that were clearly delivered.
In this composite case, that was not what happened.
The application built a request that was too large for its configured limit, then silently trimmed content before sending it to the provider. The model received plenty of the user's document, but not the product rules that governed the answer. This is a common context length failure in an LLM application, where the system prompt is cut off in production before the model even runs.
The important question was not, "Are the rules in the prompt file?" It was, "Did the final request sent to the provider still contain them?"
How to diagnose if an LLM context window dropped instructions
To diagnose whether your LLM context window dropped instructions or the model failed to follow them:
- Capture the exact payload leaving your request builder before it hits the API.
- Measure the token count of system instructions, conversation history, and user input separately.
- Verify whether automated string truncation removed the earliest tokens in the prompt.
- Check if the response matches an unconstrained model default rather than your structured template.
The team reproduces the failure
The team sends two inputs through the same product path:
- A short document
- A document long enough to put pressure on the request limit
The prompt text and model settings remain unchanged.
The short document produces the required format. The long document produces an answer that addresses the content but misses the expected format and boundaries. The difference points to request size. No prompt change explains it, and the user supplied the document directly, so a failed document lookup is not the cause.
That narrows the investigation, but it does not prove that instructions disappeared. When a prompt is too long and a model appears to have ignored rules, an oversized request can actually fail in three distinct ways:
- The provider can reject it before the model responds.
- The application can trim another part of the request and preserve its rules.
- The model can receive the rules and still fail to follow them.
Those are different failures with different fixes. Addressing them is a critical part of improving code reliability. The team needs to inspect request preparation: where the product counts content, what it does when the assembled request is too large, and what the provider actually receives.
What the application sent: System prompt cut off in production
In this case, the application assembled the request with product rules first and user content after them. Before the API call, it applied a length limit by trimming from the start of the assembled content.
That preserved the recent user text, including much of the pasted document, but removed the earlier product rules. This resulted in the system prompt being cut off in production without raising any runtime exception.
The provider received a request that fit the application's limit, so it did not reject it as oversized. The model answered the content it received. The missing format had a concrete cause: the instruction requiring it was absent from the outbound request.
The team confirmed this by comparing the rules the product intended to include with the request after trimming. Looking only at the answer would have left several explanations open. Inspecting the outbound request identified an application-side defect.
It also explained why the prompt looked correct in the repository while production behaviour differed, highlighting the wider verification gap between code generation and runtime execution.
Request tracking should make that comparison possible without turning sensitive documents into routine logs. Record:
- Estimated request size before and after request preparation
- Whether trimming occurred
- Which type of content was trimmed
- Whether each required rule block remained present
- A request identifier that links the preparation decision to the reported failure
Where engineers need to inspect content, use an appropriately controlled debugging path. Do not store users' pasted documents in ordinary logs simply to diagnose request length.
The conclusion is deliberately narrow. This incident does not show that models ignore long prompts in general. It shows that this application's trimming step removed its rules on this request path.
The fix protects the rules instead of rebuilding the product
The team changes request preparation rather than replacing the product.
First, the application checks length before making the API call. If user content will not fit alongside required instructions and the expected response, it no longer removes instructions silently to make room.
Next, the team shortens the essential rules and reserves fixed space for them in the request. Shorter does not mean weaker. The rules still define the required format and product boundaries, without background material that is unnecessary on every call.
The application then sets an explicit limit for user content. When an input exceeds that limit, it can return a useful message asking the user to shorten it. If the product supports trimming user content, that decision must be deliberate. When omitted content could change the answer, the product should tell the user what was excluded.
The length check covers more than the pasted document. A proper request budget includes:
- Product instructions
- Message formatting overhead
- User content
- Other messages sent on the path
- Space for the model's response
Token budgets are simply the available space for all of that content in one request. Exact counting depends on the provider and model, so the team verifies its calculation against the API it uses rather than relying on character counts.
Finally, the team reruns the long-input case and inspects the outbound request. The required rules are present, or the product returns the intended over-limit response before calling the model.
That verifies the application no longer causes this specific missing-rule failure. It does not guarantee that a model will follow every rule it receives. Model behaviour after receiving a rule is a separate problem and needs separate testing.
Lesson 1: Reserve space for rules users cannot consume
A request budget decides what gets space before user content arrives. Start with the context available for the selected model. Reserve room for the response, required instructions and message structure. The remaining space is the maximum user content the product can accept on that path.
Without that reservation, a long input can consume the space the product needs to behave as intended. Users will paste full files, long chat histories and material far beyond development examples. That content may be valid input, but it cannot have an unlimited claim on the request.
Separate must-follow product rules from optional background material. A rule that controls output format or a product boundary needs protected space. Examples, long explanations and supplementary context may help, but they should not compete with core rules for the final available tokens.
Write a clear trimming policy as well. Depending on the product, the application may remove older conversation turns or decline part of an oversized paste. It must not quietly remove the instructions that define the product's behaviour.
If excluding user content would make the answer misleading, reject the request with a clear explanation instead.
Trimming order is not a harmless implementation detail. Cutting from the start can remove rules placed first. Cutting from the end can remove the user's latest question. The right policy depends on the role of each part of the request. Make that policy explicit, then enforce it in the request builder.
Lesson 2: Test the request builder, not only the answer
Answer checks can reveal a missing format. They cannot tell you whether the model broke a rule it received or the application failed to send that rule.
Add tests where the application finishes assembling the outbound request. This step is critical across the entire agentic AI lifecycle as prototypes move into maintained software.
Run short, near-limit and over-limit inputs through the same path. Include pasted files if the product accepts them. Include long chat histories if the product carries conversation history forward. These cases exercise the content that grows while product rules remain fixed.
For accepted inputs, assert that:
- Required instruction blocks remain in the assembled request.
- The assembled request fits the budget, including response space.
- The application records any permitted trimming decision.
For over-limit inputs, assert the intended behaviour: a clear rejection or an explicit, permitted reduction of user content. The test should fail if the application removes required rules and sends the request anyway.
These checks do not require identical model wording. They verify the application's own decisions, which are direct and repeatable. A separate answer check can still test whether the response follows the rules that were sent.
Keep the investigation focused on request size and assembly. The prompt did not change in this case. The document came directly from the user. The defect sat between accepting that input and making the API call, so that is where the strongest check belongs.
Lesson 3: Make context length failures visible in production monitoring
Tests cover the cases a team anticipates. Production inputs will still expose awkward boundaries.
At minimum, request tracking should record:
- Estimated request size
- Whether the request was accepted or rejected
- Whether content was trimmed
- What type of content was trimmed
- Whether required rules were included in the outbound request
Those signals help an engineering lead separate similar-looking incidents.
If the application trimmed its own rules and sent the request, inspect the request builder. If the provider rejected an oversized request, inspect the length check and its assumptions. If the rules were present and the response still broke them, investigate model behaviour as a separate failure.
The same user complaint can point to three different engineering tasks.
Track decisions and sizes without routinely retaining sensitive user documents. When content inspection is necessary, use controlled access and follow the product's data-handling requirements. The goal is to establish what happened to a request, not to create another store of pasted files.
Before changing models or redesigning retrieval, review the failing request path. A record showing that the product removed its own instructions gives the team a narrow, testable fix and evidence that it holds at the boundary that exposed the problem.
Fix the failing request path before it becomes a recurring incident
Start with one reported failure. Inspect the request the application actually sent, including every length check and trimming step. Then reserve space for essential rules, set an explicit user-content limit, and test short, near-limit and over-limit inputs through the affected path.
This is a focused improvement to an AI product your team has already built. If senior engineering capacity is the blocker, Smicolon can help you inspect the request builder, implement the boundary and verify the fix without a rebuild.
Learn more about our senior engineering support, review subscription plans, or book a discovery call to discuss the failing path.