Prompt engineering helps, but context engineering usually matters more once AI is part of repeated operator work. If your sessions are bloated, your instructions are vague, and your outputs have no durable structure, a better prompt will not save you. The right move is to design the working environment around the job instead of trying to wordsmith your way out of a systems problem.
A lot of AI advice is pointing people at the wrong problem.
Teams keep trying to fix a systems problem with better wording.
They polish the prompt.
They tweak the phrasing.
They ask for one more rewrite.
Meanwhile the session is bloated, the instructions are vague, and the work has no structure. So the model drifts anyway.
A RevOps lead will spend 20 minutes polishing the perfect prompt for a pipeline review...
Meanwhile the model is sitting in a giant chat thread full of half-finished ideas, old assumptions, stale context, and three different tasks jammed together.
Then the output comes back muddy.
And the conclusion is usually that the model isn't good enough.
I don't think that's the real problem.
The problem is usually the setup around the model.
I ran into this building an attribution system in HubSpot.
The first job was simple enough - figure out which fields the system actually needed to look at so we could update first-touch and last-touch attribution cleanly.
But the session got polluted.
We were debugging field logic, auditing the portal, thinking through edge cases, and half-designing the final workflow all in the same conversation. The output got more and more complicated. It wanted to keep checking 10 fields. It split the work into 4 separate automations. Field checks everywhere. Extra conditions we didn't need. Token burn for no reason.
Once we stopped, took the learning from the debugging pass, and restarted with fresh context, the path was obvious. We already knew the fields that mattered. The clean version only needed 6 prescribed fields, and the workflow dropped from 4 automations to 2. No repeated audit step. No dragging the debugging logic forward. Just get to work with the right fields and update the attribution cleanly.
Why better prompts stop helping
A prompt matters.
It just usually matters less than people think once the work gets real.
If you're doing repeated operator work - pipeline reviews, call summaries, CRM audits, follow-up drafting, lifecycle cleanup, meeting prep - the output quality is mostly being shaped by four things:
- what context the model already has
- what junk it is still carrying from the last task
- whether the task is scoped tightly enough to do well
- whether your rules live somewhere durable or only inside that one prompt
That's why the same model can look smart in one workflow and sloppy in another.
It isn't random.
You changed the working conditions.
I've seen versions of this show up in CRM work more than once. A team wants AI help with sales-call follow-up, action items, and pipeline notes inside HubSpot. So they throw the transcript, a few instructions, and a big ask into one long-running chat. First pass looks decent. Then they keep going. They add another transcript. Then a manager coaching request. Then a scoring idea. Then a request to draft the rep follow-up email.
By the fourth or fifth turn, the output starts blurring together.
The action items are too generic.
The scoring logic gets inconsistent.
The manager notes start repeating the transcript instead of pulling out the part that matters.
Nothing "broke" exactly. The context just got sloppy.
The attribution example was the same pattern in a different form.
The first pass was useful because we needed to debug what fields mattered.
The second pass should have been a fresh execution workflow built around that answer.
Instead, if you keep everything in one thread, the model keeps dragging the debugging baggage forward. So a workflow that should just update first-touch and last-touch attribution starts carrying a bunch of audit logic and field checks that no longer belong there.
That is how you end up checking 10 fields when 6 will do, and building 4 automations where 2 are enough.
That's not intelligence. That's residue.
Why long AI chats get worse over time
Most people treat one giant conversation like momentum.
A lot of the time it's contamination.
When a session gets too long, a few things start happening.
Some context is no longer useful, but it is still hanging around. Some useful context gets compressed. Different goals start colliding with each other. Planning, execution, editing, and random side quests all end up in the same drawer.
That matters more than people realize. The research points the same direction. Liu et al.'s "Lost in the Middle" study (Stanford and Princeton, 2023) showed that even models with large context windows degrade sharply when the relevant information sits in the middle of a long conversation. The model's attention gets worst exactly where most operator work ends up over a long session: buried under other tasks.
If I'm working on a client pipeline review, I do not want the model still leaning on old assumptions from a content draft, a positioning discussion, and a totally different analysis from an hour ago. That's not helpful memory. That's carryover.
This is why I keep coming back to a simple pattern...
Plan in one context.
Execute in another.
Write the handoff down.
Then move.
That sounds obvious, but it changes the quality of the work fast.
A clean execution context with a tight handoff will usually beat a "smart" prompt inside a dirty conversation.
What context design actually means
Context design is just the discipline of deciding what the model should know, what it should ignore, and what should persist between tasks.
Not forever. Not in some mystical "second brain" way.
Just enough to make the work hold up.
If you want the practical version, I think it comes down to six decisions.
- What is the actual job for this session?
- What context is required for that job?
- What context should stay out?
- What rules should live outside the prompt?
- What output needs to survive as an artifact?
- When should this session end so the next one can start clean?
That is context design.
In practice, that usually means:
- durable instructions live outside the prompt
- project context has a home
- one task has one session
- handoffs become artifacts, not memory
- examples are specific
- constraints are named
So instead of writing a giant prompt every time you want a sales transcript analyzed, you define the system once.
Maybe the transcript-analysis workflow always needs:
- the ICP definition
- the call scoring rubric
- the CRM field mapping
- what counts as a real next step
- what should become a rep action item vs a manager coaching note
That stuff should not be reinvented in every prompt.
It should live in the workflow.
If you want the adjacent layers around this, read AI Memory Layer for Workflow Automation: What It Is and Why It Matters for the memory architecture and How to Set Up a Claude Code Project Without Overbuilding It for the workspace setup.
Then the prompt can stay simple because the system is doing the heavy lifting.
Prompt engineering vs context design
| Category | Prompt engineering | Context design |
|---|---|---|
| Main goal | get one better answer | make repeated work hold up over time |
| Lives where | inside the current prompt | in files, rules, project structure, and handoff artifacts |
| Best for | one-off asks | recurring operator workflows |
| Failure mode | clever wording, weak system | too much context, bad boundaries |
| What compounds | almost nothing | reusable instructions and cleaner execution |
This is the part people miss.
Prompting is usually about one turn.
Context design is about the operating environment.
One giant chat vs scoped sessions
| Workflow style | What happens |
|---|---|
| One giant chat | tasks bleed together, stale assumptions stick around, output drifts |
| Scoped sessions | each task has a clear job, clear inputs, and cleaner output |
| No handoff artifact | the model has to remember too much |
| Written handoff artifact | the model can execute against something stable |
I don't think teams need to become obsessive about this.
But they do need to stop pretending the prompt is the whole game.
If a workflow matters, the context around it needs design.
Signs your context setup is broken
- the output gets worse the longer the conversation goes
- the same instruction has to be repeated every session
- the model mixes strategy work with execution work
- different tasks start sounding the same
- the AI forgets the real constraint and defaults to generic advice
- the work feels better after starting a fresh session than after improving the prompt
That last one is usually the tell.
If "new chat" works better than "better prompt," you're not looking at a prompting problem.
The minimum context layers that actually matter
Most teams do not need a huge system on day one.
They usually need four things.
- a stable place for reusable instructions
- a stable place for project context
- a clear boundary between planning and execution
- a small number of real workflows worth reusing
That's enough to get serious results.
You can add more later if the work justifies it.
But if you build a giant AI operating system before one useful workflow works, you built theater.
Where prompts still matter
Prompts still matter.
They just are not the main event.
A good prompt is still useful for:
- framing the exact task
- naming the output format
- calling attention to the constraint that matters most right now
- telling the model what kind of judgment you want
But that's different from expecting the prompt to compensate for a bad operating setup.
It won't.
A beautiful prompt inside a messy workflow still gives you messy output.
The better model
I think the better mental model is this:
The prompt is the request.
The context is the working environment.
The artifact is the memory.
The workflow is the leverage.
If the working environment is bad, the request won't save you.
So before you rewrite the prompt again, check the system around it.
- Is this one job or three jobs mashed together?
- Did the useful context survive, or is the session carrying residue?
- Should this rule live in the workflow instead of inside the prompt?
- Does the next step need a fresh execution pass instead of one more turn in the same chat?
That's a more useful checklist than most prompt advice.
That's why I keep ending up in the same place with AI tools. The teams getting the most out of them are usually not the teams with the fanciest prompts. They're the teams that got specific about the job, the context, the constraints, and where the output is supposed to go next.
That's less exciting than prompt magic.
It's also why the good setups keep compounding while the bad ones keep turning into demos.
Frequently asked questions
Is prompt engineering still useful?
Yes.
It's useful for shaping a task inside a good system.
It's a weak substitute for the system itself.
Why does AI output get worse in long conversations?
Because context gets crowded, stale, and mis-scoped. Different tasks start colliding. Important details get compressed or diluted.
What is context design in practice?
It means deciding what should persist, what should be task-specific, where instructions live, and when to start a fresh session instead of dragging old baggage forward.
Do I need Claude Code to do this well?
No.
This shows up in Claude Code, ChatGPT, Codex, Cursor, and basically any serious AI workflow. The tool changes. The operating problem doesn't.
You can see versions of the same argument in Ruben Hassid’s anti-prompting framing and Hannah Stulberg’s writing on long AI sessions and context compaction.
What should live in instructions versus in the prompt?
If it applies repeatedly, it probably belongs in instructions or project context. If it only matters for this one task right now, put it in the prompt.
Practitioner view
"A beautiful prompt inside a messy workflow still gives you messy output. Before you rewrite the prompt again, check the system around it."
Sebastian Silva, Founder, HigherOps
Key takeaways
- Prompt engineering still matters, but it stops being the main lever once AI is part of repeated operator work. Context design takes over.
- Long AI chats compound residue, not memory. Different tasks crowd each other out and output quality drops over time.
- Context design answers six questions: the job, the required context, what to keep out, what rules live outside the prompt, what artifact to preserve, and when to end the session.
- The prompt is the request. The context is the working environment. The artifact is the memory. The workflow is the leverage.
- If a fresh session beats a polished prompt, the problem is the setup, not the wording.
Bottom line
If your AI workflow keeps getting worse, stop rewriting the prompt for a second.
Look at the working conditions.
That is usually where the problem is.
And it is usually where the fix is too.