Hypothesis backlog
A prioritized list of hypotheses, each grounded in specific evidence and scored on expected impact and confidence.
Optimization
Optimization is a loop, not a one-time project
Hypotheses tested at a sample size that can actually detect an effect, sequenced so you know which change caused which result.
Optimization gets sold as a checklist: change the button color, shorten the form, add a testimonial above the fold. Some of those changes help. Most of them do nothing measurable, and a few quietly hurt conversion while looking fine in a screenshot. The difference between optimization that compounds and optimization that just generates activity is discipline in three places: how the hypothesis is framed, how the test is sized, and how changes are sequenced so you can tell what actually caused a result.
A hypothesis is not 'let's try a shorter form.' A hypothesis states what you believe is causing the current behavior, what change should address that cause, and what metric should move if you are right. That framing forces you to articulate a reason before you spend budget on a test, and it means a failed test still teaches you something because you had a specific belief to disprove.
Test sizing is the part most teams skip, and it is the part that quietly invalidates the most experiments. A test run on too little traffic for too short a time will produce a result that looks decisive and is actually statistical noise. We calculate the sample size and duration a test needs to detect a meaningful effect before it launches, and we say upfront when a site does not have enough traffic to run a valid test on a given page, because running it anyway would just manufacture a false confidence.
Sequencing matters because most sites are changing more than one thing at a time: a paid campaign shifts, a new page launches, an email goes out, and a test is running underneath all of it. If you cannot isolate which change caused which movement, you cannot learn from any of it. Part of this engagement is deciding what gets changed when, so results stay attributable to something specific.
What it is
How the optimization loop actually runs
It is a repeating cycle, not a single sprint that ends with a report.
The loop starts with a prioritized backlog of hypotheses, each tied to a specific piece of evidence, whether that is a segment finding, a drop-off point in the funnel, or a pattern from session recordings. Hypotheses get scored on expected impact and confidence, not just picked because they are easy to build.
Before a test launches, we calculate the minimum sample size needed to detect the effect size we expect, given the page's current traffic and conversion rate. If the page does not get enough traffic to reach that sample size in a reasonable window, we either widen the test to a broader page set or recommend a different method entirely, such as a phased rollout with before-and-after comparison instead of a true split test.
During the test, we hold other variables on that page as steady as we can and track what else changed elsewhere on the site or in paid channels that could confound the result. This is where sequencing discipline pays off: a test that overlaps with an unrelated traffic surge or a pricing change becomes unreadable, no matter how well it was designed.
After the test concludes, the result feeds back into the backlog. A win gets rolled out and the next hypothesis in priority order gets picked up. A loss or a null result still gets documented, because knowing what does not move the needle narrows the search space for what will next time.
Fit
Who this is for, and who it is not for
We would rather say no early than sell a program that cannot work.
Right fit
- You have enough traffic to run statistically valid tests and want a disciplined process rather than ad hoc changes.
- Previous testing has produced conflicting or unconvincing results and you want the process itself examined.
- You want optimization treated as an ongoing program with a backlog, not a single redesign followed by silence.
Not the right fit
- Your traffic volume is too low to reach valid sample sizes in a reasonable timeframe. We will say so rather than run a test that cannot produce a trustworthy answer.
- You want every test to be a guaranteed win. Most well-designed tests still come back flat or negative, and that is expected, not a failure of the process.
- You need a one-time redesign rather than a recurring test-and-learn cycle. That is a different kind of engagement.
Deliverables
What the optimization program includes
A running backlog and a disciplined testing process, not a single batch of changes.
Test design and sizing
Sample size and duration calculated before a test launches, so results are read with statistical confidence instead of guesswork.
Change sequencing plan
A schedule for what changes when across the site and in paid channels, so results stay attributable to a specific cause.
Test tracking log
A record of every test run, its hypothesis, its result, and what it ruled in or out, so learning compounds over time.
Rollout of winners
Confirmed wins get implemented permanently and folded into the baseline before the next hypothesis is tested.
Cycle review
A recurring review of what the loop has learned so far and where the next round of hypotheses should focus.
How we run it
How the loop is run
Five steps that repeat, not a linear project with an end date.
Step 1: Source hypotheses from evidence
Every hypothesis traces back to a specific data point, whether that is a funnel drop-off, a segment finding, or user behavior data.
Step 2: Size and design the test
We calculate the sample size and duration needed to detect a real effect, and reject tests that cannot reach validity.
Step 3: Sequence and launch
The test is scheduled against other planned changes so its result will not be confounded by something unrelated happening at the same time.
Step 4: Read the result honestly
Wins, losses, and null results are all documented with the same rigor. A flat result is still information.
Step 5: Roll out and repeat
Confirmed wins get implemented, the backlog reprioritizes, and the next hypothesis moves into the design step.
Why are most conversion tests underpowered?
Most tests run on too little traffic for too short a time, which produces a result that looks decisive but is statistical noise. A valid test requires calculating the sample size needed to detect the expected effect size before launch.
How do you keep test results attributable to a single cause?
By sequencing changes so only one meaningful variable moves on a given page during a test window, and by tracking other site and campaign changes that could otherwise confound the result and make it unreadable.
Where this connects
Where the optimization loop connects
Hypotheses come from evidence elsewhere in the analytics pillar and feed back into it.
Hypotheses are sourced from segmented findings uncovered through analytics insights rather than guesses about what might convert better.
A test on a checkout or lead form depends on accurate conversion tracking to measure the outcome the test is actually trying to move.
Which tests get prioritized depends on which success metrics the engagement is judged on in the first place.
Results and open tests are surfaced on an ongoing basis through reporting dashboards so stakeholders can see the loop moving without waiting for a summary call.
When a hypothesis involves a landing page served from paid traffic, it often ties directly into retargeting and how that audience is treated after the first visit.
Questions
Strategy and optimization questions we get asked
Run tests that actually tell you something
Sized correctly, sequenced correctly, documented whether they win or lose.