Blog
Completes over shown, not sends over the list
An 8,400-row ESP blast produced 48 answers and a 0.6% rate. The widget showed the same question 72 times; 48 people answered.
Tuesday, 09:40. Jordan owns CX at Northstar, a 40-person project-management SaaS. Survey product-nps v2. This week the widget sat on /projects after eight seconds on the board. It appeared 72 times. Forty-eight people scored. Twenty-four dismissed it. Results: NPS +22 (n=48), response rate 67%. Eleven of the 48 comments say the same thing: "CSV export still takes three clicks."
09:52. Growth exported 8,400 HubSpot contacts, filter Customer is true, last mailed in 2024. Same NPS, from the ESP. 8,400 sends, 6,120 delivered, 410 opens, 48 completes. Eleven of those 48 already answered the widget on Monday.
10:00. Weekly. Slide 4: "NPS response rate 0.6% (48 / 8,400). Industry benchmark is 12%. We need volume."
The weak version, and what it costs at 10:00
The program that produced that slide looks like this:
Denominator: the HubSpot list, or the send count. 48 / 8,400 = 0.6%.
The widget's 67% is "just in-app," so it does not go on the slide.
Forty-eight completes beat last week's 31, so the blast worked.
Next action: send 20,000 on Thursday.
The version that keeps Tuesday readable looks like this:
Widget: 48 answers / 72 shown = 67%. The 24 dismissals stay in the denominator.
Email: 48 completes over the people who opened the survey, not over 8,400 addresses that never saw the question.
The eleven people who answered twice are one person. They already earned silence.
Next action: read the CSV comments and the 24 dismissals. Do not buy more sends.
The weak version costs you Thursday. Twenty thousand more addresses, 91 completes, 0.45%. The room decides the program is dying. The widget is still 67% of the people who saw it. The 24 dismissals still say the moment is wrong. Nobody reads Monday's CSV comments because 0.6% looked like a delivery problem.
A show is the question on the screen
A response rate is completes over shown. Shown means the ask actually rendered: the person was on /projects, the delay elapsed, the widget appeared. Scoring it counts. Closing it without a score also counts. People who never saw it do not belong in the denominator.
UserVane's results view uses that population. Measured answers over measured answers plus dismissals. The headline prints N and the rate together, the same way it prints the plus/minus. That is the delivery question: of the people we interrupted, how many answered?
The measurement honesty contract names the rest: people who saw the ask and did not answer are not the same as people who never saw it. Non-response is part of the story. Hiding it in a mailing-list denominator is how 0.6% gets on the slide.
A send is not a show
An ESP send is a hope. Delivered is closer. An open is closer still. None of those is the question on screen. If the only number you have is the list, you are measuring list rot, deliverability, and whether Tuesday is a good day to mail, mixed in with whether the question is answerable.
The same trap shows up in support. A CSAT hosted link in every Zendesk close macro is not shown 140 times because 140 tickets closed. It is shown when someone opens the page. Tickets closed is an invitation count. Invitation count is the wrong denominator. The channel chooser is which surface fits. This post is what you divide by once you pick one.
If the same person already scored the widget on Monday, the close-macro link and the Thursday blast are a second ask. One answer earns silence. The quiet-period post is that rule. Counting them as two completes does not make 0.6% honest. It makes n look busy.
Imported history is not a show either
A Delighted CSV of 2,400 old rows can sit next to this week's 48 widget answers. Those 2,400 people were never shown this widget. Fold them into the numerator against 72 shows and the rate jumps past 100%, which is a visible lie. UserVane keeps imported history in the score, labeled as imported, and leaves it out of the rate. The rate is what this program did this week, on the asks it actually rendered.
Once the rate is real, size the sample
Forty-eight answers at 67% of 72 shown is a real week. It is still a small n. Check it on the sample size calculator before slide 4 treats +22 as the climate. The calculator wants the number of people who answered, not the 8,400 you mailed. Sending more without lifting the rate moves the interval not at all. The older post on why a naked score lies is the decision rule once the denominator is honest.
If 24 of 72 closed the widget, the work is the moment, the page, or the question, not the ESP. Read those 24 the same sitting you read the CSV comments. The weekly comment read is that hour.
What the rate does not prove
Completes over shown proves how the people who saw the question behaved. It does not prove those people represent the customer base. Anyone who never logged in, bounced before eight seconds, or sat on a page the widget does not use never entered the denominator. That is the limit.
Say it on the slide once. Then read the 48 and the 24. The 0.6% can go.
Bottom line: divide by the people who saw the question. Keep dismissals in that number. Leave the mailing list out of it. The 48 and the 24 are the week. The 8,400 are a list.