The metric said 99%. The customers said otherwise.

Situation

Windows Hello, facial-recognition login for Windows, was reporting a 99% success rate. By that number, the feature was a triumph. But the feedback told a different story. Customers complained they couldn't log in, and a senior leader personally experienced repeated failures. Two truths can't both hold. Either the customers were wrong, or the metric was.

The metric was counting authentication attempts, not whether someone could actually log in. A user who failed four times and succeeded on the fifth registered as one failure and one success, mostly fine by the math. By the experience, that user just stood in front of their laptop being rejected four times.

Task

I was Group Engineering Manager over the data team at the time, a team I grew from 2 to 30 people, and had already designed a session architecture to understand customer usage as a whole engagement rather than a stream of disconnected events. Windows Hello's gap between its 99% and the negative feedback was the opportunity to prove that approach out. The job was to redefine what "quality" meant for the feature using aggregate, session-level modeling, and to show the team what that kind of measurement could reveal that event-level tracking couldn't. It was also to get an organization that liked its 99% to accept a much worse number, starting with a dev manager who didn't trust data going in, and ending with a leadership team that had already formed its opinion of the feature from personal, frustrating experience.

Action

The insight behind the session architecture was that no "session" existed in the raw data. Nothing connected a burst of attempts into one human experience of trying to log in. For Windows Hello, I applied that architecture directly. I defined the session as the whole engagement, lock screen to desktop, not any single attempt inside it. Then I designed the pipeline that would reconstruct those sessions from raw attempt telemetry, directing my engineers and partnering closely with my architect on the ETL implementation. On top of that I defined a happiness metric, the share of users who succeed on most of their attempts, session over session, a measure of whether the person was having a good experience rather than whether any single attempt succeeded.

That took a six-stage pipeline, detailed below, a durable instrument the team could steer by daily. A login isn't one component either. It spans hardware, the Windows security stack, and Hello itself, three subsystems, each built by a different org with its own priorities, and none of them reported to me. I worked across the Windows security team, the OEM teams supplying hardware telemetry, and the Hello team to pull the right signal out of each stack into the same session. That meant making the case, subsystem by subsystem, that a shared session was worth more to each of them than the local view they already had.

That got me a number. The Hello dev manager didn't trust the data and hadn't bought into the approach, and no amount of arguing was going to change that, so I stopped arguing. Every iteration, I added more signal and tied it to a specific failure the team could go fix, not just a lower percentage they were supposed to feel bad about. One iteration surfaced a timing race condition, cameras that weren't starting up fast enough for facial recognition to catch the signal. A failed attempt like that records as one line, indistinguishable from a hundred others; only the full session showed the gap. The dev manager fixed it, watched the number move, and started asking what else the data could show him. Then he asked again, and again, and the asks kept getting bigger, until one day it wasn't a question about data at all.

Would I be open to letting my data engineer join his team, and he'd give me a headcount to replace him.

Before I answered, I went to the engineer. I walked him through what the move would look like and asked whether he wanted it. He did. He was excited about it.

So I said yes, and I didn't say the rest of it out loud. The dev manager had gone from not trusting the number to spending his own headcount to keep the person producing it. Nothing I could have argued would have got me there.

With the dev team on board, I pointed the same architecture at the bigger problem. Windows leadership's own confidence in the feature was personal as much as political; several of them had hit failures themselves. I built a self-host dashboard that tracked every Windows leader's login on their own dogfood devices, drillable to the individual session. That flipped the team from reactive to proactive. Instead of waiting for a leader to complain, they could walk up with the answer already in hand. Then I presented leadership with the real number, about 65%, replacing the 99% they had been reporting. The dashboard behind it could show any one of them exactly why their own login had failed, session by session.

Architecture deep-dive optional, for the technically curious

The pipeline reconstructed human experience from a raw event stream in five processing stages, filter through publish, then fed a sixth stage, reporting.

raw Windows event stream

01

Filter

Reduce the full event stream to the events relevant to Hello.

02

Sessionize

Find start/end events; create session rows as the bookends.

out-of-order arrival

Events arrive non-deterministically across a distributed pipeline; an end event can land before its start, leaving incomplete sessions: start-only, end-only, or orphaned events.

03

Merge

Cross-hourly processing matches new events into incomplete sessions, completing them as more data streams in; once a session has both a start and an end, everything between them is a "complete" session, ready to process.

04

Metrics

Calculate the per-session metric set for each complete session.

05

Publish

Write session metrics to a SQL store, where the pipeline proper ends.

pipeline ends: consumption layer

06

Report

Two views off the same store: an aggregate happiness score tracked per user over time, and a per-session detail view developers use for diagnostics, since one session carries everything relevant across the entire stack.

Design notes and tradeoffs:

• Sessionization is the whole ballgame. Defining the bookends (what counts as a session's start and end) directly shapes the metric. We validated the rules against known-bad user reports rather than tuning for a flattering output.
• Complete versus incomplete is a real state the pipeline has to carry. Distributed telemetry doesn't arrive in order. Cross-hourly reprocessing exists specifically to keep completing sessions as their missing half shows up, rather than treating a late event as noise to drop.
• User-level, not attempt-level, aggregation. Averaging attempts hides distribution; a small population failing constantly disappears inside a large population succeeding. The pipeline defines happiness per user first, then aggregates it across users.
• Durable over clever. Publishing to SQL as a maintained pipeline, rather than a notebook analysis, is why the metric could catch a live anomaly at a hardware partner.
• The session is a diagnostic as well as a score. Once a login's full journey was reconstructed, root-causing a failure stopped being guesswork. A race condition in camera startup timing showed up only as a gap in the session, invisible at the level of any single failed event.

Result

• The real number was ~65%. A third of users were not reliably logging in. Leadership accepted it because the methodology was airtight and the metric matched what customers were saying.
• The team drove happiness past 90%, fixing failure modes the old metric had been hiding.
• The dev manager tried to hire my data engineer off me, and offered a headcount to backfill him. That is the one I'd point at. Windows leadership came along the same way, on a string of fixes the data pointed to that worked.
• The self-host dashboard turned reactive complaints into proactive fixes. The senior leader who'd been hitting repeated failures had a multi-monitor setup with his camera mounted off to the left, catching the wrong angle of his face. Session data pointed straight at it. We moved the camera to center and the failures stopped, before he'd even filed a complaint.
• It caught a design flaw before launch. The metric flagged an anomaly on a hardware partner's prototype before it shipped, traced to camera placement that put the layout where Hello couldn't reliably see a face.
• The session/happiness pattern spread. HoloLens and Windows Mixed Reality adopted the same approach to measuring experience quality, and Microsoft Edge picked it up too, all without anyone mandating it.

Learning

A metric without usage context can look accurate and still be wrong. It measures the event and misses the person having it. Before trusting any success metric, ask what a frustrated customer would look like in the data. If the answer is "invisible," the metric is a liability.

I stopped trying to win the argument and started handing over fixes that worked, one at a time, until doubting the data cost him more than trusting it did. Leadership came the same way. They signed off on a 65% because the dashboard behind it had already found their own failed logins before they'd reported them.

Neither of them changed their mind because I was persuasive. They changed it because the thing kept being useful to them.

---

All selected work