How we measure vendors

Methodology version 1.1.0 (25 October 2026). The code that implements this page is pipeline/grading/. Every number below is a named constant in pipeline/grading/params.py.

What we measure and why

When a serious vulnerability lands, people ask one question: is the vendor on it? Days to Patch answers it with timestamps. Each vendor gets a report card with two subjects, graded separately and never blended:

1. Response to exploited zero-days (the headline). When attackers used a bug before a fix existed, how quickly did the vendor:
   • publish an advisory
   • ship a first fix
   • ship fixes for every supported version Openness, meaning how easy the vendor makes it to follow along, adds a small part.
2. Private fix. How fast the vendor fixes bugs that researchers report to it privately.

Each subject has a score from 0 to 100, a letter from A+ to F, an 80% interval, a confidence level, and a provisional flag when the evidence is still thin. A subject is NR (not rated) when there is not enough evidence for it. A vendor can be rated on one subject and NR on the other.

Grades are absolute. Every score comes from fixed curves anchored to public norms, never from a ranking against other vendors. One vendor's grade never moves because another vendor got better or worse.

We do not grade how many bugs a vendor has. Its count of exploited bugs (CISA's KEV catalog) is shown beside the grades as context, with weight 0.

The five moments

An incident is one exploited vulnerability event at one vendor. We include every vendor CVE that CISA adds to its Known Exploited Vulnerabilities (KEV) catalog from 22 October 2024 onwards. That inclusion rule is objective, so nobody can pick which incidents count. For each incident we record up to five moments:

Every moment carries evidence: a URL, a verbatim quote, and a precision of minute, hour or day. Its origin is one of:

1. observed: our collector saw it, at a time we recorded ourselves
2. claimed: the date the source publishes
3. curated: a researcher found it, with a quote
4. inferred: derived by a documented rule

When several candidates exist for a moment, the earliest wins, and all of them are shown. Grades use exactly the moments the site shows. An automated hint that nothing corroborates never counts. A third-party post, news story or research write-up can start a clock only when a researcher has reviewed it, because anyone can post, and a post can be back-dated.

Only researcher-reviewed incidents count toward grades. Incidents assembled automatically from vendor feeds are shown on the site, labelled "awaiting review", and are not graded. This matters because automated timelines can make systematic mistakes, such as picking a later backport as the first fix. A reviewed incident has a curated file with sources and quotes for every moment.

Which incidents the response clocks use

Exploited bugs fall into three groups. Of the 144 reviewed incidents in the current data:

The response clocks use only zero-days: reviewed, exploited, not end-of-life, and exploited before a fix existed. Concretely, that means the researcher recorded exploited_before_patch as true, or the incident is still unpatched. We do not choose incidents by how they turned out, so a zero-day fixed in an hour counts exactly like one fixed in a month.

N-days are left out because their clocks are always zero. Counting them would raise a vendor's grade every time one of its old, already-fixed bugs was exploited. They are still shown on the vendor page ("N of M exploited bugs had a fix public before exploitation was reported"), together with how long fixes for every branch took when that came later.

The score curve

Every time-based sub-score uses the same smooth curve:

score(t) = 100 / (1 + (t / t50)^2)        where t50 = N x sqrt(3)

N is the public norm for that measure. The curve is pinned so that meeting the norm exactly scores 75 (a B). It falls smoothly as time grows, so slower is never better and there are no cliffs. The same shape applies to every measure:

Notes on the anchors:

• Our clocks are more lenient than the norms. Project Zero's 7 days and the CRA's 72 hours run from when the vendor learns of the bug. Our clocks start at the first public report, which is usually later, because vendors often know before the public does.
• BOD 26-04. CISA revoked BOD 22-01 and replaced it with BOD 26-04 on 10 June 2026. Since then most new KEV entries carry 3- or 14-day due dates. Version 1.0.0 keeps the 21-day norm because nearly all the evidence we grade dates from the BOD 22-01 period.
• Private fix uses ZDI's deadline. That is the deadline these cases actually ran under. Project Zero's stricter 90 days would score 84 (B+) on this curve.

Subject 1: Response to exploited zero-days

The three clocks

Each zero-day scores each of its clocks on the matching curve:

time to first advisory   = vendor      - world
time to first fix        = patch_first - world
time to fix all versions = patch_all   - world

A clock never goes below zero. Three rules apply:

• Still-open incidents. A zero-day with no fix yet is counted at the time elapsed so far. That is a lower bound on the true time, so its score is the best case for the vendor. It is counted only if counting it lowers the score. An open incident can never raise a score, and once it passes the vendor's typical performance it pulls the score down further every day.
• All versions. Time to fix all versions needs its own evidenced date; we never assume that the first fix covered every branch. It is never shorter than time to first fix, even when date-only rounding would make it so. Patched incidents without a public all-versions date count against openness instead (below).
• Day precision. Many sources give only a date. A date means "some time during that calendar day, in some time zone", so we treat it as a 48-hour window, from 12 hours before to 36 hours after midnight UTC. The clock is then a range of possible gaps, and its score is the average of the curve over that range. The numbers shown on the site count whole calendar days whenever a date-only timestamp is involved. When a single document at a single timestamp is the evidence for both moments, the gap is exactly zero. On the current data, moving any single date by one day changes a response score by at most 0.4 points. A letter can change only when a score sits that close to a grade boundary.

Openness

Openness scores how easy the vendor makes it for customers to know what is going on. It awards points out of 100:

Each share is smoothed as (yes + 1) / (known + 2). The incident-based shares use the same zero-days as the clocks, so an n-day never changes the response subject. Three rules keep openness from rewarding silence:

• Unknown counts as half. A component with no evidence scores 0.5; it does not drop out. Withholding information therefore never raises the score.
• Early warning applies only where it can. It counts only when the vendor had a zero-day with a public window.
• Disputed exploitation is not credit. If our research notes record that the vendor disputed the exploitation reports, and there is no structured flag, the incident counts as not stated.

Openness needs at least two components with evidence; otherwise it is NR.

Weights within the response subject

If openness is NR, its weight goes to the clocks. If a clock has no evidence, its weight is shared among the other clocks in proportion to their weights. Openness never gains weight. The response score is the weighted sum of the sub-scores as displayed, so anyone can recompute it from the page.

Subject 2: Private disclosure outcomes

Private clocks start on the source's recorded vendor-notification date. Advisories from one source, vendor, and report date form a case. Same-day publication batches still share evidence weight. These are sampled reports, not all vendor vulnerabilities.

Each case has one explicit measurement:

An advisory's date or a date in its URL is a research candidate, not a confirmed release date. Confirmation is recorded in data/curated/private-fixes.json with release evidence, a source URL, the advisories covered, and the review date. A batch is confirmed only when evidence covers every member. The latest applicable release date ends that batch's clock; this does not prove every supported branch was fixed unless the source explicitly establishes that scope.

The private grade scores confirmed durations on the 120-day curve. For patched cases without confirmed release dates it uses publication upper bounds as a conservative score input. Cases unpatched at publication score zero for that private-disclosure outcome. This is not an estimate of their eventual fix time or current status. The statistical score interval describes sampling uncertainty; it does not quantify uncertainty in the missing release dates.

The confirmed-fix median never mixes confirmed dates, upper bounds, and lower bounds. A 120-day outcome is known met for a confirmed fix within 120 days or a patched publication within 120 days. It is known missed for a confirmed fix after 120 days or an unpatched publication at/after day 120. Other cases are unknown. The deadline-miss percentage uses only the known outcomes, with its denominator shown. A late publication alone never proves a late patch. The 120-day comparison is a common benchmark, not a determination of compliance with exceptions or programme-specific disclosure policies.

Context: exploited-bug burden

The page shows the vendor's KEV entries: 12-month and 24-month counts, ransomware links, and a log-scaled score, 100 / (1 + B / 3), where B is roughly exploited bugs per year, with recent ones weighted more. It has weight 0 and never moves a grade. Large vendors ship far more software, so a raw count says more about size and attacker interest than about response. We show it so readers can see it apart from the grades.

Recency and severity

Recency. Evidence loses half its weight every 12 months, and evidence older than 36 months is ignored. Twelve months reflects the last one to two years of behaviour without letting a single month dominate.

Severity. Within one vendor's record:

• critical bugs count 1.5x
• high bugs count 1x
• medium bugs count 0.75x
• low bugs count 0.5x

Severity changes how much one case counts relative to the vendor's other cases. It does not create extra evidence.

Small samples: shrinkage, intervals, confidence

Three cases tell you much less than thirty, and we do not hide that. Each sub-score is pulled toward a starting value (the prior), and the pull fades as evidence grows:

sub-score = (E x vendor's average + 4 x prior) / (E + 4)

E is the amount of evidence: roughly the number of zero-days, or of private-case batches, with older ones counting less. The 4 means the prior counts as much as four fresh pieces of evidence. With 1 piece the sub-score sits 80% of the way to the prior. With 50 pieces it sits within about 7% of the vendor's own record.

The prior is the norm anchor: 75, the score that exactly meets the public norm, on every measure. Until a vendor shows otherwise, we assume it meets the industry norm. Thin evidence pulls a grade toward a B, not an A, so an A has to be earned with evidence. The same rule applies to both subjects, and it does not depend on other vendors.

In practice:

• Three recent zero-days, each fixed within the hour, give a clock score of about 85 and a response grade of about 83 (B+, provisional).
• Fifteen such zero-days give about 94 and 91 (A).
• Three zero-days that each waited 10 days for a first fix and 30 days for every version give a B− (71).

Evidence moves a score away from 75 in whichever direction it points. For comparison, the typical scores on the current data are 90 to 96 for the clocks and 65 for private fix. Shrinking toward those typical values instead put every rated vendor at A or A−, with the letter mostly set by the prior.

The 80% interval gives the range in which the score plausibly sits. It widens when evidence is thin or results vary. The spread we assume blends the vendor's own spread with a typical spread of 25 points. In the response subject the three clocks share their incidents, so their uncertainties are added as if they move together: the more cautious, wider range. The interval covers sampling noise only, not data errors. The sensitivity report gives a leave-one-out range for each vendor: how far the score moves when any single record is removed.

Confidence:

• high: the interval is at most 10 points wide and rests on at least 10 data points
• medium: at most 16 points wide, with at least 5 data points
• low: anything else

Not rated, provisional, and letter grades

The rule, exactly as published with the data:

NR (not rated) means a subject has fewer than 3 data points from the last 36 months, or their recency weights add up to less than one current data point. For response, a data point is a reviewed exploited zero-day. For private fixes, it is a private report (one vendor, one report date). KEV listings alone never rate a vendor. A grade is provisional when the vendor's own evidence does not yet outweigh the starting value, or confidence is below medium.

"The vendor's own evidence outweighs the prior" means E / (E + 4) is above one half, so E is greater than 4. For the response subject it is averaged over the clocks by weight. A provisional letter is still computed and shown, but it is marked, and the page says why: for example "3 reviewed zero-days. Not enough yet for a firm grade." or "Wide range: 57–83".

Meeting every norm exactly scores 75, a B. Taking half the norm time scores 92, an A. Taking twice the norm time scores 43, a D−.

What a grade means

Ranking

Each subject has its own leaderboard. Vendors are ranked conservatively, by the lower end of the 80% interval, then by score; vendors with equal values share a rank. NR vendors are not ranked. This puts a vendor with a well-evidenced 88 ahead of one whose 92 rests on three incidents. It therefore favours vendors with more evidence, which is the intent: a high rank should be hard to get by luck. Provisional vendors are ranked and marked.

Trend and history

Trend describes the response subject. It compares the last 6 months with the 18 months before them and reads:

• up or down only when the change is at least 5 points and at least 1.645 standard errors
• flat otherwise
• not shown when either period has fewer than 5 zero-days

History shows the response score at the end of each of the last 12 months. Each point is recomputed using only evidence dated on or before that month's end, so there is no look-ahead. An incident that was still open at the time counts as open in that month.

Worked example

A made-up vendor, computed by the real code, as of 22 October 2026.

Its evidence:

• five reviewed, critical zero-days from the last eight months, each with an advisory stating exploitation
• one n-day, not graded
• six private cases
• a CSAF feed and a mailing list

Time to first fix. The fixes came at 0 h, 0 h, 1 day, 9 days and 20 days after the first public report. Those score 100, 100, 99.3, 64.5 and 26.9. With recency weights from 0.89 down to 0.71, E = 4.0 and the weighted average is 80.2. Shrunk toward 75, that gives (4.0 × 80.2 + 4 × 75) / 8.0 = 77.6.

The other clocks. Time to fix all versions (0 h, 3 days, 1 day, 9 days, 30 days) scores 83.3. Time to first advisory (all within 12 hours) scores 87.4. Openness is 91.1.

Response: 0.40 × 77.6 + 0.30 × 83.3 + 0.20 × 87.4 + 0.10 × 91.1 = 82.6, a B+. The 80% interval is 73 to 92. It is provisional, because E = 4.0 does not exceed the prior's 4. The page says: "A fix was available within a day of the first public report for 3 of 5 zero-days; for the other 2, a median of 14 days (longest 20 days)."

Private fix. The cases took 60, 75, 95, 120, 150 and 240 days, published on six different dates. On the 120-day curve they score 92.3, 88.5, 82.7, 75.0, 65.8 and 42.9. With E = 5.1 their weighted average is 75.0, already at the prior, so shrinkage leaves 75.0, a B. It is provisional, because the interval of 66 to 84 is wider than 16 points.

Limitations

• We only see public signals. If attackers used a bug for months before anyone noticed, that window is invisible to us.
• Zero-days fixed at disclosure get full credit. The time when the vendor knew and customers were exposed is invisible. A vendor that warns customers early can score lower on time to first fix than one that stayed quiet until its patch was ready. The early-warning openness credit offsets only part of that.
• Under consideration. Half credit on the fix clock for an early official workaround, and a clock from the first public report of exploitation to the vendor's first statement that the bug is exploited. That second clock would make n-days count, for example where a vendor disputed exploitation reports. Also under consideration: a "vendor notified" moment, so the private window of zero-days fixed at disclosure is priced.
• Silent fixes. A fix shipped without an advisory, followed by a later advisory, is not penalised unless a third party reports the bug first. A rule for this is under review.
• ZDI is a sample, not a census, and Trend Micro runs it. Private cases come from programmes that publish timelines (ZDI, Project Zero), and they cover some vendors and products far more than others. For example, most Oracle cases are VirtualBox and most Apple cases are macOS; the page says so when one product dominates. Cases still in progress are invisible until published. We say on Trend Micro's page that ZDI is its programme.
• Windows do not line up yet. Incidents start in October 2024 (24 months); private cases and the recency window go back 36 months.
• KEV is not complete, and its ransomware flag can change after an entry is added.
• Registry facts are current. Advisory channels are taken as they are today, including for past months in the history.
• Data errors. Every timestamp links to its source and a verbatim quote, but researchers can still be wrong. With few data points, one record can move a grade. That is why most grades are marked provisional today, and why we accept disputes.
• What this evidence can support. Today one vendor has a firm response grade. The others are provisional Bs with wide, overlapping intervals. That is a finding, not a defect: most vendors have only three to eight reviewed zero-days, and most of those were fixed within a day of the first public report. The method does not separate vendors further than the evidence allows.

Disputing a timestamp

Every moment on every incident page has a "Dispute this timestamp" link. Tell us:

• the incident and the moment
• the date you believe is right
• a public source that shows it (an archived copy helps)

We re-check the source. If the evidence supports a change, we correct the record, keep the old candidate visible with a note, and regrade. Vendors are welcome to dispute their own timelines in the same way. Corrections apply to the whole history, because history is always recomputed from the current evidence.

Version history

Compared with the draft, 1.0.0:

• uses two subjects instead of one blended grade
• grades only researcher-reviewed incidents, and only zero-days on the response clocks
• anchors private fix on ZDI's 120 days, ends its clock at the vendor's own fix date where known, and counts same-day batches as one piece of evidence
• shrinks toward the norm anchor (75), so thin evidence means a B and an A has to be earned
• adds the provisional flag
• tightens confidence
• makes KEV burden context only
• hardens openness against rewarding silence
• corrects wording
• uses conservative ranking

See research/grading/sensitivity.md for the checks behind these choices.

Any change to a constant, curve, weight or rule increases the version number and is listed here. Every grade on the site states the version that produced it.