Methodology and full results
This page documents how the 2027 HVAC Promise vs. Proof study was designed, measured and analyzed, and reports every preregistered result, including the one that was not confirmed.
Population and frames
The population is HVAC companies with a website listed on Google in 10 large U.S. metro areas. Frames were built from Google listings in each metro area and frozen before the sample was drawn.
Metro areas: Atlanta, Charlotte, Chicago, Dallas–Fort Worth, Denver, Houston, Las Vegas, Philadelphia, Phoenix, Tampa.
Sampling and the Explore / Confirm split
1,000 companies were drawn, stratified by metro area and by review-volume group (thirds of Google review count within each metro area). Companies used in earlier pilot runs were excluded. The sample was split at random into an Explore half and a Confirm half of 500 each. Hypotheses, tests, minimum meaningful effects and the analysis code were frozen before the Confirm half was analyzed.
Source: AgencyFix Research, 2027 HVAC Promise vs. Proof Report
| Measure | Value |
|---|---|
| Step 1 | frames: Google-listed HVAC companies with a website in 10 metro areas |
| Step 2 | 1,000 drawn (stratified by metro x review-volume group) |
| Step 3 | 500 Explore / 500 Confirm |
| Step 4 | responding websites (site nonresponse 23.1% weighted, Confirm) |
| Step 5 | per-test denominators (e.g. C1 n = 190) |
<iframe src="https://agencyfix.com/data/hvac-2027/embed/methodology-sample-funnel/" title="How the 2027 HVAC study was built" width="640" height="400" loading="lazy" style="border:0"></iframe> <p>Source: <a href="https://agencyfix.com/data/hvac-2027/methodology/">AgencyFix Research, 2027 HVAC Promise vs. Proof Report</a></p>
| Item | Value |
|---|---|
| Study | cxi-hvac-2027-1000 |
| Preregistration | prereg-1000.v2 · sha256 2f64c21314e5e0fffeed74e68ae19e8ec895334dd3b5195dad46787fb2bb0ff6 |
| Analysis code | confirm1000.v1 · sha256 cae77bd67b5edc9ac34908b0deadb440dc1246717d872696b8c5b8fdf761d56b |
| Claim versions | v1 (all published claims) |
Measurement
Website claims were read from the company's primary and secondary pages; blog posts and raw-text-only matches were excluded, so a claim had to be a prominent commercial claim. Google hours, review counts and owner replies were read from the company's Google listing. Reply rates use the newest up to 200 retrieved reviews. Whether a page clearly explains the after-hours process was judged with AI assistance and is reported as a weak result.
Our research crawler identifies itself, respects robots.txt and never bypasses bot challenges. Websites that could not be read count as nonresponse rather than being worked around: weighted site nonresponse in the Confirm half was 23.1%.
Estimation
Estimates are design-weighted by metro area and review-volume group. Confidence intervals are 95% intervals from a stratified bootstrap with 2,000 replicates (seed 20270101). The 10 confirmatory tests were corrected for multiple comparisons with the Holm–Bonferroni method, and each test also had to clear a preregistered minimum meaningful effect.
Full results
| Test | Finding | Explore | Confirm | 95% CI | Holm p | n | Decision |
|---|---|---|---|---|---|---|---|
| C1 | In the study, 57% of 24/7-claiming HVAC companies did not list 24-hour hours on Google. | 56% | 57% | 50% to 64% | <0.001 | 190 | confirmed |
| C1s | In the study, 30% of HVAC company websites claimed 24/7 service that their Google listing did not show (10% the reverse). | 27% | 30% | — | — | 346 | descriptive only |
| C2 | In the study, 8% of HVAC emergency-service websites clearly explained what to do after hours (AI-assisted review). | 8% | 8% | 4% to 13% | <0.001 | 169 | confirmed |
| C3 | Not confirmed (see disclosure below) | 14 pts | 8 pts | -2 pts to 18 pts | 0.066 | 372 | not confirmed |
| C4a | In the study, 70% of the most-reviewed HVAC companies mentioned financing vs 25% of the least-reviewed. | 54 pts | 46 pts | 34 pts to 56 pts | <0.001 | 247 | confirmed |
| C4b | In the study, the most-reviewed HVAC companies replied on average to 66% of recent Google reviews; the least-reviewed, 24%. | 44 pts | 41 pts | 32 pts to 51 pts | <0.001 | 259 | confirmed |
| C4c | In the study, 66% of the most-reviewed HVAC companies showed some pricing vs 46% of the least-reviewed. | 36 pts | 20 pts | 7 pts to 32 pts | <0.001 | 247 | confirmed |
| C4d | In the study, 69% of the most-reviewed HVAC companies promoted a maintenance plan vs 43% of the least-reviewed. | 28 pts | 27 pts | 15 pts to 40 pts | <0.001 | 247 | confirmed |
| C4e | In the study, 60% of the most-reviewed HVAC companies made a guarantee or warranty claim vs 35% of the least-reviewed. | 30 pts | 25 pts | 12 pts to 37 pts | <0.001 | 247 | confirmed |
| C4f | In the study, 75% of the most-reviewed HVAC companies advertised emergency service vs 46% of the least-reviewed. | 28 pts | 29 pts | 17 pts to 41 pts | <0.001 | 247 | confirmed |
| C6a | In the study, HVAC companies with more Google reviews tended to reply to a larger share of them (rank correlation 0.38; association only). | 0.50 | 0.38 | 0.25 to 0.50 | <0.001 | 345 | confirmed |
| C6b | Descriptive: in the study, reply rate and star rating showed no clear association (rank correlation 0.05). | 0.19 | 0.05 | -0.07 to 0.17 | — | 345 | descriptive only |
Shares in %, group differences in percentage points, associations as rank correlations. Confirm values are the published numbers; Explore values are shown for transparency only.
Limitations
Every finding is an association in public records, not proof of effect. Review-volume groups describe review counts, not how big a company is. A Google listing without 24-hour hours does not show that a company is closed at night. Some websites could not be read, and the channel measures are a lower bound from page text. Results describe 10 large metro areas in 2026 and should not be generalized beyond them.
Versioning and corrections
Every published number carries a claim version. A correction creates a new claim version with a visible changelog entry; earlier versions stay available for reference.
How to cite
Source line: Source: AgencyFix Research, 2027 HVAC Promise vs. Proof Report
AgencyFix Research. (2027). The 2027 HVAC Promise vs. Proof Report (study cxi-hvac-2027-1000, claim versions v1). https://agencyfix.com/data/hvac-2027/methodology/
Please cite the specific finding page when quoting a single number; numbers are design-weighted estimates from the Confirm half of the study.