4319 Miramar Drive

1. Leave-one-out back-test (mean absolute percentage error). For i = 1..n the full pipeline — comparable adjustment across all 66 factors, then adjustment-weighted reconciliation — is refit on D \ {(xᵢ, yᵢ)} and evaluated at xᵢ. The reported statistic is n⁻¹ Σ |ŷ₋ᵢ − yᵢ| / yᵢ. LOOCV is approximately unbiased for expected out-of-sample risk but has high variance: the n training sets share n−2 observations, so fold estimates are strongly positively correlated and the apparent precision is optimistic (Hastie, Tibshirani & Friedman, ESL 2nd ed., §7.10). At n = 3 this is the binding caveat and the reason the figure is presented as descriptive of this comparable set rather than as a population claim.

2. Adjusted spread — a dispersion measure, not an error measure. (max vᵢ − min vᵢ) / V̂ over the adjusted comparable values. It quantifies agreement among estimates, which is not the same quantity as distance from the truth and can move in the opposite direction. Three comparables drawn from one subdivision, adjusted by one rate table, share method and data; their errors are positively correlated, so a tight spread partly reflects common specification rather than convergence on the right answer. Formally, for estimates with pairwise error correlation ρ, the variance of their mean is (σ²/n)[1 + (n−1)ρ] — under strong positive ρ the effective sample size is far below n, and observed spread understates true uncertainty. Low spread with a shared bias is the classic route to a confident wrong answer, which is why this page reports spread and held-out error separately and never substitutes one for the other.

3. Mean absolute adjustment — a proxy for accumulated model error. n⁻¹ Σ gᵢ/Pᵢ, gross adjustment over net sale price. Every adjustment is itself an estimate carrying its own error, so total adjustment magnitude bounds how much of a final figure is model rather than observed transaction. Under the approximation that adjustment errors are independent with variance proportional to magnitude, the error contributed grows with √Σg² — which is the premise the reconciliation weighting rests on, and the reason gross rather than net is the right summary. It is an ordering statistic, not a calibrated error: it ranks comparables by how much work was done to them without asserting how wrong that work was.

4. Mean comparable age — exposure to temporal drift. Mean days between each comparable’s close date and the valuation date. Time adjustment removes the expected market movement, so residual risk is the variance of the index estimate over the carry interval, not the movement itself. Under the Case–Shiller error decomposition that variance grows roughly linearly in the interval, Var ≈ σ²ᵘΔt + 2σ²ᵤ, so age is a proxy for how much index uncertainty each comparable imports. A 145-day mean is modest; the relevant exposure is not the elapsed time but the month-to-month volatility of the county index across it, which the market-timing section quantifies directly.

What the four are jointly evidence of. Taken together they describe the provenance of the estimate, not its accuracy: how much of the final figure is observed transaction and how much is model (mean absolute adjustment), whether the independent indications agree (adjusted spread), how much temporal extrapolation was required (comparable age), and how the method performed when it could not see the answer (the back-test). Only the last is an error measure. The first three can all look excellent on a valuation that is wrong.

Why they cannot simply be combined. A reader’s instinct is to average them into a single quality number. They are not independent, and the dependence runs the wrong way. Comparables selected for similarity have small adjustments by construction — the cascade chose them for it — so a low mean adjustment partly reflects the selection rule rather than the market. Tight agreement among three comparables adjusted by one rate table reflects the common rate table. Formally, for estimates with pairwise error correlation ρ, the variance of their mean is (σ²/n)[1 + (n−1)ρ]: at ρ near 1 the effective sample size approaches 1 regardless of n, and observed spread understates true uncertainty by an unknown factor. Nothing here estimates ρ — three observations cannot.

The failure mode this section exists to expose. Comparables that are genuinely alike, adjusted by a shared and slightly wrong rate table, will produce a tight spread, small adjustments, and a low back-test error — while every one of them is biased in the same direction. Correlated error is invisible to all four measures simultaneously. That is not a hypothetical on this property: two comparables sit on double the land, both take a large adjustment from the same land rate, and if that rate is wrong both move together. It is the reason the land section states its rate and its source rather than burying them.

What none of these four is. None is a confidence interval, and none should be read as one. Coverage statements live in Cross Validation, where they are derived from the held-out error distribution rather than from agreement among comparables — agreement among correlated estimates being precisely the thing that can be high while the answer is wrong.

Specification. Bailey, Muth & Nourse (1963), A Regression Method for Real Estate Price Index Construction, JASA 58(304), 933–942. For a dwelling transacting at periods s and t with s < t, the model is ln(Pᵗ/Pₛ) = βᵗ − βₛ + εₛᵗ. Let X be the N × T design matrix with xᵢₛ = −1, xᵢᵗ = +1 and zero elsewhere. Then β̂ = (X′X)⁻¹X′y with yᵢ = ln(Pᵗ/Pₛ). The base period is normalised to β₀ = 0 and its column deleted; retaining it makes X′X singular, since the indicators sum to zero by construction. The index is Iₜ = 100·exp(β̂ₜ).

Identification. The estimator is consistent for the market factor under the assumption that E[ε | X] = 0 — that idiosyncratic price changes are mean-independent of when a property happened to transact. Dwelling fixed effects difference out exactly because the same unit appears with opposite signs on both sides, which is what removes omitted-variable bias from unobserved quality and is the method’s central advantage over a hedonic index.

Error structure. Case & Shiller (1987, New England Economic Review; 1989, AER 79(1), 125–137) decompose εₛᵗ = (Hᵗ − Hₛ) + (uᵗ − uₛ), where H is a Gaussian random walk in the property’s own value and u is i.i.d. transaction noise. The implied variance Var(ε) = σ²ᵘ(t−s) + 2σ²ᵤ grows linearly in the holding interval, motivating their three-stage weighted least squares: OLS, regress squared residuals on the interval, then GLS with weights 1/√(â + b̂(t−s)). This index does not apply that weighting. It instead excludes holds under 180 days and annualised changes beyond ±60% outright — a trimming rather than a down-weighting strategy, which is more conservative, discards information the WLS would retain, and is stated rather than left implicit.

Shrinkage. County-month estimates are shrunk toward the metropolitan estimate in log space, ln Iᶜₜ = w ln Iᶜₜᶜ + (1−w) ln Iᵐₜ with w = n/(n+k), k = 20. This is the James–Stein / empirical-Bayes posterior mean under a normal-normal hierarchy where k is the ratio of within-month sampling variance to between-county prior variance; here it is fixed rather than estimated. The effect is that thin months borrow strength in proportion to how little evidence they carry of their own — September 2026, with 38 pairs, sits at w = 0.655.

Known limitations. (i) Sample selection. The index is estimated only on dwellings that transacted at least twice, which is not a random sample of the stock; properties that turn over frequently differ systematically, and no Heckman-type correction is applied. (ii) Revision. Estimates for recent periods move as later transactions arrive, so the most recent months are the least stable — a property shared with all repeat-sales indices including S&P CoreLogic Case–Shiller. (iii) Renovation. The dwelling-constant assumption fails where a property was materially improved between sales; the ±60% annualised filter removes the extreme cases and cannot remove the moderate ones. (iv) Aggregation. A single county index assumes a common market factor across submarkets within the county.

Empirical note on smoothing. A trailing three-month average of the log index was tested against the raw monthly series by ten-fold cross-validation on 20,631 Williamson pairs, scoring out-of-sample RMSE of predicted log price change. It was worse in 10 of 10 folds (mean difference +0.00171, 95% CI [+0.00139, +0.00202], t = +10.70); a five-month window was worse again (t = +22.77). Monotonicity in the smoothing window is the signature of signal removal rather than noise suppression, consistent with the shrinkage above already performing the variance reduction a second pass would duplicate.

Part one — what a valuation should be measured against

This section is reserved. The standard is being supplied separately and will be placed here in full. It is deliberately empty rather than filled with a plausible-looking threshold: a benchmark that this report happened to pass would be indistinguishable, to a reader, from one it had actually been measured against. Until the paper is here, no accuracy standard is being claimed or implied anywhere in this report.

What is known today, pending that standard. Three sets of numbers exist and none of them is a standard — they are measurements, and they are not measuring the same thing:

1. This property. Median error 6.71%, FSD 11.44%, RMSLE 0.0953, 68% interval ±9.28%, from three held-out comparables. Specific to this house, and thin: three observations describe this comparable set accurately and generalise weakly.

2. This model, across its blind holdout. MdAPE 7.77%, MAPE 11.22%, RMSLE 0.1605, 95% coverage 89.87% over 19,486 sales the model never saw during training. By price band on single-family homes: $250–400k 6.09%, $400–600k 7.20%, $600k–1M 8.35%, under $250k 9.92%, over $1M 13.10%. This is the right basis for any general claim about the method, and it is deliberately not what the tiles above report — those answer “how accurate is this valuation”, which is a different question.

3. What the vendors publish for themselves. HouseCanary: national MdAPE 2.8% over 1,994,203 transactions internally, 2.9% on a blind third-party test. Zillow: median error 1.79% on-market and 7.20% off-market. Two cautions before either is treated as a bar to clear. Both are national aggregates dominated by dense, homogeneous tract housing where an automated model does best, and neither is conditioned on the kind of property this is. And the on-market figure is not independent of the asking price — both vendors consume list price, which is the whole point made in the Benchmarks section.

What a standard would need to specify to be usable here, and what the forthcoming paper will presumably settle: which statistic is the criterion (MdAPE, FSD, coverage, or a hit rate); whether it is a point threshold or a distribution; whether it applies per property or per portfolio; how it is conditioned on price band, property type and evidence depth; and whether an interval must be calibrated — that is, whether a stated 68% interval must actually contain the outcome 68% of the time, which is a materially stricter requirement than a median error threshold and the one most AVM disclosures avoid.

Where this report already exceeds common practice, whatever the standard turns out to be: the comparables are named, every adjustment is itemised with its reason, the estimator and its weights are stated, and the three held-out test results are printed so every summary statistic can be recomputed by hand. A standard can be applied to a number; it can only be verified against a number whose derivation is visible.

Part two — how this valuation was measured

Why these figures are worse than an earlier version of this report. Every figure above is computed from the three held-out rows and can be reconciled by hand. They are materially worse than earlier versions because the land difference is now carried in full. When the two acre-sized comparables were adjusted by a limited amount the median error was 2.65% and the FSD 3.53%; carrying the full $114,876 and $112,533 they are 6.71% and 11.44%. Both things are true at once: the full figures are what the method produces, and the smaller ones predicted these three sales more closely. The honest reading is that two comparables on double the land are hard to adjust reliably in either direction.

Why hold anything out at all. Scoring a method on the same sales it was given measures how well it can interpolate data it has already seen, which is not the quantity anyone cares about. The quantity of interest is expected loss on an observation drawn from the same population and not used in fitting: Err = E_(X,Y)[L(Y, f̂(X))]. Resubstitution error is downward-biased for it by construction. The land difference is where this bites on this property. Two of the three comparables sit on about double its land, so each time one was held out the method had to price roughly half an acre with no recent sale of its own to price it from. Those are the misses in the table above.

Design: leave-one-out cross-validation. For i = 1,…,n the entire pipeline — all 66 adjustment factors, the repeat-sales time carry, and the inverse-gross-adjustment reconciliation — is refit on D_−i = D \ {(xᵢ,yᵢ)} and evaluated at xᵢ, giving CV = n⁻¹ Σᵢ L(yᵢ, f̂_−i(xᵢ)). Critically, the held-out comparable is excluded from the reconciliation as well as from the adjustment grid; leaving it in the weighting while removing it from the grid would leak the answer through the weights.

Properties, and the one that governs here. LOOCV is approximately unbiased for Err — it trains on n−1 of n points, so the pessimistic bias of k-fold at small k is largely absent. It pays for this in variance: the n training sets are nearly identical, sharing n−2 observations, so the fold-level estimates are strongly positively correlated and Var(CV) does not contract at the 1/n rate an independent average would. At n = 3 this is the binding constraint on everything below (Hastie, Tibshirani & Friedman, ESL 2nd ed., §7.10–7.12; Efron & Tibshirani, An Introduction to the Bootstrap, 1993, ch. 17).

Forecast standard deviation. FSD = sd{ln(ŷᵢ/yᵢ)}, computed on the held-out pairs. The log ratio is scale-invariant, so the statistic is comparable across price points and across vendors without normalisation. Under an approximately log-normal error, exp(±FSD) bounds a one-standard-deviation multiplicative interval; the log-normality is an approximation and is not tested at n = 3. This is the AVM industry's disclosure convention, which is what makes the comparison against HouseCanary's 3% and Zillow's 5% like-for-like in definition — though not in sample, since theirs are national aggregates over millions of transactions and this is three comparables for one house.

Interval construction, and why not a normal-theory interval. The 68% and 95% figures are empirical quantiles of the absolute percentage error distribution {|ŷᵢ−yᵢ|/yᵢ}, not μ̂ ± zσ̂. A normal-theory interval would import a distributional assumption the sample cannot support and would be symmetric in levels, which a multiplicative error process is not. The trade-off is explicit: a quantile at the 95th percentile of three observations is an extrapolation of the fitted distribution, not an observed exceedance rate, and is labelled as such wherever it is quoted. Coverage validity requires exchangeability between the held-out comparables and the subject — the assumption underpinning conformal prediction (Vovk, Gammerman & Shafer, Algorithmic Learning in a Random World, 2005) — which same-subdivision, same-era selection supports without guaranteeing. It is worth noting that conformal width calibration was measured for this pipeline and rejected: a six-month-forward holdout violates exchangeability outright, and coverage did not improve.

RMSLE. RMSLE = [n⁻¹ Σ (ln(1+ŷᵢ) − ln(1+yᵢ))²]^1/2. Squaring in log space penalises proportional rather than absolute error and bounds the leverage of a single large residual relative to RMSE. It is asymmetric in levels: under-prediction is penalised more heavily than over-prediction of the same absolute size, since |ln(1−δ)| > ln(1+δ). The 1+ offset is vestigial at these magnitudes. RMSLE is a relative statistic with no absolute interpretation — it exists to rank methods, and quoting it alone says nothing.

MdAPE and MAPE. MAPE = n⁻¹ Σ |ŷᵢ−yᵢ|/yᵢ; MdAPE is the median of the same summands. MAPE is undefined at y = 0 and structurally asymmetric: unbounded above for over-prediction, bounded by 1 for under-prediction, so it systematically favours methods that under-forecast. The median is reported alongside because it has a 50% breakdown point against the mean's 1/n. Here 7.80% against 6.71% indicates no single dominating residual.

Hit rates. PPE_k = n⁻¹ Σ 1{|ŷᵢ−yᵢ|/yᵢ ≤ k} at k = 0.10 and 0.20 — the empirical CDF of absolute percentage error at two points. On this property: 2 of 3 within 10%, 3 of 3 within 20%. Reported as counts, not percentages: the estimator takes only four values at n = 3, and a Wilson score interval on 2/3 spans roughly [0.21, 0.94], which is most of the parameter space.

What is not claimed, stated plainly. No standard error is attached to any statistic here. The sampling distribution of a standard deviation at n = 3 is severely right-skewed, and a nonparametric bootstrap resamples the same three points — it would produce an interval, and that interval would be an artefact of the resampling rather than evidence about the population. No hypothesis is tested and no p-value is reported, because there is no null here worth rejecting at this sample size. These figures are offered as descriptive statistics of this comparable set, which is what they are and the most that three observations can support. The corresponding population-level figures — MdAPE 7.77%, RMSLE 0.1605 and 95% coverage of 89.87% over a blind holdout of 19,486 sales — exist and are the right basis for any claim about the method in general; they are not what this page reports, because George asked for the numbers specific to this property and those are these.

Why an undisclosed model is not a defensible statistic. A figure is defensible when someone else can reproduce it: the estimator is specified, the data are identified, and the validation protocol is stated. Both vendors publish outputs and accuracy summaries; neither publishes the estimator, the trained parameters, or the data. Their accuracy figures are therefore self-reported and self-scored on samples they select. This is normal commercial practice and not a criticism of their competence — it is a statement about what an outside reader can check, which is: the claim, but not the calculation.

Zillow, in its own words. The Zestimate is described as combining “public records, MLS data and user-submitted home details into Zillow’s proprietary home valuation model,” offered as “a transparent, free starting point,” and explicitly “not an appraisal and can’t be used in place of an appraisal.” Published accuracy: nationwide median error 1.79% on-market and 7.20% off-market — meaning half of on-market Zestimates fall within 1.79% of the eventual sale price. Zillow states directly that on-market estimates are more accurate “because more up-to-date information is available, including listing details and recent market activity.” No specification, feature list, or training data is published.¹

HouseCanary, from its published technical brief. Considerably more is disclosed. The algorithm runs in three stages: (1) query and clean data; (2) build localised price indices; (3) train machine-learning models on time-adjusted historical prices. Models are fitted at census-tract level, borrowing from neighbouring tracts where a tract is too thin to model alone, with neighbours used for training but excluded from accuracy testing. Two index models are built — median price and median price per square foot — at census-block level, and all historical sales and list prices are carried to current values through them. The response variable is each price expressed as a percent deviation from its block’s current median for that property type; several ML models per tract are then fitted to explain that deviation and combined into one estimate. Inputs include 3,100+ county assessors, 2,700+ county recorders over 20 years, MLS characteristics, listed prices and contract prices, mortgage balances and distress measures. In non-disclosure states such as Texas, MLS contract prices substitute for recorded sale prices where an arm’s-length sale can be jointly verified — directly relevant here.²

HouseCanary’s FSD, and an important caveat they state themselves. FSD is trained on the census-tract empirical error distribution and depends both on that spread and on how much the component models disagree for the individual property. The confidence score is simply 1 − FSD, which is why the 97% confidence reported for this property corresponds to an FSD of 0.03. The interval is constructed for approximately 68% coverage: P(1 ± FSD). And in their own words: “We assume symmetry but not normality... using 2*FSD to estimate a new interval is not guaranteed to yield an approximate 95% coverage probability.” Anyone doubling their FSD to get a 95% band is doing something the vendor explicitly warns against.

What HouseCanary validates, and how. Monthly internal testing on a rolling six-month window, plus quarterly blind third-party testing. As of July 2019: national MdAPE 2.8% over 1,994,203 transactions internally, and 2.9% on the third-party blind sample for Q1 2019. They report hit rate, MdAPE, median signed error (as a bias check), within-5/10/20%, and the share of sales falling inside their own 68% interval — a coverage check, which is the right diagnostic and more than most vendors publish. Results are available by state and MSA on request, which is the boundary of the disclosure: the validation design is public, the validation data are not.

The comparison this page makes, and its limit. FSD is a shared convention, so this report’s 11.44% and their 3% and 5% are the same quantity — the standard deviation of log(estimate ÷ outcome). But they are computed on different samples by different methods: theirs across millions of national transactions, this one across three comparables for this house. A national median error says what happens to a typical home; it does not say what happens to a half-acre property whose nearest comparables sit on twice the land. Reading their 3% as a promise about this house would be a category error, and reading our 11.44% as vendor-grade inaccuracy would be the same error in reverse.

What this report exposes by contrast. The estimator is stated, the comparables are named with MLS numbers, every adjustment is itemised with its reason, the index method is a published 1963 regression with its exclusions listed, and the three held-out test results are printed so every summary statistic can be recomputed by hand. Whether the answer is right is a separate question; whether it can be checked is not.

¹ zillow.com/z/zestimate — quotations from Zillow’s published “What is a Zestimate?” page. · ² HouseCanary Valuation Technical Brief, © 2019 HouseCanary Inc., read in full. Figures are as of that publication and may have moved since. · ³ cotality.com (formerly CoreLogic) — no valuation estimate was obtained; the contract is pending renewal.