Lee Ovington, a real estate appraiser who works and blogs in Elgin, IL has successfully reverse-engineered Zillow.com’s Automated Valuation Method:
The above examples give us some indication of how Zillow arrives at its value estimates (or Zestimate). Quite simply, the Zestimate relies on a calculated relationship of assessed value to sale price. Zillow merely takes selected transactions and calculates the relationship between the Assessed Values and the Sales Prices. It then applies that ratio to the subject’s assessed value (plus or minus some adjustments) and “whala”, you have Zestimate!
The above examples show that even when Zillow has a large margin of error in its Zestimate of 10-15%, the Zestimate is still highly correlated with the Assessor’s Values. We can conclude from this analysis, that the Zestimate is a derivative of the Assessor’s Values. Zillow may be slightly modifying the data by some weighting or factor like time or distance. That “tweaking” of the data could be the “secret” part of its formula; but clearly, the Zestimate is based on the underlying Assessor’s Values as indicated by the high correlation coefficient.
This is not surprising, by itself. It’s how AVMs work, after all. But by deconstructing Zillow’s results, Ovington demonstrates how little sauce there is in the vaunted “secret sauce”…
Technorati Tags: blogging, disintermediation, real estate, real estate marketing
jf.sellsius says:
It’s no secret that if the assessors’ values are stale, so too is the sauce.
It is also odd how any sauce can be made when the key ingredients are missing– beds, baths, square feet & total rooms. This multifamily, despite missing these, still has a zestimate proudly published. Oh, and you might tell the chef he left out the # of units in this multifamily. Not very tasty.
http://www.zillow.com/HomeDetails.htm?city=Staten%20Island&state=NY&zprop=68319161
October 3, 2006 — 2:17 am
Dan Green says:
More importantly, it’s nice to know that there is an appraiser out there that understands “Correlation Coefficients”.
Nice find, Greg.
October 3, 2006 — 4:46 am
Greg Swann says:
> Nice find, Greg.
He found me. He came in as as a comment to Debunking Zillow.com.
October 3, 2006 — 6:03 am
anon says:
hey, it’s free, what do you want? as the saying goes, you get what you pay for.
btw, the area he chose is listed on the zillow data coverage page as only having “some” home information and only 65% of the homes in this area are within 10% of the selling price. ie. they admit they don’t have good data in this area yet.
http://www.zillow.com/howto/DataCoverageZestimateAccuracy.htm
a more scientific review of the zillow valuation model would take into account areas where they have “some” information, “more” information as well as areas where they have “most” information about the homes. until i see that, this is just more myopic FUD.
October 3, 2006 — 6:16 am
Robbie says:
The real “secret formula” of Zillow currently isn’t the inaccurate Zestimates in some parts of the country. It’s in getting the assessor’s data for every county & state in the country into their database, and then getting regular updates of that data.
Besides, Zillow’s Zestimates are getting more accurate over time. The race is a marathon, not a sprint.
October 3, 2006 — 11:48 am
David G from Zillow.com says:
Hi, it’s David from Zillow.com.
Lee’s findings are correct but his conclusions (repeated here) are not. This confusion is understandable; it looks like Zillow uses simplistic math in Lee’s neighborhood but its just because we don’t have complete facts about the houses there (yet). Where Lee is wrong is that if we do have complete information, we do use all of it — the resulting Zestimates are typically more accurate.
If you’re ever in the position of having to explain Zestimate accuracy to homeowners, please swing over to Lee’s blog and read through the more detailed comment I left on this topic.
jf –
If you read Lee’s review of Elgin Zestimates, you’ll see that extremely good accuracy is very achievable, even in this data-starved scenario.
Dan –
I seriously hope he’s not the only one. 🙂
anon –
Bless you, you seem to get it.
October 3, 2006 — 12:10 pm
David G from Zillow.com says:
It’s David again.
Robbie –
I just saw your comment – you also seem to get it.
Remember too that Zillow just launched a new way to import data — homeowners can now update their house’s facts on Zillow (after proving ownership and claiming their home).
This feature is a welcome addition for many site visitors — if you represent a seller who believes their home is misrepresented on Zillow, please encourage them to either update their home’s facts or publish their own value estimate for the house.
I’m very excited about the potential for using owner-generated data to complete our data base and keep it fresh in a way that would be impossible had we continued to rely exclusively on public records.
You are so right about the marathon – IMO, we’ve only just lined up and are waiting for the starter’s pistol to fire. 26 miles to go.
October 3, 2006 — 12:21 pm
Galen says:
Robbie’s right – the accuracy of Zillows estimates won’t be what we’re debating in a couple of years if they keep moving. They’ll be a side show. It’s the data in a homogenized and usable formate that is valuable.
October 3, 2006 — 12:29 pm
jf.sellsius says:
Robbie
Even assuming all home & environmental facts & unzillowables could be number crunched, there still exists a fundamental flaw in the recipe.
Whether it’s a sprint or a marathon, an AVM will always be two steps behind the current market conditions. In a rising, or falling market, you will lag.
An example: In NYC where last year’s market required you to overbid the asking price, your zestimate would have been useless. You would have bought nada. An analogy: It’s like trying to buy Google stock at the IPO price, the next day. Sorry that train left and jog or sprint you’ll never catch it. I think Galen has it right. In a homogenized stable market of cookie cutter homes you might get in the ballpark but that place only exists in the city of OZ.
Now with owner input to correct or add, zillow at last acknowledges that the published zestimate cannot self validate the public data it is fed.
BTW: Questions for Robbie (not required to answer )
Should Owners have the right to opt-out of the published zestimate? If not, why not?
(Greg, sorry to borrow your comment board to ask)
October 3, 2006 — 1:30 pm
Kevin says:
Makes sense that Zillow’s approach would be different in different areas, depending on the quantity and quality of the information available.
Here in California, thanks to Proposition 13, homes are rarely re-assessed, so it’s doubtful the official assessor’s values would weigh as heavily in the Zestimate as they appear to in Lee Ovington’s area.
October 3, 2006 — 3:16 pm
Robbie says:
jf.sellsius
In a rapidly changing market, I agree that any AVM will be two steps behind (assuming it’s using old data). Except there are 2 points, which I think you are neglecting.
1) Most markets aren’t rapidly changing. Using your analogy, if I wanted to buy Google tomorrow, it’s very likely that tomorrow’s price will be very close to what today’s closing price is (about $404/share).
This isn’t saying the market is static. However the market is closer to static than rapidly changing chaos. There are exceptions (such as GOOG at IPO, my house could burn down, etc.) in both equity markets and housing markets, but generally speaking the price of something tomorrow is about the same price of something today is.
When markets are quickly changing, they usually reach a state of equilibrium after a period of time. Even a broken watch is still exaclty right twice a day.
2) They’ll eventually be only one step behind. Government assesors have a vested interest in collecting the full amount of tax they are entitled to under the law. All this cool technology that everybody else uses, will eventually find it’s way to the assesor’s office, and assesments will become more accurate and more timely over time. Do you really think the tax man is going leave money on the table if he can help it?
Regarding your question:
I think you should be asking if governments should be allowed to collect data on property owners and should citizens (both corporate and individuals) be allowed to get government data via Freedom of Information Act or other means? Currently, under law, Zillow has a right to that public data.
You or I may not like what they are doing with it, but as a corporate citizen it’s their right to do what they are doing. If I don’t like it, I’ll have to write my congressman, state attorney general, etc, and complain, because as it stands now, Zillow isn’t breaking any laws.
Maybe someday, people will force Zillow to start a do not Zetimate list, like https://www.donotcall.gov?
October 3, 2006 — 4:38 pm
jf.sellsius says:
Robbie,
Point taken on market change. To the extent a market is not rapidly rising or falling, I agree that unzillowable factor is greatly minimized. But there are so many more unzillowables. You can’t price the next door neighbor, his junk yard dog, traffic noise, zoning laws or a new public works project coming. Case in point: If you move to NYC & buy a condo where the Second Avenue subway will be built with only your zestimate tucked under your arm, you’re in for 10 years of noise & vibration and a quickly depreciated piece of real estate. Another: I used to live near the largest dump in the world (a few miles away & somewhat hidden.) In the winter no problem— I could sell a newbie at full asking. In the summer, a whole different story. Markets may not move fast but the value of that house fluctuated every season. The problem w/ AVMs is they don’t live in the real world. They are intellectual while real estate is emotional, sensual.
Cutting to the chase, Zillow’s mantra: A starting point. Heck, my neighbors are a starting point.
Bottom Line: There is a BETTER starting point than Zillow. An experienced professional with eyes, ears & a working brain that actually experiences your home & it’s environs armed with raw data
You miss my point on public data & zestimates. Public data is free to all, I agree. But a zestimate is NOT public data, it USES public data to create a proprietary value, an opinion, which is published without an owner’s consent & which, if grossly inaccurate, can materially interfere with an owner’s marketing of their house. We’ll have to await the court case on this one.
The “do not zestimate list” is ok but maybe it should be: Get an owner’s consent before you publish your opinion of the value of their property.
October 3, 2006 — 7:54 pm
Robbie says:
I think we can all agree that an experienced professional is better. But are they free? Are they fast? Are they on call 24/7? If I’m putting my home on the market and money is on the line, I’ll call a professional. If I just want a “close enough” estimate, I’ll visit Zillow, and let the professional serve paying clients who are willing to pay for the superior assesment he’ll offer, instead of the cheap assesments for the just curious.
I agree that some homes are unzillowable. But do we know what percentage of homes in the US that is? Is it 10%? 1%? 0.1%? 0.01%? Furthermore, whoose to say that a human won’t miss some of these same facts when making their own assessments? An assessor from western Wyoming would be equally worthless in downtown Seattle or NYC. Is my home sitting on top of oil reserves or indian burial grounds? Some things are unknowable, as well as unzillowable.
I agree, that a Zestimate is Zillow proprietary value. However, since Zillow owns that value, they can do with it what they want. Now, I agree if a grossly inaccurate Zestimate, materially interfered with an owner’s marketing of their house it could lead to a court case. But as you said, that’s for lawyers to decide.
October 4, 2006 — 4:54 am
jf.sellsius says:
Robbie,
I respect and value your opinion. If we agree pros are better than zillow, great. Now for cost. A CMA is free and fast (their data is proably fresher too), though not as fast as a mouse click, agreed. But accuracy should trump speed. Even if the margin of error is single digit % points, the actual $ amt can be huge, especilly as you get into million $ homes.
Unzillowables: A pro may not get them all, but much much more than the AVM. And local market conditions & market pool are kings of the unzillowables which AVM dont even touch.
To avoid problems w/ inaccurate zestimates, an opt-out seems fair.
Do you think an owner should have the right to opt-out?
October 5, 2006 — 9:27 am
anon says:
opt out? nope.
do i get an opt-out from the county recorder site that shows sold info? nope.
do i get an opt-out from the tax assessor’s site that shows that they assess it at? nope.
is kbb.com going to take your car off the site because you don’t want some poor buyer to know what it may be worth? nope.
October 5, 2006 — 10:28 am
jf.sellsius says:
Anon
County recorder and tax assessors’ records are PUBLIC data, a zestimate is not public data –it is a proprietary creation.
KBB gives the price of Chevys–NOT the price of MY Chevy.
See the difference. Nope?
October 5, 2006 — 4:30 pm
anon says:
sure, but i could look at your chevy’s mileage through the window, plug it into kbb.com and use their valuation model to arrive at the price of YOUR chevy. it’s the same thing, only the consumer has to do a little more work in the car case.
October 7, 2006 — 12:37 pm
jf.sellsius says:
Anon,
1. kbb does not look in car windows before it publishes
2. I park my car in a garage.
3. My car is out of alignment, it needs new brakes, & the transmission is shot. I’ll sell it to you at kbb value though. It’s nice & shiny. Deal?
October 8, 2006 — 4:07 pm