We asked Zillow and other online appraisers to predict the sale price of homes. The winner was right just half the time. But these tools still have their uses.
By Marilyn Lewis
Playing with online home-appraisal sites like Zillow is a great game. Simply plug in your address to find what the site says your house is worth. The real fun is checking on your friends' homes -- and your neighbors.
But are these electronic wizards -- called automated valuation models (AVMs) in real-estate lingo -- real tools or just toys? We set up a test to find out.
The market value of a house is what a buyer will pay for it. An accurate estimate comes as close as possible to the market price. Heres how we compared six popular online appraisers:
- We contacted real-estate firms around the country and asked for prices for just-closed single-family home sales;
- We plugged those addresses -- two for each test city (Seattle, Minneapolis-St. Paul, Scottsdale, Ariz., Baltimore, Cincinnati and Portland, Ore.) into each appraisal site to get estimates of values. These cities were chosen to give the AVMs the best possible shot: Zillow.com, the only site that posts its confidence ratings by region, shows these as among its most accurate metro areas.
- We worked quickly to gather the online estimates before the sales were recorded and picked up by the AVMs. After all, 100% accuracy is no trouble at all for an AVM that has the real sales price. But at least one of the estimates -- RealtyTrac's estimate for the house called "Seattle 2" -- already had incorporated the price of the recent sale.
What's accuracy, anyway?
It seemed fair to judge AVMs by the standard to which appraisers hold themselves. Cincinnati Realtor Steve Casper, of Comey and Shepherd Realtors, said that the typical margin of error is 5% -- tops. "When relocation companies ask for broker price opinions," Casper said, "they expect the two they request to be within 5% of each other, or they will request a third to determine which is the more accurate."
Related news and commentary on MSN Money
Appraiser-Realtor Francois "Frank" K. Gregoire is a little stricter -- he grades himself on a standard of 4% to 5%. "My clients are making decisions and settlements involving real dollars based on my opinions and conclusions. Wide variations cost them money," he said.
Gregoire, of Gregoire & Gregoire in St. Petersburg, Fla., has a dog in this fight, of course. If the AVMs get too good, they'll start taking away business from appraisers like him. Already, bankers and big mortgage insurers have found uses for them, including assessing certain low-risk loan applications.
But 5% sounded fair. As it turned out, that was a high bar for AVMs.
Strictly speaking, Domania won the contest, with a 57% accuracy record. But Domania gets no prize: It only delivers a range, not a number, and while a reasonable range would be great, Domania's ranges can be so vast as to include every number from the price of eggs to the cost of the federal budget.
For example, Domania's estimate for the Cincinnati 2 property was a range of $394,800 to $545,200. The home sold for $470,000. Adding or subtracting 5% gives a window of $446,500 to $493,500, or $47,000 in slack -- plenty of room to nail the price. In fact, the midpoint of Domania's estimate was spot on. And Domania's estimate was at least partially within the 5% window in numerous other cases. But precise? Hardly.
And the winner is . . .
At $29.95 for a report, Electronic Appraiser is not cheap. But it appears you get what you pay for -- a full report with mapped (but not interactive) comparables, a rundown of census data for the county, including contact information for schools, churches and businesses -- and a roughly 50% chance that your estimate is within 5% of the likely sales price. Being right just half the time might get a human appraiser fired. Yet, in the world of online AVMs, this is top performance.
| How close are online appraisals? | | | | | | | | City | Baltimore 1 | Baltimore 2 | Boulder 1 | Boulder 2 | Cincinnati 2 | | Sale price | $85,000 | $243,000 | $939,000 | $479,900 | $470,000 | | Electronic Appraiser | $86,000 | $206,000 | $745,000 | $419,000 | $468,000 | | RealtyTrac | $83,000 | $161,000 | $732,000 | $413,000 | $418,000 | | Domania | $87,100-104,300 | $180,400-265,000 | $767,400-865,400 | $379,300-454,300 | $394,800-545,200 | | HomeSmart | $96,913 | $215,724 | $602,804 | 395,766 | $533,000 | | Zillow | $91,940 | $196,670 | $816,765 | $443,280 | $503,196 | | RealEstateABC | $120,000 | $213,000 | $911,000 | $526,000 | $525,000 | | | | | | | | City | Cincinnati 2 | Minneapolis 1 | Minneapolis 2 | Portland 1 | Portland 2 | | Sale price | $236,000 | $232,000 | $430,000 | $565,000 | $289,500 | | Electronic Appraiser | $229,000 | $217,000 | $349,000 | $540,000 | $283,000 | | RealtyTrac | $215,000 | $217,000 | $349,000 | $502,000 | $305,000 | | Domania | $211,200-219,800 | $203,900-234,500 | $331,800-458,200 | $491,500-565,500 | $294,300-312,500 | | HomeSmart | $212,116 | $227,559 | $419,627 | $344,193 | $295,026 | | Zillow | $187,222 | $221,441 | $494,127 | $482,976 | $294,925 | | RealEstateABC | $207,000 | $216,000 | $498,000 | No result | $300,000 | | | | | | | | City | Scottsdale 1 | Scottsdale 2 | Seattle 1 | Seattle 2 | | | Sale price | $1.4 million | $717,500 | $510,000 | $955,000 | | | Electronic Appraiser | $1.39 million | $675,000 | $479,000 | $963,000 | | | RealtyTrac | $1.39 million | $675,000 | No result | $965,000 | | | Domania | No result | $658,700-742,700 | $499,000-597,800 | $882,000-1,056,400 | | | HomeSmart | $1.4 million | $755,903 | $541,381 | $883,496 | | | Zillow | $1.5 million | $718,620 | $491,038 | $893,497 | | | RealEstateABC | $1.68 million | $797,000 | No result | $1.09 million | |
|
Notes: Locations, square footage, year built, bedrooms/baths: Baltimore 1: 1,241 sf, 1926, 3BR/2BA; Baltimore 2: 1,290 sf, 1928, 4BR/3BA; Boulder 1: 3,138 sf, 1920, 4BR/2BA; Boulder 2: 2,592 sf, 1959, 6BR/3BA; Cincinnati 1:NA sf, 1958, 4BR/2.5BA; Cincinnati 2: NA sf, 1992, 4BR/2.5BA; Minneapolis 1: 2,087 sf, 1950, 3BR/2BA; Minneapolis 2: 3,642 sf, 1982, 4BR/ 4BA; Portland 1: 3,160 sf, 1940, 3BR/1.5BA; Portland 2: 1,650 sf, 1963, 4BR/2.5BA; Scottsdale 1: 3,750 sf, 2001, 3BR/4B; Scottsdale 2: 2,306 sf, 2000, 3BR/2BA; Seattle 1: 1,770 sf, 2002, 3BR/3.5BA; Seattle 2: 2,850 sf, 1929, 3BR/1.75BA.
Garbage in, garbage out
The old adage about data quality applies in spades here. That's why AVMs are bound to get better. This new industry is constantly snarfing up more and better information -- from tax assessors, satellite photo records, the Census Bureau, real-estate databases, crime statistics, builders' information and lots of other public and private records. Of the 88 million homes the Census Bureau counted in 2004, said Amy Bohutinsky of Zillow.com, "we now have data on 65 million homes and Zestimates (Zillow talk for valuation) on 47 million." Housing statistics are not available everywhere, though. About a dozen states do not report home sales to the public record, adding to the challenges facing AVMs.
Think of the online AVMs as retail versions of industry models developed by scholars and researchers. Each online site probably purchases an AVM model and massages in a few more data ingredients to form its unique product, said Ted Durant, vice president of analytic services for Milwaukee-based Mortgage Guaranty Insurance Corp., the country's largest private mortgage insurer. (Read Durant's article, "AVMs -- The State of the Art.")
Here's how AVMs work. They:- Gather, in electronic form, the property characteristics used by appraisers;
- Write an algorithm that selects comparables and that adjusts their sale prices based on the differences between the subject house and the comps;
- Arrive at an estimated value.
"None of this is a precise science," Durant said. The best you can expect is not a precise number, but an accurate price range together with an estimate of the probability of its being correct.
As Gregoire, the appraiser, puts it, AVMs are a blunt instrument, as precise in predicting home value "as a wetted finger is to determine wind speed." That's because of what they don't include: view, landscaping, remodeling, condition and wear, curb appeal, fine fixtures and materials, floor plan, room sizes, neighbors and the appeal of neighboring properties, access, noise, odors, design, the owners' taste and all the other human factors that comprise a good chunk of a home's value.
The field got a big start from the work of Robert J. Shiller, the Yale University economist and housing-market expert who wrote "Irrational Exuberance, about speculative bubbles. Shiller's firm, Case Shiller Weiss, invented one of the first models, called Characteristics and Sales Analysis, the basis for many AVMs now in use.
Shiller continues to drive the AVM field: He's a principal in a company that's expected to soon sell real-estate futures on the Chicago Mercantile Exchange, letting investors hedge against movements in several major cities' housing prices.
What are they good for?
Improved AVM accuracy is good for those of us with money tied up in houses. But even now, as Durant points out, they have their uses:
- Selling your house: An AVM estimate is one piece of the mix to use in pricing your home for sale:
- Calculating home equity: AVMs help you see roughly what the appreciation of your property has added to your bottom line.
- Timing a home-equity loan: Do a reality check to find out if you have enough equity to make it worthwhile asking for a loan.
- Buying a house: Inform your sense of what you are willing to pay by checking AVM estimates for the house you want and for comparables in the neighborhood. Use AVM reports to learn all you can about the area.
- Challenge the tax assessor: In building a case that your property has been overvalued, this can be one piece of evidence. Pay for a full report to get comparables and details.
The user-friendliest
After accuracy, ease of use is the distinguishing feature. Some online AVMs are quite cumbersome. Take RealtyTrac (21% correct). It gives probably the best report, chock-full of detail on the property, its history, neighborhood, crime, taxes and census data. A good value at $19.95. But the two-page form you must fill out is neither quick nor easy. It requires details (best time to contact you, target date for your sale, reason for sale, etc.) that RealtyTrac uses to generate leads for Realtor clients. Several other AVMs -- including Domania, HomeValues.com and RealEstateABC.com -- appear to be agent lead-generators, too, but they don't charge for it.
This is why Zillow (with 29% correct), although it's not the most accurate, may be the most popular online AVM. It is simply fun to use. And since accuracy is not what these sites are all about -- yet -- we may as well have fun with them.
| Rating the AVMs | | | | | Electronic Appraiser | RealtyTrac | | Estimated value? | Yes | Yes | | Estimated range? | No | No | | Fee | $29.95 | $19.95 | | Required input | Billing info, e-mail, phone, property address. | Name, address, e-mail, phone, best time to contact, property address, desired sale price, when selling, home type, # bd & ba, sq ft, why selling, is home listed, is a real estate agent involved, credit history. | | Pros | Quick, easy. Provides value, comps, map, tax data, legal description & property history, county census, school and business info. | Outstanding report includes history, tax data, mapped comps, census data on neighborhood, crime and schools. | | Cons | Price | Too-lengthy a form. | | Accurate within 5%? | 50% | 21% | | | | | Domania | HomeSmart Reports | | Estimated value? | No | Yes | | Estimated range? | Yes | No | | Fee | Free | $24.95 | | Required input | Name, address, e-mail, phone, property address; estimated value; why you want information (curious? preparing to sell? Refi?). | Billing info, property address and zip code. | | Pros | Easy to use. | Quick, easy. Great report includes confidence rating for estimate; data on local market (foreclosure rate, median price, flipping), history, zoning, tax data comp maps. | | Cons | Slow processing, cumbersome form, often no result. | Price | | Accurate within 5%? | 57% | 29% | | | | | Zillow | RealEstate ABC | | Estimated value? | Yes | Yes | | Estimated range? | Yes | No | | Fee | Free | Free | | Required input | Property address | Property address | | Pros | Fast, interactive satellite & street maps, accuracy info. Report has legal description, history, lot size and some property details. | Fast, interactive satellite and street maps of property and comps. Report includes square-foot price, year built, description, lot size and comps. | | Cons | Report lacks detail. | Slightly slower than Zillow, report lacks detail. | | Accurate within 5%? | 29% | 14% |
|
|
