Can We Create More Meaningful City Rankings?

They seem to be coming out at an ever-increasing pace: rankings of cities and nations based on how livable they are, or how bicycle friendly, or how green and happy, put together by various advocacy groups, think tanks and magazines. The media loves to pick these up, and let’s face it, they’re fun. But as Alex Steffen points out in a post today on WorldChanging, they can sometimes be counterproductive.

Such rankings typically use reams of data to make their lists, and all those numbers give them each an appearance of objective authority (although in the case of magazines, the bias is usually more evident, as in Monocle’s picks for its well-heeled readership). But the data selected make all the difference. Happiness is to an extent in the eye of the beholder.

812815193_ed9994c018_m.jpgSeattle: Not as green as it looks? Photo by mandj98 via Flickr.

Steffen takes issue with the list recently released by the Natural Resources Defense Council as part of its Smarter Cities project, whose overall mission he applauds. (The list had me scratching my head a bit when it came out, in part because it scores Los Angeles ahead of New York in the transportation category.) They’ve put Seattle in first place overall, and Steffen, who lives there, isn’t having it:

I’ve explained before why I’m skeptical of city rankings to begin with:
what’s measured by these rankings tends not to be a good set of indicators of whether these cities as a whole are actually improving in any meaningful way. And Smarter Cities in particular seems to have gotten the wires crossed between its excellent mission and its flawed measurements.

Seattle, for instance, comes in at #1 in the rankings. Living in Seattle, I feel no qualms about probing into how a city with profound sustainability problems managed to make it to the top of a national ranking for "smart cities." I can tell you it ain’t pretty.

Though sustainability itself is a somewhat slippery concept, there are absolutely standards by which we can judge progress, as they mean the same things everywhere, and are pretty good measurements of overall impact. What, for instance, are a city’s per capita greenhouse gas emissions? How many miles a day do its citizens drive? How large is their average home and how compact are their communities? How much water do they use? How much energy? How much solid waste do they
generate? These sorts of numbers actually tell us something about how the people live, and about their overall levels of impact.

But Smarter Cities counts more easily measured, but sort of pointless data.…

[For instance,] "energy production and conservation" was rated by solely by the percentage of green power sources for its electricity, not total direct energy usage (much less total embedded energy usage). This means that a city like Seattle — with a highly auto-dependent population, which wastes more or less about as much energy as other Americans (more than the average Californian, and far more than the average German or Japanese) — looks great, because of the region’s abundance of hydropower, while in fact not being particularly ahead of the curve in any other way. We happen to have rain and mountains, so we’re "green,"
never mind the landfills full of dead appliances and the smog hanging in the sky.

Why should we care? Steffen continues:

The point here is not to pick on Smarter Cities (or Seattle). The point here is that unless we start defining real success (and measuring our progress in light of it), comparative measurements are worse than useless: they can even become a form of greenwashing. Many, for instance, argue that Seattle’s environmental performance (when you take away the hydro and the mild climate) is actually sub-par, but the accolades of others make it hard to hold elected officials feet to the fire over this city’s lack of density, low standards and continuing auto-dependence.

I look forward to a city ranking that does the opposite: that makes it easier for individuals to measure their own efforts, easier for citizens to judge progress, and easier for cities to set goals that might in fact make them truly bright green place to live. A truly smarter city would judge itself not by its neighbors, but by what’s
needed to save the planet.

Interesting stuff. Do any of you have examples of rankings that you find contradicted by your personal knowledge or experience? I, for instance, was surprised to see Mississippi ranked 24th in bicycle-friendliness by the League of American Bicyclists. My in-laws live there, and I’ve traveled to many different parts of the state several times. Over the course of a recent ten-day stay during the most beautiful part of springtime, I saw fewer than a dozen people riding bicycles. And when I passed a "Share the Road" sign for bicyclists, it was unusual enough that I stopped to take a picture of it.

More from around the network: Hard Drive reports on the open source philosophy of Portland’s TriMet transit agency, which has made independently produced iPhone apps available at its website. Hugh Bartling looks at a new book by Forbes writer Christopher Steiner called "$20 Per Gallon: How the Inevitable Rise in the Price of Gasoline Will Change Our Lives For the Better." And Austin on Two Wheels posts on a proposed bikeshare system that would add power back to the grid.

  • Greg

    This is a great point I think. Careless rankings not only allow officials to get off the hook with greenwashing, they also obscure genuine efforts by other cities to be greener. Hopefully at some point we’ll be able to establish more comprehensive indexes, as the author envisions.

  • Someday rankings may become deadly serious stuff. For instance, if the federal government starts fuel rationing.

  • rankings can be fun but are just a general starting point, way too many factors influence what an individual considers livable…

    to me it doesn’t matter who is #2 vs #7…but in general knowing that a city is in the top 10/20 in a particular category is a good indicator

  • Paul

    Is there a rapid transit station within 500 meters of every building in Los Angeles? Didn’t think so. Fail.

  • Larry Littlefield

    The best analysis I read on this subject was by a city planner in Planning magazine many, many years ago response to the Rand McNally places rated series.

    He divided well being into one’s own personal standard of living, the goods and services one could get, and the quality of life, the well being one receives simply by being in a community.

    The latter is squishy, but the former can be measured. He proposed that places that continued to attract in-migrants despite a low standard of living — low wages relative to housing prices — must have a high quality of life. And those that require a high standard of living to keep people from fleeing (or have people fleeing despite a high standard of living) must have a low quality of life.

    At the time, resort town MSAs were shown to have the highest quality of life. People, even people with other options, particularly young people, will take a minimum wage job and live four to a room to live in Boulder or Colorado Springs, just because that is where they want to live.

    Worst on the list was the Detroit MSA, with its (then) high wage auto jobs, low housing prices, and fleeing people.

    New York was average.

    Today? Look at housing prices (though falling) relative to income (aside from those on Wall Street), throw in taxes (aside from senior citizens who don’t pay them), and the fact that people aren’t fleeing here shows a high quality of life — despite unconstitutionally bad schools. (Data I produced back in the day showed a particular out-migration of married couples with children from NYC — single parents stayed, presumably because they couldn’t afford to leave).

    In fact, my observation is that during the past decade young people have been willing to put up with an abysmal personal standard of living to live in NYC, packed four to a room in increasingly poor neighborhoods while working for less and less wages (often as independent contractors without benefits) or (worse) as unpaid interns. And my reason is that the demand for places that provide NYC’s particular combination of characteristics far exceeds the supply.

  • John Allen Paulos, in his 1997 book A Mathematician Reads the Newspaper, provides the best demolishing of the entire ranking industry. The whole exercise is entirely subjective, of course.

    Steffens writes: “Though sustainability itself is a somewhat slippery concept, there are absolutely standards by which we can judge progress, as they mean the same things everywhere, and are pretty good measurements of overall impact.” Standards don’t mean the same things everywhere. The ones he describes are perhaps more dependent on economic activity than on sustainability.

  • the demand for places that provide NYC’s particular combination of characteristics far exceeds the supply

    Oh, absolutely. Every city in America used to provide the same “urbanity” that NYC still provides. Now only a handful do. Smaller cities should be attracting young people by capitalizing on what’s left of their urbane qualities instead of imitating the suburbs like the small city I left (Buffalo).

  • PS. I used to *inhale* the Places Rated Almanac when I was a kid… what a geek.

  • Thank you for your comments and criticism and for the opportunity presented here to discuss ways to improve Smarter Cities’ research. As background, this year’s Smarter Cities report was conducted independently of the NRDC and only recently found hosting at the NRDC website. In the sections below, we describe the criteria and the scoring methodology used to develop the 2008 rankings. A Smarter Cities wiki for this discussion will be in place by the beginning of August. We encourage you to participate in what promises to be a valuable process.

    For our current scoring and criteria, please see “How We Scored Cities” and “Data Sources and Point System” at We appreciate all the suggestions for criteria that we have received and are finding measurements of per capita power consumption, CO2 output, waste output and other measurements. The Smarter Cities wiki will serve to gather comments and suggestions on ways to improve the criteria and study methodology for the next research round. All responses are welcome.

    Although much of our data was derived from the EPA, the Census, the DOE and other comprehensive sources, survey responses did have a significant result on scores in several criteria, as noted in the Data Sources section. While our survey response rate was a fairly high 24.3%, we want to push for much higher response rates in the years to come.

    Last year was the first year we considered cities with populations below 100,000, which added over 400 cities to our list but presented a problem in locating comprehensive data sources that included them. For energy production and conservation, we used the survey to provide us more precise data than the state-level energy consumption provided by the Department of Energy. We asked about the top three fuels used for power generation, energy conservation incentives, and if the utility provided a renewable power option. Seattle’s reliance for 90 percent of its energy on hydroelectric power helped its score and it also indicates reduced per capita carbon output for residential and commercial electricity usage. In 2002, the Energy Information Administration reported that Washington residents released .25 lbs of CO2 per kilowatt hour, while New York State residents released over three times as much at .86 lbs/kWh. Granted, that isn’t per capita data for Seattle or New York City residents, but indicates that even if Seattlites consume three times the electricity per capita of New Yorkers, they would still produce less CO2.

    For transportation emissions per capita, thanks to the Center for Neighborhood Technology we now have CO2 emissions from transportation per household for Seattle and New York. Here, we can see that for much of Seattle itself that CO2 emissions are in a range from 0 to 6.5 metric tons per household. For much of New York City, the household emissions are in the 0 to 3.3 metric tons range, about half the amount of Seattle households.

    One additional point I would like to make is that the ranking was never intended to be an end in itself but a means to identify the full array of leader cities and to cast a spotlight on the innovative programs they are implementing to make their towns more efficient, cleaner, more just and more livable. As a new media resource, it is our aim for Smarter Cities to be a destination site for all who have a stake in their city’s future—from students to policy makers and city planners, from business leaders to community groups and urban dwellers—to learn about best practices and initiatives, share ideas and innovative solutions, ask questions and get answers and find out what’s smart near them, from farmers markets to certified solar panel installers.

    We invite your participation and would like to hear your thoughts, criticisms and suggestions when the Smarter Cities Wiki is launched live in early August or at
    Paul McRandle
    Consulting Senior Editor
    Smarter Cities

  • Although the conversation has wound down here, as a follow-up I wanted to mention that the Smarter Cities CityWiki has launched at We invite your participation in shaping and strengthening our criteria, methodology, data sources and survey questions as we prepare for our next research round. We would also be happy to hear about sustainability efforts in cities worldwide.



Is Charlotte the Least Walkable City in America?

Walk Score just released its annual list of the nation’s most walkable cities, and the top of the list generally looks like what you’d expect. New York, San Francisco, Boston, and Philadelphia took the top four spots, though our friends at Transit Miami did take strong exception to their hometown rating right up there in fifth place. All […]