Caution: the Tax Foundation's State and Local Tax Rankings are Unreliable
August 13, 2008
As it has annually for several years, the Tax Foundation has attempted to measure the current impact of state and local taxation on the residents of each of the 50 states. And once again, the results are very different from the Tax Foundation's own previous attempts to do this calculation. The Tax Foundations figures -- both the national aggregates and the rankings for individual states -- differ markedly from the estimates released in 2007, which in turn were very different from the estimates in previous years. The fact that the Tax Foundation revises its results with such frequency calls into question how robust its methodology is -- and how seriously the estimates should be taken.
- The Tax Foundation has abandoned its 2007 headline finding that taxes were at a 25-year high. In the current report the Tax Foundation says state and local taxes in 2007 were lower than in the mid-1990s and about the same as in several other periods in the past. (Taxes in fiscal year 2008 were even lower than in 2007, the Tax Foundation says.)
- The Tax Foundation has revised downward by more than a full percentage point its estimate of the share of income Americans pay in state and local taxes. (This appears to result from a major change in how the Tax Foundation calculates "income.")
- Many rankings have changed. The Tax Foundation no longer asserts, as it did in 2007, that taxes in Vermont were the nation's highest in 2006 and 2007; Vermont's rankings for those same years have now fallen to 4th and 6th (and fallen further in 2008 to 8th). Similarly in Maine, the ranking for 2006 and 2007 has gone from 2nd in each of those years to 7th and 14th respectively (and to 15th for 2008). Neither state has ever ranked higher than 4th, the new report says.
- The Tax Foundation now says that residents of New Jersey (rather than Vermont) paid the highest state and local taxes in fiscal years 2006, 2007 and 2008. But as recently as last year, the Tax Foundation said New Jersey has never ranked higher than 10th. (Note, by the way, that the Federation of Tax Administrators, using a more commonly accepted methodology of simply dividing total tax collections by total personal income, finds that New Jersey's state and local taxes ranked 10th in 2006, the most recent year for which actual data are available.)
- The Tax Foundation's assessment of other states is equally inconsistent with past years. Last year, the Tax Foundation indicated that Louisianans have paid relatively high taxes since at least 2002, with rankings between 6th and 20th this year, the Tax Foundation contends that Louisiana's rankings this decade have been fluctuating between 29th and 42nd. Other states' rankings have changed dramatically as well.
No state has altered its tax policy in the last year to a degree that would produce anywhere near such changes in rankings.
Such dramatic revisions are common for the Tax Foundation's state-level estimates, as a previous CBPP report described in detail.  These fluctuations show that any given state's ranking has much more to do with the Tax Foundation's methodological adjustments and with fluctuations in the data than it has to do with the actual level of taxes paid by people who live in the state.
It is hard to know what to make of these changes. But it is extremely unlikely that the Tax Foundation has, this year, for the first time simply fixed all the problems with past years' rankings. As in past years, the Tax Foundation still uses a mix of actual data and projections, in substantial part derived from years-old data and from national samples that were never intended to be used for state-by-state estimates. It is simply not possible to produce meaningful rankings of state and local taxes for the fiscal year that just ended five weeks ago, because data aren't yet available for every state, much less for the tens of thousands of units of local government all over the country, that would be needed to rank the states with any reasonable degree of confidence.
The Tax Foundation further continues to blur the actual data with extensive adjustments made by the Tax Foundation that are supposed to reflect the extent to which one state's residents pay taxes to other states. (The methodology behind these projections and adjustments is described in a separate new paper on the Tax Foundation's website. There is no evidence that this methodology has ever been formally submitted to outside review, and the paper confirms the extent to which the Tax Foundation's model uses a range of data sources, very few of which are up-to-date at the state or local level through fiscal year 2008. ) Although these adjustments have a conceptual basis, they introduce yet another source of uncertainty.
In short, both historical examples and common sense tell us that the Tax Foundation figures for state and local taxation are likely to turn out to be wrong, and therefore should not be relied upon.
The Tax Foundation has published on its website a sharply worded response to this analysis. Its response should not increase anyone's confidence in the Tax Foundation's state-by-state rankings.
- The Tax Foundation argues that in producing data heavily reliant on projections and samples, it is merely following the practice of the U.S. Census Bureau and other federal agencies and independent research organizations. But when revisions and uncertainty are large, federal agencies always explain that fact and document it. The Congressional Budget Office, for instance, publishes a detailed review of its past forecasts to assess their accuracy. The Census Bureau includes estimates of statistical error in order to allow state-to-state comparisons to be made in an appropriate way. The Urban-Brookings Tax Policy Center labels its tables as "preliminary results," indicating they are subject to revision. The Tax Foundation, by contrast, has never documented the extent to which its own past estimates of state tax levels have been revised, nor included a measure of the likely error in its figures. Such information would be needed for anyone trying to understand whether the 2008 figures are likely to be reliable. Moreover, while the Tax Foundation indicates in its actual report that 2007 and 2008 numbers are estimated, it is far less careful to do that on its web pages for specific state data. And the term "estimate" or "projection" does not appear at all in its press release, which undoubtedly is in part responsible for the widespread reporting of the rankings as "facts" rather than projections.
- The Tax Foundation argues that this year's changes to state rankings and findings of tax levels, compared to previous years' reports, should not be surprising because the report's methodology has totally changed. This may be true (although the Tax Foundation does not document the extent to which methodological changes account for the changes in findings), but in any case this argument should not increase a reader's comfort level with the Tax Foundation results. The actual state-by-state data which are being fed into the Tax Foundation model are still several years old; there is no state-by-state 2007 or 2008 data available for local government revenues or for various other measures used by by the Tax Foundation. So it remains the case that when actual data for 2008 are released, the Tax Foundation's estimates of 2008 tax levels will probably be different - and, if history is any guide, probably quite different.
- The Tax Foundation argues that using years-old data as the basis of a statistical model is acceptable practice. This is true if there is strong reason to believe that when more recent data are published, the findings will be essentially the same, or if the lack of certainty is clearly acknowledged. But neither is the case here. We know from experience that the Tax Foundation is likely to change its rankings for any given year, possibly quite substantially. Yet this fact is never clearly documented in Tax Foundation reports.
- The Tax Foundation accuses the Center of describing the Federation of Tax Administrators' measure of tax levels as "superior" - a misquote. In fact, the Center merely described FTA's calculation of tax levels (tax collections divided by income) as "more commonly accepted." There are legitimate arguments as to whether FTA's measure of tax levels is, in fact, a useful measure of the impact of taxes in a given state on the state's residents. But it is incontrovertible that this is a measure that has long stood as one of the standards of state-by-state comparisons of tax levels; the Census Bureau itself used to publish exactly the same statistic, as did the now-defunct Advisory Commission on Intergovernmental Relations and the National Conference of State Legislatures; the Rockefeller Institute of Government and the Urban-Brookings Tax Policy Center continue to publish the same calculation.
- The Tax Foundation correctly points out (and the Center's analysis has been revised to acknowledge) that it has now produced a 30-page report on the methodology behind its model, although the Tax Foundation continues to acknowledges that the methodology has not been subject to outside review. But nowhere in the 30-page methodology can one find any analysis of the likelihood that the latest year's rankings have a degree of uncertainty and may turn out to be incorrect, which is the major thrust of this critique. The important question is not whether the Tax Foundation has used the best available data and methods to produce its findings; the important question is whether those findings - particularly the widely reporting state-by-state rankings - are meaningful or useful.
- Finally, the Tax Foundation in its response to the Center emphasizes that its figures are "estimates" and therefore, by implication, should not be expected to be absolutely correct. But absolute correctness is indeed the impression that the Tax Foundation repeatedly seeks to convey in its most public materials. Just as one example: At the top of the Tax Foundation's website yesterday, August 12, 2008, the bold headline read: "New Jersey Edges out New York for Nation's Highest State-Local Tax Burden." Neither that headline, nor the text below it, conveys even the slightest uncertainty about whether the calculations are correct. The Center's repeated analyses of the Tax Foundation's reports on this topic have proven that there is a great deal of reason to doubt that the Tax Foundation's findings will hold up over time, even by its own measures. Were the Tax Foundation to acknowledge prominently in all its published materials that its findings on state and local tax levels are preliminary estimates that may well turn out to be substantially off the mark, and were it to prominently discuss the sources of uncertainty and the track record of its estimates, there would be little or no need for the Center to highlight the data's lack of reliability.
 The 2008 report, entitled "State-Local Tax Burdens Dip As Income Growth Outpaces Tax Growth," is available at http://www.taxfoundation.org/files/sr163.pdf. The 2007 report, entitled "State and Local Tax Burdens Hit 25-Year High," is available at http://www.taxfoundation.org/files/sr153.pdf.
 The extent of these revisions is particularly dramatic because last year's estimates were supposed to reflect findings for calendar year 2007, while the latest findings (due to a change in Tax Foundation methodology) reflect fiscal year 2008, which in most states actually began in July 2007 - that is, creating a six-month overlap. With such an overlap, one would expect little change in tax levels or rankings.
 Note that some of the changes to methodology employed by the Tax Foundation this year seem likely to make state by state estimates even less reliable. For instance, for the first time, the Tax Foundation asserts that it is adding a figure for capital gains realizations to its measure of income. But the latest publicly available state-by-state data for capital gains realizations is for tax year 2006, so it is unclear how the Tax Foundation generated these data.