Fire-Walking: Ensuring Credible Economic Impact Studies
 

Economic impact analysis is not a matter of applying a multiplier factor to make the benefits of a project look bigger. Rather, economic impact analysis is a means for identifying the nature of changes in jobs, personal income and business activity that can occur in a given area as a result of some project, program or policy. The net actual impacts of a given project can in fact be significantly smaller or larger than the project alone, as is illustrated later in this article.

The need for economic impact studies is real. Economic developers and public agencies have a vested interest in seeing benefits maximized for their communities. As a result, there is a continuing need to assess the positive and negative economic impacts of proposed economic development strategies and projects. The complication is that the method selected to assess economic impacts, and the type of information applied, can determine the usefulness of the final assessment. That is not to say the most carefully constructed study will necessarily give you the answer you are anticipating – but even valid surprises can be informative and valuable.

We know that economic impact analysis can be an important tool for assessing project/policy impacts. Many local and state agencies, as well as private companies and developers currently use it to help assess the value or benefit of facilities and projects. However, it can also backfire and taint the credibility of an agency or a professional. A badly done study, or one that is misinterpreted, unfortunately lives on to embarrass! A credible study on the other hand, can help address the justification of appropriate public or private investments and also help us to better understand their implications.

Like fire walking, if you undertake an economic impact study ignorant of the pitfalls and what the analysis is all about, you may not emerge from the public debate pit unscathed! Yet it is possible to avoid getting burned. Computer based economic impact studies have now been around for several decades. While methods are continually being improved, we can still learn much from past experiences. As analysts, we have played roles conducting impact studies, advising others on appropriate applications, critiquing other studies and reviewing the literature on available methods. Building upon those experiences, this paper presents cases which highlight pitfalls of economic impact studies, as well as illustrating recommendations on how to avoid them.

It’s All in The Approach

Economic impact studies have been criticized in part because of confusion over what they encompassed and how they should be interpreted. There is a simple solution to ensuring a successful reception of your analysis findings by your audience – avoid “black box” modeling. This pertains not only to the economic analysis tools you choose but how you approach the analysis. “Transparency” is paramount in each stage of building the impact analysis. Your audience must understand the choices and assumptions you made in assembling the analysis. Credibility hinges on three major points: (1) defining the issue, (2) conducting the analysis, and (3) presenting your findings. Let’s look into each of these.

(1) Define the Issue Appropriately

To define the issue appropriately, you must be sure that you are clear on (a) definition of the project, program or policy whose impact you want to measure, and (b) definition of the perspective for measurement. That involves clarification of two questions:

• What is the question that your analysis is intended to answer ? Are you trying to describe the economic role of an existing activity in your region’s economy or identify the net impact of a proposed future project, program or policy? Many analyses get a luke-warm reception because they were initiated on the wrong premise. For example, a community was considering an investment to expand its convention center facility. An agency performed a study of the importance of the existing convention center by measuring the direct and indirect impacts of the existing operation. Based on its current usage, it reported that the facility was directly or indirectly supporting 2,700 jobs in the region. That is all well and good, but this approach failed to make a persuasive case for investing in expansion of the facility because it failed to portray what would be difference between the future situation with the investment and without it.

In this case, the type of analysis that was conducted was not appropriate for justifying a proposed new investment, though it could have been useful if the purpose was merely to educate the public about the existing role of the facility in its local economy. That kind of study is commonly done for local airports, public transit agencies , and universities.

• What is the relevant base case, and what is the project definition ? If the analysis is intended to comparing the incremental impact of some project, program or policy, then there must be some basis for comparison. Thus it is important for the analyst to clarify the following questions: (a) Is the event a one-time project or a a change with continuing consequences? (b) Is the proposed new project/policy to be viewed relative to the existing situation now, or to a future situation in which current policies or conditions continue to apply? (c) Would failure to implement a project/policy lead to a continuation of the status quo for the community, or would it represent deterioration in current conditions? (d) Conversely, would the proposed project or policy enhance current economic conditions, or would it just maintain economic performance by avoiding business loss? The analyst must ask each of these questions and answer them as explicitly as possible since this can affect the study area and choice of economic analysis tool.

----------------------------------------------------------------------------------------------------------------

Case No 1: Picking the Right Base Case (An Airport Expansion Project)

The Context: A community was faced with an airport proposal, which is intended to increase airport capacity by 35% over the next ten years. This proposed expansion was justified as necessary to meet current growth projections.

To assess project impacts, the analyst was asked to compare regional economy with airport expansion to the corresponding regional economic conditions without the project. Since the proposed project impacts would differ over time, the analyst chose to use a regional impact forecasting model that has capabilities to forecast alternative outcomes (see below on Enlisting the Correct Model). That model required the analyst to compare a ‘base case’ forecast of regional economic conditions, against an alternative scenario forecast.

The Simple Way to Model Impacts: Initially, the analyst thought the correct thing to do was to measure airport growth in terms of additional growth in passenger volumes and dollars they would spend, and then apply regional multipliers to calculate the additional economic expansion (beyond the base case) in terms of jobs, income and business output in the region.

The More Credible Way to Model: The ‘simple way’ stated above was flawed. The proposed airport expansion was actually necessary to meet the aviation growth requirements associated with already expected regional economic growth. All base case forecasts of future economic growth actually assumed (implicitly) that current air accessibility and transportation level of service conditions for customers (travelers) would continue in the future much as they are now, rather than deteriorate. In this case, the economic impact analysis should actually have focused on estimating how much the region would lose out, if the investment isn’t made. The difference between gaining extra jobs .vs. losing existing business can play out differently for your audience!

----------------------------------------------------------------------------------------------------------------

(2) Conduct the Appropriate Type of Analysis

To use the appropriate analysis method, you must be sure that you are clear on: (a) your selection of the right tool, and (b) the way that you apply that tool. This involves two requirements:

• Enlist the appropriate impact model. If you decide to perform a formal economic impact analysis you will need one of these. There are (1) static and (2) dynamic models. Picking from among these two types of models is a major decision that can affect the success of your analysis (not your project). The model decision should be based on the nature of the questions to be investigated. Other factors, such as budget, in-house resources, deadline and ease of use, often come into play.

The static models are input-output frameworks. One is the RIMS-II (Regional Impact Multiplier System) system ( www.bea.gov/rims2 ). It provides regional job, income and business sales multipliers for indirect, induced & total effects associated with the proposal’s direct effects. Another is IMPLAN Pro ( www.implan.com ). Like RIMS-II, it is calibrated for your region of choice. However, unlike RIM-II, it also provides the user with an automatic calculation of the overall impacts of a project, and it provides the user with latitude to alter default settings (regarding local capture rates, retail margins, commuter income flows and production functions) before obtaining the multipliers. All of these input-output models, have limits in terms of the type of project information they can handle. There is no time component to the results; instead these models create a snapshot of the impacts. As such, these models would be very well suited for measuring the economic role of some existing activity of your current economy such as a municipal airport or a factory, or a one-time event such as a summer festival.

On the other hand, suppose you are analyzing a program of proposed infrastructure investments for your region’s future growth. It is very likely the stream of impacts will vary over time due to the growth realized in any particular year and changes in capacity utilization of facilities. This is one example where a year-by-year analysis is needed. If the project or program is also expected to affect relative costs of doing business and cost-competitiveness relative to other areas, then you need some type of model that can also assess how that will affect business location shifts and population movements. In that type of situation, a dynamic model is warranted because impacts are time specific.

These models are also termed simulation models. They typically have an input-output structure embedded at their core but also include numerous behavioral equations (to varying degrees depending on the vendor) to describe more about key interactions in the regional economy than a static model can. For example, the REMI model ( www.remi.com ), possesses a large policy variable set that allows the analyst to feed the model a great deal more project information beyond just jobs or sales. This modeling approach is particularly well suited when cost competitiveness and population changes are thought to be crucial parts of the impacts, since static models cannot directly address either of those two factors.

• Do your Homework in Advance. Now that you’ve identified the questions to investigate and chosen a model appropriate for your analysis. What next? Read the model’s “Operator’s Manual” because even a sound model run poorly leads to a shoddy analysis.

Economic impact models require thoughtful use. Economic models attempt to represent how an economy is structured and how it might respond to certain changes ceteris paribus. You may need to consider how closely the model describes the current events (e.g. a change in labor market conditions) of your region even before thinking about your impact analysis. You can ultimately still go wrong with your impact analysis if you fail from the start to recognize some key, current realities (such as the existing unemployment level, which can affect how much new population may be attracted by a new plant). Likewise since models built on an industry level of aggregation are apt to describe the workings of the average firm, the impact of a specific type of new firm activity may not necessarily match to the average firm in that industry category. All of these issues can be directly addressed and handled in the economic impact analysis, if they are identified in advance. It is thus important to do some reconnaissance on the critical issues prevailing in the region and especially over your project. Then assess your ability to fine-tune default settings/ assumptions in the model to take this information into account.

 

----------------------------------------------------------------------------------------------------------------

Case No.2: Inputting Full Information (A Casino Proposal)

The Context: A casino/resort development was proposed in a depressed city that suffers from low wages and falling population. Certain interested parties believed this project could help correct these problems. Casino management promised to hire a portion of its staff from the inner city and also promised to purchase a significant portion of goods & services, for both construction and operation’s phases, from local and statewide businesses. Since the city’s labor market area extends well beyond its county, and the county had a high cost of housing, the project was expected to spur more commuting rather than population movement for new jobs not filled by county residents. The market study indicated that 10% of the proposed casino’s revenues would come from other casinos in the state.

The Simple Way to Model: A set of county I-O multipliers, applied to either the casino’s projected revenue or payroll, for the complete build-out year, will identify the gross impacts on jobs and personal income occurring within the county.

The More Credible Way to Model: The flaws in the simple approach are numerous. These stem from certain assumptions as well as omissions in pursuing a static multiplier approach without recognizing likely shifts in wage rates and commuting patterns. If you have ample information about a project, then make sure you’re not constrained in putting that information to work because of the limits of a particular model.

All impact models carry default assumptions based on prevailing regional and industry trends. Inform yourself regarding the default values, including average wage levels for the affected industries and assumptions about the portion of earnings (generated in the county’s workplaces) that flows out of the region with commuters from outside the county. In this case, the following changes needed to be made to the REMI model’s default assumptions for the county being studied:

· Wages – The casino’s average wage is higher than the typical business in the Amusement & Recreation, Hotel and Eating & Drinking industries. The REMI and IMPLAN models allows for adjustments to recognize higher than average earnings for the specific project without inflating the pay of all workers in the affected industries.

· Workers commuting from outside counties - The REMI default for the county considering the casino is 20 percent leakage of income to outside residents. We increase this by 10 percentage points based on information from local experts and the city’s location in the county, its labor market area and the county’s cost of housing.

· Population - The standard I-O framework (e.g. IMPLAN and RIMS-II) does not address population changes. In REMI, the economic signals (job and wage growth) lead to forecasts of population changes by affecting economic migrant flows. We can adjust the REMI model’s default response rate as needed to represent an expected reduction in the county’s rate of population loss of working age residents.

· Displacement of Existing Business - The base assumption in most impact analysis is that 100 percent of the projected new business sales is from new spending to the county. In reality, the casino could displace some spending on other types of purchases (e.g. other forms of entertainment or retail spending) occurring in the county or state. For this analysis, we include a spending offset for casino activity already occurring elsewhere in the state, equivalent to 10 percent of the new casino’s projected revenues.

----------------------------------------------------------------------------------------------------------------

 

(3) obtain Data for the Appropriate study area and Time frame

To obtain the appropriate data, you must be sure that you have identified: (a) the relevant study area, (b) the relevant time frame for study, and (c) way that you apply that tool. This involves clarification of three issues:

• What is the relevant study area for analysis ? The study area defines the geographic boundaries over which the impacts will be measured. This should normally match to the jurisdictions of interest. For example, a metropolitan planning organization conducted a study of the benefit to the metro area of a proposed retail shopping center. This was fine for its own uses, but failed to be useful when the agency requested financial assistance at the state-level and its study was unable to distinguish how much of the new business activity was new to the state and how much of its was merely a shift of activity from elsewhere in the state.
• What is the relevant time frame for analysis ? Equally important is an understanding what the time frame of the analysis. For instance, if a project is characterized by several different phases (e.g. construction versus operations) then the analysis should span enough years to reflect the transition to a complete “ramp up” of the operation. Likewise if a project is financed in part by public debt then the time frame should encompass the ramifications of that debt maturing as part of the net impact assessment. The debt payments can be viewed as an annual impact over each year of the analysis time frame. (If the term of the debt repayment exceeds the time frame of the analysis model, then the remainder can be expressed in terms of its net present value in benefit-cost analysis.)

Time frame is also crucial in evaluating the economic impacts of any project/policy change that affects market costs or competitiveness for firms and households in the region. As will be discussed in the next point, the analyst may first need to use a model that can handle such labor-capital market and supplier interactions and then make sure to run the analysis over enough years to allow the model to show the impacts of these market adjustments.

• Fueling the Model with Real Project Data. Most models come equipped with a background regional database on current economic patterns and inter-industry purchasing relationships, but you must provide inputs to the model which represent the subject of analysis – correctly! This means taking inventory of what you know about the proposed project/policy or an existing activity under examination. You must then consider how this project data fits into the model. Think about what holes exist in your project data and what you can do to fill them. Whether the data has been handed to you or you “collected’ it using different methods, you must first, apply the “laugh-test” and second, take time to benchmark the project data against some comparable situation already in existence. This can save time and frustration when reviewing the results later on.
• You want to ensure you don’t double count direct effects (which are input into the impact model), such as business sales in retail sectors and tourist spending on these same items. There are model specific details to get acquainted with such as ‘am I entering the project data as ‘changes’ from my base case year-by-year’; ‘if I enter salary data, am I affecting the region’s entire industry in the model or just the earnings at this one business’; ‘do I need to adjust the base year dollar concept of my data before introducing it to the model’.

----------------------------------------------------------------------------------------------------------------

Case No.3: Distinguishing Net vs. Gross Impact (A Scheduled Plant Closing)

The Context: A community is facing an upcoming, permanent plant closure. A recent but persistent shortage of skilled occupations in the county has been constraining growth of a competing firm. As a result, a significant number of skilled workers affiliated with the plant closing will be reabsorbed by a neighboring firm in the county.

The Simple Way to Model: You go straight to the plant-level data and ask ‘how many jobs’. You enter this job loss into any model and the respective multipliers tell you at minimum, total job, income and sales lost in the community as a result of the closure.

The More Credible Way to Model: The flaw above has nothing to do with the model. The analyst, in failing to look beyond the obvious details of the business in question, missed out on some important mitigating conditions (unknown to the economic model) in the local economy. In this case, the upfront job loss is less than the number of jobs at the plant. It is this modified number that you introduce into your analysis. If the skilled worker’s new job is in a different industry (using a similar skill set) make sure wage and productivity differentials also factor into your analysis.

---------------------------------------------------------------------------------------------------------------

 

Gauging the Results. If the data preparation stage is the most time consuming aspect of your analysis, then reviewing your results is the second. If you’re seasoned in developing impacts analyses then perhaps you can quickly judge your results for reasonableness. Remember - different models not only take different amounts and types of inputs but their feedbacks and the ability to make adjustments also vary. Barring a data entry error in one instance, this is primarily due to structural differences in the various modeling frameworks: how much project data the model could accept in the first place and what gets done to that information once inside the model. So take nothing for granted!

Perhaps you have little experience with a particular model. Review the model’s results for the indicators that you think are most consistent with the goal of the project/policy – is it job creation or higher income. You most likely have some expectation (and basis) on the direction and perhaps size of the impacts. Ask yourself “do these results make sense” and “can I explain these convincingly to an interested, lay audience”. If your expectations are at odds with the results then you must examine why.

You can do an ‘internal check’ and an ‘external check’. The internal check asks ‘are the base case data (e.g. industry wages, jobs & sales) in the model in line?’ Sometimes the data are unreliable if they are collected at the county-level and the county has sparse economic activity or is thinly populated. The external check benchmarks your results against a comparable existing project to see if they are in the ‘realm of believable variation’.

Present Convincing Findings to Your Audience

Remember Transparency. So much of your effort has gone into constructing a careful analysis yet the ‘what & how’ of your presentation will also affect how the study is perceived. Your presentation should provide decision-makers and stakeholders with the intention of the project and study (include a description of the project), an understanding of your analysis method (assumptions & limitations, what went into the analysis and the consequences of what did not), and the findings (the relevant set of results along with a convincing and correct interpretation). The findings are where you get to tell the story based on the initial project information and feedback mechanics within the economic model. At minimum you can address the project’s direct effects, the supplier purchases effect and the worker’s re-spending effects. This is possible with an IMPLAN, PC-IO, or REMI model and the RIMS multipliers. A dynamic model (e.g. REMI) used for the analysis will also be able to address price effects and population impacts.

Ring True and Balanced. You should focus on indicators that help gauge whether the goal of the proposed project was met or not. Don’t bury your main message with other modeling information and economic data that are tangential to the main issue. Also it is your job to help the audience navigate through the results and reach conclusions that are consistent with your analysis. You can report sales and income impacts separately but make sure everyone understands it would be double-counting to combine these dollar concepts.

If you had reason earlier in the analysis to define the study area as reaching beyond the area of direct impact, say to include the rest of the state, then address the key findings in that region as well as the aggregate area. Make sure you convey the difference between the gross impacts (on the local area) and net impacts (extending beyond the local area.) Even though the project’s impacts may dwarf in comparison to the backdrop of a statewide economy that doesn’t mean the findings don’t spell ‘success’ at the local level. Once again, the findings must be interpreted with respect to the project goals.

Conclusion

You can avoid falling into situations like those shown here where the credibility of your economic impact analysis is doubted. First , be aware of how the results are used. There are different constituencies or stakeholders for specific areas of geographic jurisdiction as well as financial jurisdiction. Your results can potentially be presented to emphasize current versus potential future impact, local versus broader area impact, and/or the project effect versus alternative uses of investment money.

Credibility and usefulness should be the goals of your analysis efforts. Before your analysis is completed you should understand the counter concerns and be able to address them adequately. Know your process and results better than anyone else in your audience to avoid embarrassment. And remember – the outcome has more to do with the modeler than the model!