Effective Capital Stewardship with Good Quantitative Risk Assessment

BY TASHA HIGGINS, PE, PMP – Project Claims Analyst

If you are a high-ranking government official, mayor of a small town, small business entrepreneur or head of your household, understanding the importance of capital stewardship is crucial. Whether you were impacted directly by the downfall of Enron, observed from the sidelines, or were a victim of the crushing blow of the banking and real estate melt-down, you were most likely affected by a lack of good risk assessment. Poor risk assessment and capital stewardship is one of the prime causes of these economic failures and can result in a loss of confidence, trust and reputation of an organization. In House of Cards: Confessions of an Enron Executive, author Lynn Brewer discusses that poor risk assessment, or not performing “holistic” risk management, is one of the leading causes of losing trust and reputation, which can devastate the financial health and welfare of an organization.

Care and management of capital resources, for state and federal projects, is the responsibility of many stakeholders and should occur at all points along a project life-cycle. The government official, private consultant, field engineer and land surveyor, among others, have the responsibility of being stewards of funds; both public and private. They are obligated to carefully and responsibly manage the provided resources to the best of their ability and seek assistance if they lack expertise in an area that will help ensure proper stewardship.

Good risk assessment implies that more effort is spent on preventative measures and less reliance on crisis management. While acknowledging that not all risks are avoidable, taking the time early in a project to identify, quantify and develop strategies to prevent risks fosters a project team environment with the intent on ensuring effective capital stewardship. For example, the Federal Highway Administration (FHWA) is proactive in using risk management in its oversight responsibilities. On January 7, 2003, the FHWA affirmed the need to evaluate the implementation of risk assessments and provided instruction on how to create project work plans that incorporated those assessments.

It stated: "Because of the large amount of public funds involved, construction programs are inherently high-risk areas. The division office risk assessments should include an assessment of the risk associated with the State and local transportation agencies’ Federal-aid construction programs for the purpose of determining oversight priorities. Consistent with the Stewardship Policy, the primary focus should be the identification and prioritization of high-risk construction areas such that the appropriate level of division office resources can be allocated to manage the associated risk."

In ARCADIS’ experience, if a Qualitative Risk Assessment program is implemented early and monitored regularly, the benefits to the capital position, reputation of an organization, and the effect on overall project objectives are much greater.

Keys to Good Risk Management

Performing good risk management is essential to project success. Risk management first starts
with the evaluation of “risk events”. A risk event is a potential result that will have a negative
impact on the project. An opportunity event is something that will have a positive impact on the
project if it occurs. A risk management plan is a project manager’s identification and approach to handling these risks and opportunities of the project. The risk management plan should not be done at project initiation and shelved. It is a living document that is continuously monitored and revised. Figure 1 (click here to download the PDF) shows the components of the Risk Assessment Cycle. Each critical risk should be evaluated through the complete cycle at least once.

At the Qualify and Prioritize phase, any risks identified as non-critical should be placed on a list for periodic re-assessment. For maximum benefit, the Risk Assessment Cycle should be initiated at the project planning phase and be repeated throughout construction.

There are several fundamental factors that, if implemented, aid in risk assessment
and successful capital stewardship. These include:

• A well-informed and involved project team
• Full support of upper management
• Assigning each critical risk to a champion for regular monitoring
• Focusing on the key issues
• Quantifying risks to isolate and focus efforts on critical risks
• Regularly monitoring and re-assessing risks
• Independent and experienced subject-matter experts to assist the project team
• Maintaining documentation (Risk Register, Risk Management Plan, etc)
• An accurate baseline scope, cost and schedule

Good Cost and Schedule Modelings

The genesis of good capital stewardship is having a solid baseline cost estimate and baseline CPM schedule network. There are numerous methods used to develop both cost and schedule estimates. The Association for the Advancement of Cost Engineering (AACE) categorizes the methods of estimating as conceptual (or stochastic) and deterministic. They further define an estimate and estimate accuracy as:

"An estimate is a prediction of the expected final cost of a proposed project [or item] for a given scope of work. By its nature, an estimate is associated with uncertainty, and therefore, is also associated with a probability of overrunning or under-running the predicted cost."   

Some methods of obtaining project estimates include end-product units, historical data, parametric estimating, expert judgment, deterministic and more. Each of these, if used in isolation, could prove inadequate for obtaining a realistic estimate without consideration of uncertainty. Uncertainty is the estimated range rather than an exact (single) estimated value. The best form of estimating that includes uncertainty is a conceptual (or stochastic) method. It is also referred to as the “order-of-magnitude” or conceptual method which “…gives a probable estimate with sufficient accuracy to ensure that the results are meaningful for management to make the decision at hand.”

Have you ever seen a project cost estimate that allocated a 20% overrun contingency to the bottom line? How do the estimators know if 20% is enough? What if it is too much? Good capital stewardship is questionable when such large contingencies are used without reasonable justification. Using both uncertainty and risk in calculating project estimates will aid the estimator, project manager and management staff in verifying (and validating) the contingency values and creating a more probable estimate. To illustrate this further, we will examine three common estimating methods along with how their results are used.

1 - Parametric Estimating

In this method, historical data about a known item is usually used to calculate the value of the current item. This value yields the most likely cost or single-point estimate. An example is multiplying the planned quantity of an item by its historical cost per unit of that same item.

Example A: A project calls for 100 cubic yards (cyd) of imported soil and historical data shows that soil costs are $100/cyd. The total estimate for this item is therefore $10,000.

Explanation: While not incorrect, the parametric estimate does not fully allow for a range of possible outcomes, nor does it account for estimate accuracy. A more accurate representation of real world scenarios is possible.

2 - Three-Point Estimate

The three-point estimate takes the weighted average of values that are uncertain. These values are based on the assumptions and expertise of the managers providing those values.

Example B: Estimate = (min +4most + max) / 6
Using the same scenario in Example A, assume the minimum (“min”) value is $80/cyd, the maximum (“max”) value is $120/cyd and the most-probable (“most”) value is $100/cyd. The cost estimate for this item would also be $100/cyd, yielding a total estimate of $10,000.

Explanation: In this case the estimator provided a range of estimates based on historical data for the minimum and maximum and therefore the mean estimate stayed as it did in Example A. Yet, this estimate is not optimal because it gives equal weight to each value to obtain the weighted average. While this example does consider a degree of uncertainty, it does not allow for estimate accuracy. Although appearing more accurate than the parametric estimate, it results in a single outcome -- in our case, the same as in Example A ($10,000). In the real world, a range of probable outcomes will exist.

3 - Conceptual Estimating

The best form of estimating that incorporates uncertainty and the probable outcomes is the conceptual (or stochastic) method. The idea is to reach an accurate estimate that is around the range of possible outcomes. To achieve this, ARCADIS uses the Monte Carlo Simulation techniques (using line-item correlation) to develop quantitative estimates. Monte Carlo methods are a class of computational algorithms that rely on repeated random sampling to compute their results.

Monte Carlo Explained

What may seem like a complicated mathematical concept is simple at its core. Every time the Monte Carlo Simulation technique is implemented, imagine the following takes place:

1 – First, choose a model shape through which all the random points will distribute. In this case, we chose a “triangular” shape (distribution).

2 – Second, the parameters for the triangular shape must be known: a (the minimum limit), b (the maximum limit) and x (the most likely value, not necessarily the midpoint value). The triangle is placed inside a box of which you can measure the area of that triangle and the area of the box. (Diagram 1, see PDF)

3 – Third, a poor dart thrower throws darts at the box aiming to hit only within the triangle area -- assuming the dart thrower is throwing randomly, but not so poorly to miss the box altogether. Throwing darts at a triangle will yield some targets landing inside the triangle and some outside (Diagram 2). In a real simulation, dart-throwing will be done thousands of times to get a good sample of data. The more samples there are, the better the results.

4 – Finally, tally the proportion of targets that hit inside the triangle and the proportion of targets that hit outside the triangle but within the box. These computations are performed seamlessly with programs like @Risk® by Palisade and other MS Excel add-on programs.

When performing Monte Carlo techniques, the first step is to choose the distribution model to be used. There are a number of distribution models. The estimator should choose one that best fits the item based on its behavior and information. Some of the various distribution models include normal (or bell shaped), Beta (or skewed normal), PERT, uniform and triangular.

Now that we explained the basics of Monte Carlo, let’s see how it can change our example.

Example C: This example will use the values from Example B and the estimator’s ±20% values, but for both quantity and cost. Assuming a triangular distribution model shape, we ran random samples 1,000 times and produce the following data:

Explanation: The distribution curve is thus a mathematical model or “shorthand” for the actual data and facilitates the analysis. We program the model to select random values of multiple trials (1000+ times). The computer program counts and plots the results. In the example above, the count of values (or frequency of darts, y-axis) that landed in a certain range (x-axis) are shown in Figure 2 (see PDF). Finally the proportion of targets that hit the triangle is the ratio of the area of the triangle to the area of the box, known as the cumulative distribution. The cumulative distribution, or confidence curve, provides the probable outcome of the sampling.

The result of performing Monte Carlo on our model shows the overall chance of the cost value not exceeding a specific amount. In our example, Figure 2 shows that there is an 80% confidence that the item will cost between $8,090 (10th percentile value) and $12,520 (90th percentile value). The owner then has the availability of choosing an estimate and confidence that follows their organizational recommendations and guidelines. By using the stochastic process in quantifying estimates, the owner can be more confident that their values are realistic and probable. These values are not fail-safe and require continual updating, monitoring and adjusting for changing conditions. As with all estimates of time or money, the more realistic and accurate the parameters, the more accurate the projection. They are also only as good as the experts and resources used in supplying the input data; however quantifying risks using the Monte Carlo Simulation will help achieve the most realistic and probable value.

Qualification of Project Risks and Opportunities

Once a baseline estimate is validated, risk assessment begins. A risk specialist (analyst) helps the project team navigate through this process. Activity 1 in the Risk Assessment Cycle is to Identify project risks. Risk identification is a collaborative effort of the entire project team. The analyst helps the team by providing samples of risks from previous projects and facilitating the identification process. The output of Activity 1 is a list of risks; it can be brief or detailed depending on the complexity of the project. After a list of risks is identified, the team moves to Activity 2 which is risk Qualification and Prioritization. Risks/opportunities are considered qualified when the project manager, task manager, and at times, the entire project team are working in cooperative workshops and decide both the probability (likelihood) and impact of each risk. The analyst then collects each assessment, giving it a weight value and rank for each risk according to its criticality. Both the criticality descriptor and the rank number determine which risks need further assessment with quantification and mitigation. Figure 3 is the Risk Criticality/Ranking Diagram. The values are then assessed and plotted on the graph. From here, we can determine the ranking of each risk.

Quantification of Project Risks and Opportunities

Activity 3 in the Risk Assessment Cycle is the Quantification and Analysis of the cost and schedule impact. This quantification also uses the Monte Carlo Simulation method described earlier. Risk quantification is a three-step process:

Step 1 – Input Uncertainty Estimates & Choose Distribution Model
The simplest type of model is triangular. This model, and the Pert model, involves inputting the minimum value (or best case), most likely value and maximum value (or worst case) for each risk. The triangular shape can be skewed toward either end (non-symmetrical) indicating that the most likely value is closer to the minimum or maximum value. In addition to triangular and Pert, there are other distribution models that can be used to predict the impact results. These include beta, normal (or bell) and uniform.

Step 2 – Run the Model XX Times
Usually when the Monte Carlo model is activated, random samples are generated in excess of 1,000 times (or trials) to obtain optimal results. Running the model involves the simulation software randomly choosing values to input into the selected distribution model from Step 1 and calculating the resulting impact. The cost impact is tabulated for each risk and the overall impact for the entire budget is also tabulated. After running the optimal number of runs, the result’s “fit” within the distribution model upon verification. This fitting process is known as goodness-of-fit which tells “how well the data agree with the proposed distribution.” A poor fit means that there was either an inadequate number of random throws (trials) or that the random values generated skewed too far from the model shape. Figure 4 shows three models of the same random data (blue area). The triangular model shows a good fit. If results prove that the fit is not good, a different distribution model (shape) is chosen and the model is re-run. Fitting the proper model is performed seamlessly with add-on simulation software.

Step 3 – Choose Confidence Level & Analyze Results
The stochastic values generate useful information for determining risk priority. A probability or cumulative probability graph is used to assess the confidence level of each impact. A sensitivity or tornado diagram ranks and prioritizes to show which risks has the most overall impact. Figure 5 is the risk impact results of the quantification process. It shows the overall risk impact to cost or schedule. This is what will be used to both establish and justify the cost of risk for a project. Usually this graph plots several items so upper management can make an informed decision.

Another graphical output that is useful for upper management is the sensitivity or tornado diagram. This type of graph shows the prioritization of all risks graphically. The top items on the diagram are the risks that will produce the most cost or schedule impact (risk) or the most benefit (opportunity). By identifying the critical risks both quantitatively and graphically, the team can be certain to isolate which risks need to be addressed further through mitigation strategies.

Activity 4 of the Risk Assessment Cycle is the Mitigation of critical risks which includes developing risk response strategies. Response strategies explain the approach that will be taken in response to the identification of a critical risk event/opportunity. At this time, having an active and involved risk champion is very important. During the mitigation process, the risk champion is responsible for regularly monitoring the risk, assessing if the event occurrence is impending, and implementing appropriate strategies to both minimize probability and impact of the risk when possible. The champion should also be aware of ways to maximize the realization of opportunities. According to AACE, responding to risk events has four main approaches: avoidance, prevention, reduction and transference. Although not identified by AACE, it should also be noted that risk acceptance is also a valid response. Mitigating risks is the primary benefit for identifying and quantifying them. Not taking action in response to critically identified risk events increases the likelihood of experiencing their negative consequences and reduces the ability of benefitting from the opportunities. The following are some examples of mitigation methods:

Risk avoidance – Not performing a certain activity or the project as a whole because of the increased liability associated with it. This strategy minimizes the probability of the risk occurring and avoids the risk all-together.

Risk prevention – Conduct regular safety updates, inspections and meetings. This strategy can minimize both probability and impact of a risk event. The risk champion is vital. It will require consistent control to ensure mitigation strategies are performed.

Risk reduction – Enforce strict compliance to the storm water permit requirements. This strategy can reduce the impact of a risk event. The risk champion is vital for this type of response. It will require consistent control to ensure mitigation strategies are performed.

Risk transference – Hire a consultant firm with expertise (like IT) to absorb the risk or purchase
additional insurance to cover the impact.

Risk acceptance – Do nothing about the risk but accept its consequences should they occur. Mitigation strategies are crucial to the process and once developed, requires monitoring to ensure the project will benefit from the efforts. If the team identified critical opportunities, these need to be monitored as well to ensure realization and maximization of the benefit. The blue curve in Figure 4 shows the projected result of the mitigated risks and realized opportunities. As you can see, if the same identified risks/opportunities are mitigated, it will produce a savings of $1,500 to the overall risk-based estimate. Calculating a valid estimate is very important for any project budget.

If city or governmental officials knew what means and methods are available to better predict the cost of highway improvements, perhaps the stress of cost and schedule overruns would not be quite as high. By performing thorough and proper risk assessment and quantification, officials have a better view on the level of uncertainty and can develop a systematic approach to prepare for each outcome. ARCADIS believes quantitative risk assessment is the most effective means for ensuring not only a good budget, but a project that stewards all resources by the most diligent, conservative and intelligible means.

For questions or additional information, please contact the author, Tasha Higgins, at 213-486-9884 or Tasha.Higgins@arcadis-us.com.


ABOUT THE AUTHOR

Tasha Higgins, PE, PMP, has recently joined ARCADIS as a Project Claims Analyst in Los Angeles. In this role, Ms. Higgins provides technical support for our new and ongoing claims consulting assignments. She also applies her forensic engineering experience to assist clients throughout the western U.S region. Ms. Higgins brings more than 17 years of construction industry experience in the areas of construction and project management to ARCADIS. Her previous position as Transportation Engineer for Caltrans has solidified her as a risk management and capital programs expert in the field. She is also knowledgeable in roadway/traffic design and local land development. She can be reached at Tasha.Higgins@arcadis-us.com.