LM05 Backtesting and Simulation 2025 Level II Notes

LM05 Backtesting and Simulation

1. Introduction ........................................................................................................................................................ 2

E
2. The Objectives of Backtesting ...................................................................................................................... 2
3. The Backtesting Process ................................................................................................................................ 2
3.1 Step 1: Strategy Design ........................................................................................................................... 3
3.2 Step 2: Historical Investment Simulation ........................................................................................ 5
3.3 Step 3: Analysis of Backtesting Output ............................................................................................. 6
back
4. Backtesting Multifactor Models .................................................................................................................. 9
Testing 4.1 Step 1: Strategy Design ........................................................................................................................... 9
4.2 Step 2: Historical Investment Simulation ..................................................................................... 10
4.3 Step 3: Output Analysis ........................................................................................................................ 10
5. Common Problems in Backtesting .......................................................................................................... 13
5.1 Survivorship Bias ................................................................................................................................... 13
5.2 Look-Ahead Bias ..................................................................................................................................... 13
5.3 Data Snooping .......................................................................................................................................... 14
6. Historical Scenario Analysis ...................................................................................................................... 14
-
7. Simulation Analysis ...................................................................................................................................... 16

simm
7.1 Historical Simulation ............................................................................................................................ 18
7.2 Monte Carlo Simulation ....................................................................................................................... 21
8. Sensitivity Analysis ....................................................................................................................................... 22
Summary................................................................................................................................................................ 24

Required disclaimer: IFT is a CFA Institute Prep Provider. Only CFA Institute Prep Providers are
permitted to make use of CFA Institute copyrighted materials which are the building blocks of the
exam. We are also required to create / use updated materials every year and this is validated by CFA
Institute. Our products and services substantially cover the relevant curriculum and exam and this is
validated by CFA Institute. In our advertising, any statement about the numbers of questions in our
products and services relates to unique, original, proprietary questions. CFA Institute Prep Providers
are forbidden from including CFA Institute official mock exam questions or any questions other than
the end of reading questions within their products and services.
CFA Institute does not endorse, promote, review or warrant the accuracy or quality of the product
and services offered by IFT. CFA Institute®, CFA® and “Chartered Financial Analyst®” are
trademarks owned by CFA Institute.
© Copyright CFA Institute

Version 1.0

© IFT. All rights reserved 1

LM05 Backtesting and Simulation 2025 Level II Notes

1. Introduction
This reading provides an overview of four methods used to evaluate investment strategies:
 Backtesting: Tests a strategy in an actual historical environment, usually over a long
period of time. Tells us how a strategy would have performed if it had been
-

implemented in the past.

 Historical scenario analysis: Tests a strategy in a specific historical environment,
usually over a short period of time, such as during recessions or periods of high
-

inflation.
 Simulation: Rather than actual historical environments, simulation tests a strategy in
a hypothetical environment specified by the user.
-

 Sensitivity analysis: Determines how changes in input variables affect a target

variable and risk profile.
2. The Objectives of Backtesting
Backtesting approximates the real-world investment process by evaluating whether a
strategy would have produced desirable results using historical data. It can be used to decide
if an investment strategy should be accepted or rejected.
Backtesting implicitly assumes that the future will be similar to the past. However, this
assumption may not hold, and complementary techniques (discussed later in the reading)
are commonly used to account for the randomness of the future.
3. The Backtesting Process
Backtesting consists of three steps: strategy design, historical investment simulation, and
analysis of backtesting output.
Exhibit 1 from the curriculum illustrates these steps:

-
-
-

© IFT. All rights reserved 2

 LM05 Backtesting and Simulation 2025 Level II Notes

3.1 Step 1: Strategy Design

In this step we identify the investment goals and hypothesis.
Typical goals for active strategies are:
 To achieve excess returns over a benchmark
-

 To achieve superior risk-adjusted absolute return

-

- An investment hypothesis is a method designed for achieving the goal. Key parameters
-

defined under an investment hypothesis include:

-

 Investment universe: This term refers to all of the securities in which we can
① -

potentially invest. The constituents of well-known broad market indexes (for

-

example, the Russell 3000 Index) are frequently used as the investment universe.
-

& Return definition: If the investment universe spans multiple countries, we must
-

decide which currency to calculate the return in. Two popular choices are:
-

-
o Translate all investment returns into the home country currency – Typically
-
used by managers who do not hedge their exchange rate risk.
o Denominate returns in local currencies – Typically used by managers who
hedge their exchange rate risk.
If the investment strategy’s goal is to achieve excess returns over a benchmark, a
suitable benchmark must also be specified.
-

③ Rebalancing frequency and transaction cost: Rebalancing frequency refers to how

often a portfolio is updated to reflect current data. It is common to use a monthly
rebalancing frequency.
When deciding on a rebalancing frequency, transaction costs should also be
considered. The more frequently rebalancing is performed, the more trading is
required, resulting in higher transaction costs.
 Start and end date: Typically, investment managers prefer to backtest investment
strategies over as long a period of time as possible, because a larger sample size
provides greater statistical confidence in the results.
However, long time periods can contain non-stationary data. There may be periods of
high and low inflation, recessions and expansion etc. Therefore, this analysis should
be supplemented with a historical scenario analysis (covered later) of the discrete
regimes within the long history.
Example: Strategy Design
(This is based on Example 1 from the curriculum.)
The following information is provided:
- Investment goal: Superior risk-adjusted absolute return.
Investment hypothesis: “Cheap” stocks will outperform “expensive” stocks. In other words,

© IFT. All rights reserved 3

 LM05 Backtesting and Simulation 2025 Level II Notes

exposure to the “value” factor will lead to outperformance

Key parameters defined for backtesting the hypothesis are:

-
 The trailing earnings yield is used to quantify the cheapness of a stock. It is the inverse of
the P/E calculated as:
Trailing 12 − month EPS
Trailing earnings yield =
Current share price
 Investment universe: Russell 3000 for the US market and S&P Europe BMI for the
-

European market.
-

 Return definition: Total returns will be hedged back into US dollars.

-

 Rebalancing frequency: monthly, including transaction costs, but returns on a 12-month

moving average basis will be computed
 Start and end date: Because data required for this strategy are widely available, we will
use a long time period: January 1986–May 2019

1. Given this information, which of the following should we be concerned about?

-

A. The strategy assumes that the US dollar will appreciate against the euro.
B. The historical period of the data includes recessions, currency regime changes, and
I

periods of varying interest rates.

C. There are serious issues with computing earnings yield for many stocks.
Solution:
B is correct. We are using a long data history that includes regime changes in inflation,
currencies, and interest rates, so the data is non-stationary. Therefore, backtesting
performance results should be supplemented with examinations of performance during the
discrete regimes.

2. Which of the following describes the relationship between rebalancing frequency and
transaction costs?
A. Changing the rebalancing frequency from monthly to weekly would likely increase
-
transaction costs.
-

B. Changing the rebalancing frequency from monthly to quarterly would likely increase
transaction costs.
C. Rebalancing frequency has no effect on transaction costs.
Solution:
-
A is correct. Typically, the more frequently rebalancing is done, the more trading is required,
which incurs more transaction costs.
-

3. Which of the following is not a potential concern of using a short time period for a
-

backtest?
-

A. The backtest will cover a limited number of business cycle, inflation, and interest rate
-

© IFT. All rights reserved 4

 LM05 Backtesting and Simulation 2025 Level II Notes

regimes.
B. The backtest may not be useful because the findings may apply only under the conditions
-
present in the time frame.
-

C. The backtest is likely to cover multiple business cycle, inflation, and interest rate
- -

regimes.
-

Solution:
C is correct. Covering multiple macroeconomic regimes is not a concern associated with
-

using a short time period for a backtest, because macroeconomic regimes tend to be multi-
- -

year in length.
-

3.2 Step 2: Historical Investment Simulation

[
Analysts typically use ‘rolling windows’ to simulate historical investment. In this approach a
portfolio is constructed at the start of a period using data from a historical in-sample period,
followed by testing on a subsequent, out-of-sample period. The process is repeated as time
moves forward.
- -

Assume we backtest a value strategy by measuring its performance month by month from
-

December 2011 to May 2012. On November 30, 2011, the process begins by calculating each
stock's trailing 12-month earnings yield by dividing EPS reported in the previous 12 months
-
(i.e., from December 2010 to November 2011, the in-sample months) by stock prices as of
-

November 30, 2011.

-

We then carry out the investment strategy as of that date - for example, buying stocks with
-

high earnings yields and shorting stocks with low earnings yields. Then, we record the
-
-

investment results for the month of December (i.e., the out-of-sample, OOS, month).
-

At the end of each subsequent month, the process is repeated by rebalancing the portfolio
with updated trailing 12-month earnings yield data and measuring the results over the
-

subsequent (OOS) month.

-

Exhibit 2 from the curriculum illustrates this process.

#

3.3 Step 3: Analysis of Backtesting Output

-

-
The final step in backtesting is generating results for presentation and interpretation. We are
-

© IFT. All rights reserved 5

 LM05 Backtesting and Simulation 2025 Level II Notes

concerned not only with the portfolio’s average return but also with its risk profile.
- -

Therefore, metrics such as the Sharpe ratio, the Sortino ratio, volatility and maximum
-

drawdown are used to evaluate performance. Apart from these measures, visuals are also
- -

frequently used – for example, the distributions of returns can be plotted against a well-
-

known distribution, such as the normal distribution.

-

- Assume that we implement our earnings yield-based value strategy as a long–short hedged
portfolio. In this approach, we sort the investable stock universe by the trailing earnings
yield, and divide the universe into quintiles. A long–short hedged portfolio is then
constructed by investing in the top quintile (the top 20% stocks with the highest earnings
yield) and shorting the bottom quintile (the bottom 20% of stocks with the lowest earnings
yield).
The results of this strategy for the US and European markets are shown in Exhibit 3 of the
=

curriculum.

I St -
S
-

This means ,
how much You drop from highest to lowest

Point before you start recovering again

.

© IFT. All rights reserved 6

LM05 Backtesting and Simulation 2025 Level II Notes

L
-

We also examine the strategy’s cumulative performance in four different stock markets: the
United States, Europe, Asia ex-Japan, and Japan. The results are presented in Exhibit 4.

> asia ex
-

Taken

----

Finally, we also plot the distribution of the strategy’s return in the US market against the

© IFT. All rights reserved 7

LM05 Backtesting and Simulation 2025 Level II Notes

normal distribution. This is shown in Exhibit 5.

>returns
- with backtesting .

>
-
roomed distribution
.

Example: Historical Investment Simulation and Output Analysis

(This is Example 2 from the curriculum.)
1. Describe how the backtest performance of value investing, based on the earnings yield
-

factor, in Europe compares with that in the United States over the 1986–2019 period, as
-

shown in Exhibit 3.
2. Describe the cumulative performance of value investing across the different markets
- -

shown in Exhibit 4 and the distributions of returns in the United States from this strategy
-

in Exhibit 5.
Solution to 1:
Over the backtesting period (January 1986–May 2019), the average annual return from the
-

value investing strategy in the United States is about 9.2%, with a Sharpe ratio of 0.75, as
- -

shown in Exhibit 3, Panel A. In Europe, the same investment strategy generated a

significantly lower (by 250 bps) average annual return, about 6.7%, but with significantly
-

lower volatility (Panel B). As a result, the Sharpe ratio for the European strategy, 0.67, is
-

comparable to the Sharpe ratio for the US strategy. In both markets, the maximum
-
-

drawdown is slightly more than three times the strategy's volatility. Therefore, as a long-
-

term value strategy, the earnings yield factor offers slightly better performance in the United
States than in Europe.
Solution to 2:
The value strategy has delivered strong performance over the long run across the several
markets, especially in Asia ex-Japan (Exhibit 4). Performance has flattened since 2016,
however, in the United States, Europe, and Japan after first leveling off in all geographies
except Asia ex-Japan after 2002. Significant drawdowns and potential structural breaks can

© IFT. All rights reserved 8

 LM05 Backtesting and Simulation 2025 Level II Notes

also be observed in late 1990s (i.e., during the tech bubble) and in March–May 2009 (i.e., the
risk rally during the global financial crisis) in most regions.
More problematically, the strategy in the United States seems to suffer from excess kurtosis
(i.e., fat tails) and negative skewness (Exhibit 5). The excess kurtosis implies that this
strategy is more likely to generate surprises—that is, extreme returns—whereas the
negative skewness suggests that those surprises are more likely to be negative (than
positive).
4. Backtesting Multifactor Models
-

Few investment managers base their investment strategy on a single signal, such as earning
-

yield. In practice, most quantitative stock selection models use a multifactor structure with a
-
-

linear combination of factors.

-

In this section, we will look at two multifactor equity portfolio strategies:

-

 A benchmark (BM) factor portfolio, which equally weights multiple fundamental

- -

factors

-

A risk parity (RP) factor portfolio, which weights factors based on equal risk
-
-
contribution
-

We will use the same three steps described previously to backtest these two portfolios.
4.1 Step 1: Strategy Design
We choose eight fundamental factors:
-

-1. Defensive value: Trailing earnings yield

-2. Cyclical value: Book-to-market ratio

Y3. Growth: Consensus FY1/FY0 EPS growth

4. Price momentum: 12-month total return, excluding the most recent month
L5. Analyst sentiment: 3-month EPS revision
~6. Profitability: Return on equity (ROE)
~7. Leverage: Debt-to-equity ratio
~8. Earnings quality: Non-cash earnings (proportion of accruals in earnings)
All eight factors have delivered positive returns over the long term in the United States.
-

For each factor, we create a portfolio by buying the top 20% of stocks and shorting the
-
bottom 20% of stocks ranked by the factor.

>For our benchmark portfolio (BM), we combine these eight factor portfolios by equally
weighting each one.

> For our risk parity (RP) portfolio, we combine the eight factor portfolios by equally
-

weighting them by their risk contribution. Risk parity is a popular alternative portfolio
construction technique that takes into account the volatility of each factor as well as the
return correlations among all factors in the portfolio. The goal is for each factor to contribute
-

© IFT. All rights reserved 9

 LM05 Backtesting and Simulation 2025 Level II Notes

an equal (hence "parity") risk contribution to the portfolio's overall risk.

-

We backtested our two portfolios in each of the following markets: the United States,
-

Canada, Latin America (LATAM), Europe, the United Kingdom, emerging Europe, Middle
-

East, and Africa (EMEA), Asia ex-Japan, Japan, Australia and New Zealand (ANZ), and
mainland China.
The portfolios were rebalanced monthly.
-

4.2 Step 2: Historical Investment Simulation

Backtesting a multifactor strategy is similar to the method covered earlier, but the rolling-
window procedure is implemented twice, once at each portfolio “layer”.
-

-7 First, we form eight factor portfolios at the start of each month using the rolling-window
procedure. These factor portfolios are then combined into two multifactor portfolios BM and
RP.
To avoid look-ahead bias, a second rolling-window procedure over the same time period is
>
-

required. At each month end, the previous five years of monthly data are used to estimate
the variance–covariance matrix for the eight factor portfolios. This information is then used
-
to calculate the weights for the RP portfolio.
-

Finally, we calculate the returns of the BM and RP portfolios during each of the “out-of-
sample” month.
4.3 Step 3: Output Analysis
Exhibit 7, Panel A shows the weights of the eight factor portfolios in the RP portfolio. The
weights are fairly stable, but they are far from equal. As a result, we can expect the risk and
return profile of the RP portfolio to differ from that of the BM portfolio.

> l
- volctalin -

3
> High
-
>wi,nuteener
-

volutur in

Although the RP portfolio appears to have a lower cumulative return that the BM portfolio
(Panel B), Panel C shows that the volatility of the RP portfolio is less than half the volatility of

© IFT. All rights reserved 10

LM05 Backtesting and Simulation 2025 Level II Notes

the BM portfolio. Therefore, the Sharpe ratio of the RP portfolio is nearly twice that of the
BM portfolio (Panel D).

- Rpend to hun
lower
volutely with
comparable retrs
.

result in

-
will
>
-
sharpe
higher
ratic mence this allows
9
major
to increase
use of levesag
,
returns
,

Exhibit 8 presents the monthly return distribution statistics for the eight factor portfolios as
well as the BM and RP portfolios. When compared to the eight underlying factor portfolios,

© IFT. All rights reserved 11

LM05 Backtesting and Simulation 2025 Level II Notes

BM and RP show moderate mean returns (0.5% and 0.4% per month, respectively) and low
standard deviations (1.6% and 0.7% per month, respectively). This highlights the
diversification benefits from factor allocation decisions.

Exhibit 9 presents the various downside risk measures for the eight factor portfolios as well
as the BM and RP portfolios. RP has much lower downside risk than any of the eight
underlying factors and BM. This suggests that the RP strategy benefits significantly from
diversification.

Example: Risk and Return Beyond Normal Distribution

(This is Example 3 from the curriculum.)
Compare return profiles for the BM and RP strategy multifactor portfolios and explain which
-

investment strategy offers the more attractive statistical properties for risk-averse investors
-
(refer to Exhibits 8 and 9).
Solution:
The BM and RP portfolios have nearly the same mean monthly returns, at 0.5% and 0.4%,
respectively (Exhibit 8). Although the maximum returns are similar, the RP portfolio has a
much smaller minimum return (–2.5%) and a significantly lower standard deviation (0.7%)
compared with those of the BM portfolio (–10.9% and 1.6%, respectively). The RP portfolio

© IFT. All rights reserved 12

 LM05 Backtesting and Simulation 2025 Level II Notes

is also slightly positively skewed (0.51%) and has moderate kurtosis (5.37), in contrast to
the negative skew (–2.40%) and high kurtosis (17.78) of the BM portfolio.
The RP portfolio offers similar returns, less downside risk (confirmed by its superior VaR,
CVaR, and maximum drawdown results in Exhibit 9), lower volatility, and slightly higher
probability of positive returns (i.e., positive skew) compared with the BM portfolio. It is also
less fat tailed (i.e., moderate kurtosis, meaning lower probability of extreme negative
surprises) than the BM portfolio. Therefore, the RP portfolio has the more attractive
distribution properties for risk-averse investors.
5. Common Problems in Backtesting
In this section, we will go over some of the most common mistakes investors make when
performing backtests.
5.1 Survivorship Bias
Companies appear and disappear from market indexes on a regular basis. For example, less
than 400 (roughly 13%) of the Russell 3000 Index constituents from1985 are still included
in the index. Stocks that have remained in the index over time are referred to as “survivors.”
Survivorship bias refers to drawing conclusions from data that only includes entities that
have survived to that point. It is one of the common mistakes that investors make when
conducting back tests using current index constituents.
Some investors argue that because you can only invest in companies that exist today,
backtesting strategies using only the current index constituents is acceptable. However, the
problem with this argument is that in the past, one could not predict which companies
would survive, which would vanish, and which would be founded and become successful
enough to be included in the index.
-To avoid this bias, analysts should use point-in-time data to conduct backtests. This is the
most complete data for any given time period and includes all companies that existed i.e. the
casualties as well as the survivors.
Backtesting results using current index constituents and point-in-time data can differ
significantly.
5.2 Look-Ahead Bias
Look-ahead bias is created by using information that was unknown or unavailable during the
historical periods over which the backtest is conducted. Survivorship bias is also a type of
look-ahead bias.
Look-ahead bias has several common forms:
 Reporting lags: For example, EPS results for a quarter are typically not available
-
until a few months after the quarter ends. Using these values as of the end of the
- -

quarter introduces look-ahead bias into the backtest.

-

© IFT. All rights reserved 13

&
LM05 Backtesting and Simulation 2025 Level II Notes

 Revisions: Macroeconomic data is frequently revised, and companies frequently re-

-

state their financial statements. Many databases only keep the most recent figures,
-

replacing previous figures with revised ones. Using such data introduces look-ahead
-

bias.
-

 Index additions: Data vendors often add new companies to their database. When
-

doing so, they enter several years of historical financial statements into their systems.
As a result, an analyst backtesting
-
with the current database would be using
information on companies that were not actually in the database during the
-

backtesting period.
-

To avoid look-ahead bias, analysts should use point-in-time data to conduct backtests.
5.3 Data Snooping
Data snooping refers to making an inference after looking at statistical results rather than
testing a prior inference. The bias occurs when an analyst selects data or performs analyses
repeatedly until a significant result is found.
The bias can be mitigated by:
 Setting a much higher hurdle than typical – for example, a t-stat greater than 3.0 (as
-

compared to the usual 2.0)

 Using cross validation – the dataset is partitioned into training data and validation
-
-

data. The model built from training data is tested on the validation data.
-
-

Rolling window backtesting is a form of cross-validation.

-

Another common cross-validation technique is to use data from different geographic

regions. For example, a strategy developed using US equities can be tested in other
markets globally.
6. Historical Scenario Analysis cliner rollingwindo
st)
Rather
-

-
determinis
When using a long historical period for backtesting, we may encounter regime changes.
than simply acknowledging or ignoring these changes, an analyst should pay close
-

attention to different structural regimes and how they affect a strategy during regime
-

-
changes.
Historical scenario analysis is a type of backtesting used to assess the performance and risk
of an investment strategy in different structural regimes and at structural breaks.
Two common examples of regime changes are:
 Expansions and recessions
Y  High-and low-volatility regimes
We can assess the BM and RP portfolios with respect to these two regimes – recessions
-

versus expansion, and high volatility versus low volatility.

-

© IFT. All rights reserved 14

LM05 Backtesting and Simulation 2025 Level II Notes

Using the Sharpe Ratio

As shown in Panel A of Exhibit 15, in terms of the Sharpe ratio, the RP strategy performs well
-

during recessions, whereas the BM strategy struggles during recessions.

As shown in Panel B of Exhibit 15, the RP strategy performs equally well in both volatility
regimes, whereas the BM strategy performs slightly worse in low-volatility regimes.

Using the probability density plot

In addition to the Sharpe ratio, we can also use a probability density plot to evaluate the two
-
investment strategies. For example, Exhibit 16 plots the distribution of returns for the BM
-
-
and RP portfolios during recession versus non-recession periods.
-

The key points to note from this exhibit are:

 In non-recession environments, the distribution of returns for both the BM and RP
strategies is flatter, implying higher standard deviations.
 Regardless of the regime, the BM strategy suffers from negative skewness and excess
kurtosis (i.e., fat tails to the left), but its average return is clearly lower in a recession

© IFT. All rights reserved 15

LM05 Backtesting and Simulation 2025 Level II Notes

environment (Panel A).

 The RP strategy also has a lower average return in the recession regime (Panel B), but
its volatility and kurtosis are both much lower when compared to the BM strategy.

7. Simulation Analysis
Backtesting implicitly assumes that the past is likely to repeat itself. This assumption,
however, may not hold true, and the markets may exhibit extreme upside and downside
risks that have never been seen before. Hence, ‘simulation analysis' is used to supplement
-

backtesting and obtain a more complete picture.

-

The two basic types of simulation are:

-

 Historical simulation: Instead of assuming that a strategy was implemented at some

© IFT. All rights reserved 16

 LM05 Backtesting and Simulation 2025 Level II Notes

past date and collecting results as the strategy runs over time, we construct results by
randomly selecting returns from many different historical periods without regard to
time-ordering.
Thus, as compared to backtesting we relax a key constraint by randomly changing the
sequence of historical periods from which factor returns are drawn.
Historical simulation is widely used in investment management, mainly by banks for
-
market risk analysis.
-

Monte Carlo simulation: The issue with historical simulation is that there is only
one set of realized data to draw from (in other words the past happened in only one
-

way). Monte Carlo simulation helps overcome this issue. Here each key variable is
assigned a statistical distribution, and observations are drawn at random from that
distribution.
CThus, the main advantage of Monte Carlos simulation is that the analyst does not have
to rely on historical distributions, he can specify any distribution and incorporate
non-normality, fat tails, tail dependence, and so on, to model key variables. But the
downside is that the method is complex and computationally intensive.
A simulation is implemented using the following eight steps:
-
7
-[Link] the target variable. Typically, this variable is the return on an investment
-

strategy and its distribution.

-

2. Specify key decision variables. Typically, these variables are the returns of each
- - -

underlying asset and their weights in the overall portfolio.

-
-

3. Specify the number of trials (N) to run. Typically, researchers choose between 1,000
- and 10,000 simulation runs. The greater the number of simulations the more reliable
- -

the results.
-

4. Define the distributional properties of the key decision variables.

- -

-  In historical simulation, we draw from historical data.

-

-
 In Monte Carlo simulation, we can select and specify an appropriate statistical
-

distribution for each key decision variable.

-

~5. Use a random number generator to draw N random numbers for each key decision
-
variable.
-

-6. For each set of simulated key decision variables, compute the value of the target
variable.
-

-7. Repeat steps 5 and 6 ‘N’ times.

-
8. We now get a set of N values of the target variable. We can then compute typical
- -
-
metrics like mean return, volatility, Sharpe ratio, and the various downside risk
metrics such as CVaR and maximum drawdown.
-

7.1 Historical Simulation

In historical simulation, a decision must be made whether to sample from historical returns
with or without replacement. Because the number of simulations required is usually greater

© IFT. All rights reserved 17

 Simulation in simplified words !

Monte-Carlo
* -
LM05 Backtesting and Simulation 2025 Level II Notes

than the size of the historical dataset, random sampling with replacement, also known as

-
bootstrapping, is frequently used in investment research.
Tristorical)
-

Steps in the simulation

We now perform a historical simulation on the BM and RP portfolios. The steps are:
-1. The target variables are the returns for the BM and RP multifactor portfolios.
2. The key decision variables are the returns of the eight underlying factor-based
-
portfolios (the weights allocated to the eight factors are already known).
-3. The simulation will be performed for N = 1,000 trials
w4. The historical simulation will be implemented using bootstrapped sampling.
5. The random number generator will then randomly draw 1,000 numbers from the
~
uniform distribution between 0 and 1. Exhibit 17 shows the factor returns for the first
five randomly generated numbers.

6. For each simulation, we calculate the returns of the BM and RP portfolios.

-

For example, for Sep 2006, the return of the BM portfolio is the equally weighted
average of the eight factor returns: 0.125 × 2.5% + 0.125 × 0.3% + 0.125 × –0.8% +
0.125 × 0.0% + 0.125 × –0.8% + 0.125 × 2.5% + 0.125 × 0.5% + 0.125 × –0.5% =
0.46%.
To compute the return on the risk parity portfolio, we use the weights allocated to
-
each of the eight factors for the final month (May 2019). The calculations are shown
in Exhibit 18.

© IFT. All rights reserved 18

LM05 Backtesting and Simulation 2025 Level II Notes

7. Steps 5 and 6 are repeated for all 1,000 trials generating a collection of 1,000
simulated returns for BM and RP portfolios.
8. We then calculate performance metrics of interest (Sharpe ratio, CVaR, etc.) and plot
the distributions of the simulated benchmark and risk parity portfolio returns.
Interpreting the results
The results of the historical simulation are similar to the results of the rolling-window
backtesting method covered previously.
As shown in Panel A of Exhibit 19, the RP portfolio outperforms the BM portfolio in terms of
the Sharpe ratio.

Also, as shown in Panel B of Exhibit 19, the RP portfolio has significantly less downside risk,
as measured by CVaR, than the BM portfolio.

© IFT. All rights reserved 19

LM05 Backtesting and Simulation 2025 Level II Notes

We can also plot the estimated probability distribution of returns for our two investment
strategies as shown in Exhibit 20. Backtesting and historical simulation both indicate that
the RP portfolio returns are less volatile and more skewed to the right, with lower downside
risk (i.e., lower standard deviation and thinner tails) than the BM portfolio returns.

R P
>
-

Benchmark .

>

© IFT. All rights reserved 20

LM05 Backtesting and Simulation 2025 Level II Notes

7.2 Monte Carlo Simulation

Monte Carlo simulation is a more sophisticated technique than historical simulation. The
most important decision in Monte Carlo simulation is the functional form of the statistical
distribution of decision variables/return drivers. The simulation is only useful if the
functional form of the statistical distribution we specify accurately reflects the true
distribution of the underlying data.
Because of its simplicity, the multivariate normal distribution is often used in investment
research. A multivariate normal distribution can be fully specified with only a few key
parameters—the mean, the standard deviation, and the covariance matrix. For K assets, we
-

need to estimate K mean returns, K standard deviations, and [K × (K—1)]/2 correlations.

However, we have to be aware of the fact that a multivariate normal distribution cannot
-
account for negative skewness and fat tails observed in factor and asset returns.
-

Steps in the simulation

We will now perform a Monte Carlo simulation on the BM and RP portfolios. The steps are:
-

~1. The target

-
variables are the returns for the BM and RP multifactor portfolios.
-

2. The key decision variables are the returns of the eight underlying factor-based
~
- -

portfolios (the weights allocated to the eight factors are already known).
-3. The simulation will be performed for N = 1,000 trials
- -
4. We choose the multivariate
-
normal distribution as our initial functional form. We
calibrate the mode by calculating the 8 factor portfolio mean returns, the 8 standard
-

deviations, and the 28 elements of the covariance matrix.

5. The calibrated multivariate normal distribution is then used to simulate the future
-

factor returns. Exhibit 21 shows

-
the factor returns for the first five sets of Monte
Carlo simulations.
-

6. For each simulation, we calculate the returns of the BM and RP portfolios. For
- -

example, for the first simulation, the returns of the BM portfolio can be calculated as:
-

0.125 × –3.2% + 0.125 × –3.1% + 0.125 × –0.2% + 0.125 × 0.7% + 0.125 × 2.3% +
0.125 × –3.3% + 0.125 × –1.7% + 0.125 × 1.9% = -0.83%
-

Similarly, to compute the return on the risk parity portfolio, we use the weights
-

allocated to each of the eight factors for the

% final month (May 2019). The calculations

are: -

© IFT. All rights reserved 21

LM05 Backtesting and Simulation 2025 Level II Notes

= 0.06 × –3.2% + 0.303 × –3.1% + 0.117 × –0.2% + 0.052 × 0.7% + 0.104 × 2.3% +
0.063 × –3.3% + 0.096 × –1.7% + 0.204 × 1.9%) = -0.86%
-

7. Steps 5 and 6 are repeated for all 1,000 trials generating a collection of 1,000
-

simulated returns for BM and RP portfolios.

8. Finally, we assess the performance and risk profiles of our two investment strategies.
-

Interpreting the results

Example: How to Interpret Results from Historical and Monte Carlo Simulations
(This is based on Example 7 from the curriculum.)
Exhibit 22 shows the Sharpe ratios (Panel A) and downside risk measures, CVaRs (Panel B),
for the returns of the benchmark and risk parity portfolios based on rolling-window
backtesting, historical simulation, and Monte Carlo simulation of the returns on the eight
underlying factor portfolios.
Discuss similarities and differences among the three approaches for simulated performance
of the benchmark and risk parity portfolios.

© IFT. All rights reserved 22

LM05 Backtesting and Simulation 2025 Level II Notes

Solution:
We can conclude that the backtesting approach provides realistic performance metrics. The
two simulations are used to independently verify the results from the rolling-window
backtesting.
As shown in Panel A of Exhibit 22, the Sharpe ratio appears relatively insensitive to the
simulation and backtesting methods used, with the RP strategy outperforming the BM
strategy by nearly the same margin for each method.
On the other hand, CVaR seems to be sensitive to how randomness is treated. In particular,
the Monte Carlo simulation appears to understate the downside risk of the BM strategy
compared with both rolling-window backtesting and historical simulation methods (Panel
B). Because the factor returns are negatively skewed with fat tails (i.e., excess kurtosis), the
multivariate normal distribution assumption is likely to be underestimating the true
downside risk of the BM strategy.

© IFT. All rights reserved 23

LM05 Backtesting and Simulation 2025 Level II Notes

8. Sensitivity Analysis
Sensitivity analysis is a technique for assessing how changes in input variables affect a target
variable and risk profiles. It can be implemented to help managers better understand the
potential risks and returns of their investment strategies.
The multivariate normal distribution is commonly used as a starting point for Monte Carlo
-

simulations. This distribution assumption, however, fails to account for negative skewness
and fat tails commonly observed in factor and asset return data.
Therefore, we should perform a sensitivity analysis by fitting our factor return data to a
different distribution and then repeating the Monte Carlo simulation. One alternative to test
C
is a multivariate skewed Student’s t-distribution. This distribution can account for the
skewness and excess kurtosis but it requires estimation of more parameters and is thus

-
more likely to suffer from larger estimation errors.
Continuing with our BM and RP examples, the procedure for the new Monte Carlo simulation
is nearly identical to one previously performed. Steps 4 and 5 are the only exception. In Step
4, instead of fitting the data to a multivariate normal distribution, we calibrate our model to
a multivariate skewed t-distribution. In Step 5, we simulate 1,000 sets of factor returns from
this new distribution function.
We then assess the performance and risk profiles of our investment strategies from the
1,000 simulated returns.
As shown in Panel A of Exhibit 24, the Sharpe ratio appears to be unaffected by any
particular simulation method. It consistently shows that the RP strategy outperforms the BM
strategy.
The CVaR for the RP strategy also is not particularly sensitive to any specific simulation
method. However, the CVaR for the BM strategy is highly sensitive to the choice of the
simulation approach.

© IFT. All rights reserved 24

LM05 Backtesting and Simulation 2025 Level II Notes

~
---

--

Estimated probability density plots, in Panel A of Exhibit 25, show that for the BM strategy,
the difference between the historical simulation and the two Monte Carlo methods is quite
large. The BM strategy displays negative skewness and excess kurtosis. This is apparent
from the shape of the historical simulation return distribution. It is not surprising that that
the two Monte-Carlo simulations fail to adequately account for this left-tail risk.
In contrast, because the RP strategy's return distribution is relatively symmetric and lacks
significant excess kurtosis, all three simulation methods produce a fairly similar picture
(Panel B).

© IFT. All rights reserved 25

LM05 Backtesting and Simulation 2025 Level II Notes

© IFT. All rights reserved 26

LM05 Backtesting and Simulation 2025 Level II Notes

Summary
LO: Describe objectives in backtesting an investment strategy.
Backtesting approximates the real-world investment process by evaluating whether a
strategy would have produced desirable results using historical data. It can be used to decide
if an investment strategy should be accepted or rejected.
LO: Describe and contrast steps and procedures in backtesting an investment strategy.
Backtesting consists of three steps: strategy design, historical investment simulation, and
analysis of backtesting output.

LO: Interpret metrics and visuals reported in a backtest of an investment strategy.

The final step in backtesting is generating results for presentation and interpretation. We are
concerned not only with the portfolio’s average return but also with its risk profile.
Therefore, metrics such as the Sharpe ratio, the Sortino ratio, volatility and maximum
drawdown are used to evaluate performance. Apart from these measures, visuals are also
frequently used – for example, the distributions of returns can be plotted against a well-
known distribution, such as the normal distribution.
LO: Identify problems in a backtest of an investment strategy.
The common problems in a backtest of an investment strategy include:
 Survivorship bias: Can be avoided by using point in time data.
 Look ahead bias: Can be avoided by using point in time data.
 Data snooping: Can be avoided by using a high hurdle rate or by using cross
validation.
LO: Evaluate and interpret a historical scenario analysis.
Historical scenario analysis is a type of backtesting used to assess the performance and risk

© IFT. All rights reserved 27

LM05 Backtesting and Simulation 2025 Level II Notes

of an investment strategy in different structural regimes and at structural breaks.

Two common examples of regime changes are:
 Expansions and recessions
 High-and low-volatility regimes
LO: Contrast Monte Carlo and historical simulation approaches.
Historical simulation is relatively simple to perform, but it has the same advantages and
disadvantages as rolling-window backtesting. For example, a key assumption shared by
these methods is that the distribution pattern from historical data is sufficient to represent
future uncertainty. In historical simulation, bootstrapping ie. random draws with
replacement is frequently used.
Monte Carlo simulation is a more sophisticated technique than historical simulation. The
most important decision in Monte Carlo simulation is the functional form of the statistical
distribution of decision variables/return drivers. Because of its simplicity, the multivariate
normal distribution is often used in investment research. However, we have to be aware of
the fact that a multivariate normal distribution cannot account for negative skewness and fat
tails observed in factor and asset returns.
LO: Explain inputs and decisions in simulation and interpret a simulation.
A simulation is implemented using the following eight steps:
1. Define the target variable. Typically, this variable is the return on an investment
strategy and its distribution.
2. Specify key decision variables. Typically, these variables are the returns of each
underlying asset and their weights in the overall portfolio.
3. Specify the number of trials (N) to run. Typically, researchers choose between 1,000
and 10,000 simulation runs. The greater the number of simulations the more reliable
the results.
4. Define the distributional properties of the key decision variables.
 In historical simulation, we draw from historical data.
 In Monte Carlo simulation, we can select and specify an appropriate statistical
distribution for each key decision variable.
5. Use a random number generator to draw N random numbers for each key decision
variable.
6. For each set of simulated key decision variables, compute the value of the target
variable.
7. Repeat steps 5 and 6 ‘N’ times.
8. We now get a set of N values of the target variable. We can then compute typical
metrics like mean return, volatility, Sharpe ratio, and the various downside risk
metrics such as CVaR and maximum drawdown.

© IFT. All rights reserved 28

LM05 Backtesting and Simulation 2025 Level II Notes

LO: Demonstrate the use of sensitivity analysis.

Sensitivity analysis is a technique for assessing how changes in input variables affect a target
variable and risk profiles. It can be implemented to help managers better understand the
potential risks and returns of their investment strategies.
Sensitivity analysis frequently uses the multivariate skewed Student’s t-distribution which
can account for skewness and excess kurtosis. But it requires estimation of more parameters
and is thus more likely to suffer from larger estimation errors.

© IFT. All rights reserved 29