Performance Audit Manual

Directorate of Audit Quality Control

September 2017
 PERFORMANCE AUDIT MANUAL

OVERVIEW

Page number CHAPTERS IN THE PERFORMANCE AUDIT MANUAL

Page 2 General introduction

Chapter 1 - Framework for Performance Audits by the European Court

Page 6
of Auditors

Page 13 Chapter 2 - The Performance Audit Approach and the 3 ‘E’s

Page 29 Chapter 3 - Planning the audit

Page 55 Chapter 4 - Examination phase

Page 70 Chapter 5 – Reporting phase

Page 89 Detailed table of contents

Guidelines, best practices, examples, support and guidance for implementing this Performance
Audit Manual are provided by the Directorate of Audit Quality Control Committee (DQC) of the
Who to contact European Court of Auditors (ECA).

Please consult the DQC intranet site or send us an e-mail:

ECA-DQC-CONTACT or [email protected]
We welcome reproduction and distribution of this manual; specific permission does not need to be obtained from the ECA.

General Introduction - page 1

 PERFORMANCE AUDIT MANUAL

GENERAL INTRODUCTION

Chapters in the
TABLE OF CONTENTS
Performance Audit Manual

General introduction Purpose & Content of the Performance Audit Manual

Chapter 1: Framework for Structure

Performance Audits by the
European Court of Auditors
Glossary of concepts and technical terms

Chapter 2: The Performance

Audit Approach and the
3 ‘E’s

Chapter 3: Planning the

audit

Chapter 4: Examination
Phase

Chapter 5: Reporting Phase

For any further information, please contact:

European Court of Auditors
Who to contact
Directorate of Audit Quality Control Committee (DQC)
E-mail: ECA-DQC-CONTACT or [email protected]

General Introduction - page 2

 PURPOSE & CONTENT OF THE PERFORMANCE AUDIT MANUAL

The Performance Audit Manual is one part of the suite of procedures and
guidance provided by the ECA. Its purpose is to:
Purpose - Quality • help to achieve high quality in performance audits, and
• promote professional competence amongst auditors in this domain.
Content The manual has been written to meet the needs of auditors and audit
management. It explains in broad terms how performance audits should be
planned, conducted and reported.
Audit Policy In carrying out its duties and responsibilities within its mandate as laid down in
the Treaty and the Financial Regulation, the European Court of Auditors
conducts its audits in accordance with the INTOSAI International Auditing
Standards and Code of Ethics, in so far as these are applicable in the
European Union context. Auditors are required to respect the ECA
Performance Audit Manual as well as all the audit procedures adopted by the
ECA.
Professional judgement essential, The manual encourages the exercise of professional judgement at all stages
throughout the audit, which is essential given the variety of potential audit
topics, objectives and data collection and analysis methods available in
performance audit.
and 'should' statements mandatory. Performance auditing procedures are written as ‘should’ statements, which
must be complied with

STRUCTURE

The manual comprises five chapters. The first two chapters provide the
necessary background material, whilst chapters 3 to 5 provide more detailed
guidance on each phase of a performance audit - planning, examination and
reporting. The manual is structured as follows:

CHAPTER 1 sets out the context of performance audit in the EU institutions,

and ECA's mandate and objectives for such audits.

CHAPTER 2 describes the performance audit approach, the application of the

concepts of economy, efficiency and effectiveness in the EU domain, and the
essential qualities of good performance audits.

CHAPTER 3 sets out the audit planning process, including the Audit Planning
Memorandum.

CHAPTER 4 describes the audit examination phase, including the conduct of

the audit, the communication of audit findings and audit management and
quality control arrangements.

CHAPTER 5 deals with the reporting process, including such activities as

planning, drafting, reviewing, clearing, distributing, and following up on the
report.

General Introduction - page 3

 GLOSSARY OF CONCEPTS AND TECHNICAL TERMS

Some of the following definitions are based on those which are to be found in
volume 6 of the MEANS Collection of the European Commission, which sets out
the methodological framework for evaluations in the area of structural policies.
These concepts are also applicable to other areas, in which different
terminologies may be employed.

DIRECT ADDRESSEE Person or organisation directly affected by the intervention. The term ’beneficiary’
is also often used. Direct addressees receive support, services and information,
and use the facilities created with the support of the intervention (e.g. farmers
using an irrigation network created by a development project).

EXOGENOUS FACTOR Factor independent of a public intervention which is partly or entirely the cause of
changes (results and impacts) observed among addressees (e.g. climatic
conditions, evolution in economic situation, performance of contractors,
beneficiaries' behaviour).

IMPACT Longer-term socio-economic consequences that can be observed after a certain

period after the completion of an intervention, which may affect either direct
addressees of the intervention or indirect addressees falling outside the boundary
of the intervention, who may be winners or losers.

INDIRECT ADDRESSEE Person or organisation which has no direct contact with an intervention, but which
is affected by it via direct addressees, either positively (e.g. a person obtaining a
job because someone else was granted early retirement under an intervention) or
negatively (e.g. firms losing business to other firms which have used technology
transfer networks set up by an intervention to innovate).

INPUT Financial, human, and material resources that are mobilised for the
implementation of an intervention.

INTERVENTION Any action or operation, carried out by public authorities or other organisations,
regardless of its nature (policy, programme, measure or project). Means of
intervention employed are grants, loans, subsidised interest rates, guarantees,
participation in equity and risk capital schemes or other forms of financing.

MEASURE Within the framework of a policy, the basic unit of programme management,
consisting of a set of similar projects and having a precisely defined budget. Each
measure generally has a particular management apparatus.

NEED Problem or difficulty affecting concerned groups, which the public intervention
aims to solve or overcome.

General Introduction - page 4

 OBJECTIVE Initial statement of the outcomes intended to be achieved by an intervention. A
distinction should be made between global, intermediate, immediate and
operational objectives:
• a global objective corresponds to the global impact of an intervention and is
generally defined by EU legislation in very broad terms (e.g. catching up on a
level of development); it is usually translated by the Commission and Member
States into intermediate objectives which correspond to the expected
intermediate impacts of programmes financed (e.g. making businesses more
competitive);
• immediate objectives concern the results of an intervention on direct
addressees and are normally defined by Members States within the
implementation of programmes financed (e.g. increasing by 20% the turnover
of businesses receiving technological support);
• operational objectives specify the outputs to be produced (e.g. providing 500
hours of consultancy services to small and medium enterprises).

OUTCOME Change that arises from the implementation of an intervention and which normally
relates to the objectives of this intervention. Outcomes include results and
impacts. Outcomes may be expected or unexpected, positive or negative (e.g. a
new motorway attracting investors to a region but causing unacceptable levels of
pollution in the areas through which it passes).

OUTPUT That which is produced or accomplished with the resources allocated to an

intervention (e.g. grants distributed to farmers, training courses delivered to
unemployed people, a road built in a developing country).

POLICY A set of different actions and operations (programmes, procedures, legislation,

and rules) directed towards a single goal or general objective (e.g. European
economic and social cohesion policy). These activities are often accumulated
incrementally through the years.

PROCESSES Procedures and activities employed to convert inputs into outputs (e.g.
procedures for delivering subsidies or selecting projects for financing). The
concept also covers the generation of management information and its use by
managers.

PROGRAMME An organised set of financial, organisational and human resources mobilised to

achieve an objective or set of objectives in a given timeframe. A programme is
delimited in terms of a schedule and a budget, and its objectives are defined
beforehand. It is always under the responsibility of an authority or several
authorities who share the decision-making. Programmes are generally broken
down into measures and projects.

PROJECT Non-divisible operation, delimited in terms of schedule and budget, and placed
under the responsibility of an organisation which implements, closest to the field,
the resources allocated to the intervention.

RESULT Immediate changes that arise for direct addressees at the end of their
participation in an intervention (e.g. improved accessibility to an area due to the
construction of a road, trainees who have found a job).

General Introduction - page 5

 PERFORMANCE AUDIT MANUAL

CHAPTER 1
FRAMEWORK FOR PERFORMANCE AUDITS BY THE
EUROPEAN COURT OF AUDITORS

Chapters in the
TABLE OF CONTENTS
Performance Audit Manual

General introduction
1.1 Introduction

1.2 Performance audit and sound financial management in the

Chapter 1: Framework for European Union
Performance Audits by the
European Court of Auditors 1.2.1 Performance Audit

1.2.2 Sound financial management: Treaty & Financial Regulation

Chapter 2: The Performance
Audit Approach and the 3 1.2.3 Management methods to implement the budget
‘E’s
1.2.4 Internal control system to achieve sound financial management

1.2.5 Relationship of performance audit & financial and compliance

Chapter 3: Planning the audit
audit
1.2.6 Relationship of performance audit & evaluation

Chapter 4: Examination 1.3 The ECA's mandate & objectives for performance audits
Phase
1.3.1 The ECA's legal obligations

1.3.2 The ECA's objectives

Chapter 5: Reporting Phase

For any further information, please contact:

Who to contact European Court of Auditors
Directorate of Audit Quality Control Committee (DQC)
E-mail: ECA-DQC-CONTACT or [email protected]

Chapter 1: Framework for performance audits by the European Court of Auditors - page 6
 1.1 INTRODUCTION

This chapter provides a framework of reference for the ECA’s performance

audits by setting out the context of such audits in the European Union, and
the ECA’s mandate and objectives in this regard.
The text mostly refers to the General Budget of the European Union ("the
Budget") and to the Commission, as these constitute the main audit area of
the ECA. However, this framework applies to all performance audits carried
out by the ECA, including those in the areas of the European Development
Funds, the Agencies and the European Central Bank.

1.2 PERFORMANCE AUDIT AND SOUND FINANCIAL MANAGEMENT IN THE EUROPEAN UNION

1.2.1 Performance Audit

A performance audit is an independent, objective and reliable examination of

whether undertakings, systems, operations, programmes, activities or
Definition
organisations are operating in accordance with the principles of economy,
1
efficiency and effectiveness, and whether there is room for improvement.

1.2.2 Sound financial management: Treaty & Financial Regulation

Article 317 of the consolidated text of the Treaty on the Functioning of the EU
(TFEU) states that "The Commission shall implement the budget...on its own
responsibility...having regard to the principles of sound financial management"
Treaty
and that "Member States shall cooperate with the Commission to ensure that
the appropriations are used in accordance with the principles of sound financial
management".
According to Article 30 of the Financial Regulation applicable to the general
2
budget of the Union (the "Financial Regulation"), the concept of sound
Financial Regulation
financial management comprises the principles of economy, efficiency and
effectiveness, which are defined as follows:
 the principle of ECONOMY requires that the resources used by the audited
entity in the pursuit of its activities shall be made available in due time, in
appropriate quantity and quality and at the best price;
 the principle of EFFICIENCY concerns the best relationship between
3
resources employed and results achieved;
 the principle of EFFECTIVENESS concerns the attainment of the specific
objectives set and the achievement of the intended results.

1
ISSAI 300.
2
OJ L 298, 26 October 2012 (last modified OJ L 286, 30 October 2015).
3
The term ‘results’ used in the context of efficiency and effectiveness is to be interpreted in a wide sense as covering outputs, results and
impacts (see Glossary on pages 4-5).

Chapter 1: Framework for performance audits by the European Court of Auditors - page 7
1.2.3 Management methods to implement the budget

Whilst the Commission is responsible for the overall implementation of the

budget, the Financial Regulation (Art. 58, 2012) provides for three different
management methods:
Direct management • directly by its departments; this method is used mainly in the area of
external actions and administrative expenditure and involves direct
management which is the responsibility of the Commission's Directorates-
4
General ;
Shared management (Art. 59 FR) • by shared management; shared management involves the delegation of
implementation tasks to Member States and mainly concerns expenditure
on agricultural and structural operations;
Indirect management (Art. 60 FR) • by indirect management with third countries and designated bodies,
international organisations, European Investment Bank, European
Investment Fund, and other bodies pursuant to Art. 58.1.c (FR, 2012); this
method involves the delegation of implementation tasks to beneficiary
countries and international organisations, generally in the area of external
actions, or to EU agencies and public- or private-sector bodies, notably in
the area of internal policies.
Each method involves a different allocation of roles and responsibilities for the
Method has significant implications
for the audit
implementation of the budget, which should be carefully taken into account
when planning, undertaking and reporting a performance audit.

4
The Commission may devolve responsibilities for preparing and implementing activities to Delegations in third countries.

Chapter 1: Framework for performance audits by the European Court of Auditors - page 8
1.2.4 Internal control system to achieve sound financial management

In order to have reasonable assurance that the objective of sound financial

management is achieved, the Commission, and other audited entities, must
5
establish an appropriate internal control system .
Information Technology (IT) systems are part of the internal control system at
6
the Commission, which follows the model proposed by COBIT regarding IT
governance in managing information and IT resources.
COSO In 2017 the Commission moved to a principle-based system with the aim of
ensuring robust internal control through consistent assessment by the
Commission, while providing the necessary flexibility to allow departments to
adapt to their specific characteristics and circumstances. The new Internal
Control Framework consists of five internal control components and
17 principles based on the COSO 2013 Internal Control-Integrated
Framework, as follows:
7
Design of the Commission's Internal Control Framework
Internal Control
Internal Control Principles
Components

Control environment 1. Demonstrates commitment to integrity and ethical values

2. Exercises oversight responsibility
3. Establishes structure, authority and responsibility
4. Demonstrates commitment to competence
5. Enforces accountability
Risk assessment 6. Specifies suitable objectives
Internal control components and 7. Identifies and analyses risk
principles in the Commission
8. Assesses fraud risk
9. Identifies and analyses significant change
Control Activities 10. Selects and develops control activities
11. Selects and develops general control over technology
12. Deploys through policies and procedures
Information and 13. Uses relevant information
communication
14. Communicates internally
15. Communicates externally
Monitoring 16. Conducts ongoing and/or separate assessments
17. Assesses and communicates deficiencies

5
The Committee of Sponsoring Organisations of the Treadway Commission (COSO) defines internal control as "a process effected by an
entity's board of directors, management and other personnel, which is designed to provide reasonable assurance regarding the
achievement of objectives in the following categories:
• effectiveness and efficiency of operations
• reliability of financial reporting
• compliance with applicable laws and regulations".
6
COBIT (Control Objectives for Information and Related Technology).
7
Communication to the Commission from Commissioner Oettinger C(2017) 2373 final from 19.04.2017 on Revision of the Internal Control
Framework

Chapter 1: Framework for performance audits by the European Court of Auditors - page 9
1.2.5 Relationship of performance audit & financial and compliance audit

Performance audit differs in many ways from financial audit; the main
differences are summarised in the table below:
ASPECTS Performance audit Financial and compliance audit

Assess whether EU funds have Assess whether financial operations

Purpose been used with economy, have been legally and regularly
efficiency, effectiveness. executed and accounts are reliable.

Policy, programme,
Financial transactions, accounting and
Focus organisation, activities and
key control procedures.
management systems.

Economics, political science,

Academic basis Accountancy and law.
sociology etc.

Methods Vary from audit to audit. Standardised format.

More open to the auditors'

Less open to the auditors' judgement.
judgement.
Audit criteria Standardised criteria set by legislation
Unique criteria for the individual
and regulation for all audits.
audit.

Special report published on an

ad hoc basis.
Varying structure and content,
depending on objectives. Annual report.
Reports
More or less standardised.
A macro view on performance of
the EU budget is included in a
dedicated chapter of the AR.

8
Financial and compliance audit aspects , including environmental
considerations in the context of sustainable development, can also be included
9
in a performance audit . An audit combining these aspects is called a
“comprehensive audit”. Whether to carry out a performance audit or a
Comprehensive audit
comprehensive audit is a matter of professional judgement and is a decision to
be taken on a case-by-case basis. Auditors need to be aware that carrying out
a "pure" performance audit is already a challenging task and that carrying out a
comprehensive audit would be even more demanding.
A comprehensive audit should therefore always be considered with great care
and undertaken only in cases where it is clear that it will be possible to obtain
sufficient, relevant and reliable audit evidence and deliver clear, useful and
timely messages at the reporting stage to satisfy performance, compliance
and/or financial audit objectives. The various elements should be clearly
must be carefully considered. distinguished in the Audit Planning Memorandum and the Audit Programme, so
that the audit team is clear about and gives due consideration to the differing
audit objectives within the audit task.
Where there is an overlap between other types of audit and performance
auditing, classification of the audit engagement should be determined by the
10
primary purpose of that audit .

8
Auditing Standards ISSAI 1000-series and ISSAI 4000.
9
Audit Standard ISSAI 3000/16.
10
Idem.

Chapter 1: Framework for performance audits by the European Court of Auditors - page 10
1.2.6 Relationship between performance audit & evaluation

Evaluation is an important element of the Commission's internal control

11
system. According to the Commission , evaluation is the "judgement of
interventions according to their results, impacts and the needs they aim to
satisfy".

The main purposes of evaluations are to:

• contribute to the design of interventions, including providing input for
setting political priorities,
• assist in an efficient allocation of resources,
• improve the quality of the intervention,
• report on the achievements of the intervention (i.e. accountability).

Similarities There are similarities and differences between performance audit and
evaluation. Both activities involve the examination of policy design,
implementation processes and their consequences to provide an assessment
of economy, efficiency and effectiveness of an entity or activity. They require
similar knowledge, skills and experience and involve similar methods for
collecting and analysing data. The main difference is the context in which they
take place and the purpose of each.

Differences Performance audit is superimposed on an accountability framework, which

implies that the Commission and other institutions and organisations
concerned are held responsible for the management of EU funds and should
provide meaningful and reliable information to demonstrate and take
responsibility for performance in light of agreed expectations. Performance
audits are carried out by auditors who maintain their independence to select
and determine the manner in which to conduct their work, and report the
results to the discharge authority (European Parliament acting on the
recommendation of the Council).
It is therefore not the purpose of the ECA's performance audits to deliver
comprehensive evaluations of EU activities. This is the responsibility of the
Commission, Member States and other managers of EU activities. However,
performance audits will usually include evaluative elements of selected
subjects and consider evaluation systems and information with a view to
assessing their quality and, when they are considered to be satisfactory and
relevant, use evaluation information as audit evidence.

1.3 THE ECA'S MANDATE & OBJECTIVES FOR PERFORMANCE AUDITS

1.3.1 The ECA's legal obligations

The legal framework for the ECA's performance audits of EU activities is laid
Treaty down by the Treaty. Article 287(2) of the consolidated text of the Treaty states
that "The Court of Auditors shall examine...whether the financial management
has been sound".

11
Commission's Communications on evaluation: (SEC (2000) 1051), Focus on Results: Strengthening Evaluation of Commission Activities
and SEC(2007)213, Responding to Strategic Needs: Reinforcing the use of evaluation .

Chapter 1: Framework for performance audits by the European Court of Auditors - page 11
1.3.2 The ECA's objectives

The ECA's Mission Statement provides that:

Mission
The EU`s independent external auditor
As the EU's independent external auditor, the ECA contributes to improving EU
financial management, promotes accountability and transparency, and acts as
the independent guardian of the financial interests of the citizens of the Union.
The ECA checks if the budget of the European Union has been implemented
correctly, and that EU funds have been raised and spent legally and in
accordance with the principles of sound financial management. As Europe
faces ever greater challenges and increasing pressure on its public finances,
the ECA’s role increases in importance.
Role
External auditors
Mission and role In democratic societies, complete, accurate and readily available information
on budgetary and policy implementation is essential for effective scrutiny and
decision-making. Such information helps promote sound financial management
and serves as a basis for accountability. Like Member States, the EU needs an
external auditor which can act as an independent guardian of the financial
interests of its citizens.
The EU’s “financial watchdog”
As Europe faces ever greater challenges and increasing pressure on its public
finances, the role of the European Court of Auditors is of increasing
importance. The ECA warns of risks, provides assurance and offers guidance
to EU policymakers on how to improve the management of public finances and
ensure that Europe’s citizens know how their money is being spent. This is the
essence of the ECA’s contribution to strengthening the democratic legitimacy
and sustainability of the European Union.
The ECA's systems-audit work in the Member States may only include an
assessment of systems and procedures for the management of the Union's
revenue and expenditure. An audit of purely national procedures, financed
access to systems and procedures solely from the national budget and not connected to the management of the
in Member States Union's revenue and expenditure, is outside the ECA's mandate. However, the
ECA is entitled to carry out fact-finding visits to the Member States but only to
the extent that such visits are necessary in order to ascertain how the Union is
performing its role of managing Union revenue and expenditure.
There is no clear legal basis for the ECA’s access to non-Member State
systems even where the EU funding regulation confers on the Commission the
power to entrust experts with the task of “control activities”, e.g. on-the-spot
access to systems and procedures
in non-Member States
checks. Each policy regulation has to be studied individually. However, the
ECA may legitimately ask the Commission to be present during such on-the-
spot checks. Such participation should be subject to the agreement of the
Commission.
As indicated earlier, the ECA's performance audits are superimposed on a
is to provide independent public accountability framework. The objective of the ECA is therefore to
information provide independent information to the discharge authority and to the
European public as a whole:

Chapter 1: Framework for performance audits by the European Court of Auditors - page 12
 • on the economy, efficiency and effectiveness with which the Commission
and other audited entities have used EU resources; and
on the 3Es • on the effectiveness of performance management systems of the
Commission and other audited bodies, including the reliability of
statements about performance produced.
The ECA aims to contribute to improving the financial management of EU
and make recommendations.
funds by making recommendations. Such improvement might involve:
• financial savings;
• better working methods;
• avoidance of waste;
• more cost-efficient achievement of stated objectives.
The perspective of the citizen that is related to the performance of the audited
entity should be taken into account where appropriate.

Chapter 1: Framework for performance audits by the European Court of Auditors - page 13
 PERFORMANCE AUDIT MANUAL

CHAPTER 2
THE PERFORMANCE AUDIT APPROACH AND THE 3 ‘E’s

Chapters in the
TABLE OF CONTENTS
Performance Audit Manual

2.1 Introduction
General Introduction 2.2 An audit approach focusing on performance achieved
2.2.1 Auditing performance directly
2.2.2 Auditing control systems
Chapter 1: Framework for 2.3 How to apply the 3 Es
Performance Audits by the
2.3.1 The use of logic models in performance audits
European Court of Auditors
2.3.2 Application of the concepts
 Economy
Chapter 2: The Performance  Efficiency
Audit Approach and the  Effectiveness
3 ‘E’s
2.4 Essential qualities of good performance audits
2.4.1 Sound judgement is exercised throughout the audit process
2.4.2 Methodologies are appropriate and combined to capture a range of data
Chapter 3: Planning the
audit 2.4.3 Audit questions are set which can be concluded against
2.4.4 Risks to delivering the audit report are analysed and managed
2.4.5 Tools are employed to help achieve successful delivery of the audit
Chapter 4: Examination 2.4.6 Evidence is sufficient, relevant and reliable to support the audit findings
Phase
2.4.7 Possible conclusions and recommendations of the final report are considered
from the planning phase onwards
2.4.8 Transparency - a 'no surprises' approach - is adopted with the auditee and other
Chapter 5: Reporting Phase stakeholders
2.5 Quality control

For any further information, please contact:

European Court of Auditors
Who to contact
Directorate of Audit Quality Control Committee (DQC)
E-mail: ECA-DQC-CONTACT or [email protected]

Chapter 2: The Performance Audit Approach - page 14

 2.1 INTRODUCTION

The audit approach to be employed must be one that produces the most
meaningful audit result, in the most cost-effective manner. For any specified
audit, a combination of approaches may be used.

2.2 AN AUDIT APPROACH FOCUSING ON PERFORMANCE ACHIEVED

Performance audits should provide information that is oriented towards the

performance achieved and is of primary interest to the European Parliament,
the Council, the Commission and other audited bodies. This means that, rather
than being driven by control and process concerns, the ECA's performance
audits should focus on performance achieved and assess the effect of audit
findings in terms of economy, efficiency or effectiveness. Performance audits
will combine the following approaches, with a different emphasis to be placed
on one or the other depending on the specific circumstances of the audit:
Approach Focus

Auditing
Inputs, outputs, results and impacts.
performance directly
Adequacy of policies and procedures
Auditing
implemented by managers for promoting,
control systems
monitoring and evaluating performance.

2.2.1 Auditing performance directly

This approach focuses directly on the performance achieved and concentrates

on inputs, outputs, results and impacts, the assumption being that, if the
performance achieved is satisfactory, there is little risk of serious problems
being present in the design or implementation of the activity or control systems.
Such audits may, for example, assess whether the adopted policies have been
suitably implemented and whether they have achieved the intended objectives
or whether there are undesirable financial, economic, social and environmental
consequences of policy decisions taken.
Examining performance directly is appropriate where there are suitable criteria
to measure quantity, quality and cost of inputs, outputs, results and impacts.
Suitable criteria essential Where performance achieved is found to be unsatisfactory, the activity and
control systems are then examined to the extent necessary to identify the
related causes.

Chapter 2: The Performance Audit Approach - page 15

2.2.2 Auditing control systems

This approach is designed to determine whether the Commission and other

audited entities have designed and implemented management and monitoring
systems so as to optimise economy, efficiency and effectiveness within the
Verify design and implementation given constraints. The audit work will involve analysing, reviewing and testing
of systems, the key components of such systems. The examination will often consider
whether measures chosen are consistent with the policy objectives, and
whether the latter have been translated into operational plans containing
operational objectives, the achievement of which is subsequently measured.
It will also consider whether the systems in place produce relevant, reliable and
timely information on the development of financial, human and other resources
(inputs), the carrying out of activities (processes) and the delivery of outputs,
which should be compared with the operational objectives by way of
performance indicators. It will examine whether, when discrepancies arise,
including information systems. timely and appropriate remedial action is taken to adjust the operational plan,
the deployment of resources and/or the carrying out of activities. This approach
will often involve an examination of the evaluation system and information in
order to assess their quality and, when considered to be satisfactory and
relevant to the audit objectives, to use evaluation findings, conclusions and
recommendations as audit evidence.

2.3 HOW TO APPLY THE 3 ’E’s

2.3.1 The use of logic models in performance audits

Set out the logic of the intervention Regardless of its nature (policy, programme, measure, project), a public
intervention can be analysed as a set of financial, organisational and human
resources mobilised to achieve, in a given period of time, an objective or set of
objectives, with the aim of solving or overcoming a problem or difficulty
affecting targeted groups. The use of logic models can help the audit team to
identify and set out the relationship between the socio-economic needs to be
addressed by the intervention and its objectives, inputs, processes, outputs,
and outcomes, which include results (immediate changes that arise for direct
addressees at the end of their participation in a public intervention) and
impacts (longer-term effects of the intervention). The following diagram shows
the example of the Programme Logic Model.

Chapter 2: The Performance Audit Approach - page 16

Chapter 2: The Performance Audit Approach - page 17
 Theoretically, it should be possible for performance audits to scrutinise all
components and relationships in such models by focusing on the 3 ‘E’s -
economy, efficiency and effectiveness.

2.3.2 Application of the concepts

Auditors should identify potential risks to achieving economy, efficiency and

effectiveness and thereby develop audit questions. Each concept is basically of
equal importance and where the specific priority lies will be decided on a case-
by-case basis; however, auditors are encouraged to consider effectiveness as
an element of the analysis whenever possible. A performance audit is not
Use potential risks to achieving the
3Es to develop audit questions.
supposed, and should not aim, to cover a simultaneous and comprehensive
examination of all aspects of economy, efficiency and effectiveness. It will
rather examine certain issues related to economy, efficiency or effectiveness,
or to a combination thereof, based on the significant potential risks identified.
By being selective in this manner, the audit is less likely to run the risk of being
overly ambitious.
The following pages identify, for each of the concepts of economy, efficiency
and effectiveness:

• the general risks to sound financial management

• issues to be addressed in an audit
• examples of risks in the EU context
• examples of audit questions related to the concept
• the focus of an audit in the area

 Economy

Issues of economy arise when an entity or activity could reduce the costs of
inputs significantly for a given level of outputs or results. General risks in this
area can include:

• waste, i.e. using resources which are not necessary for the achievement of
the desired outputs or results;
• overpaying, i.e. obtaining resources which are used, but could have been
Keeping the costs low
obtained at a lower cost; and
• gold-plating, i.e. paying for a higher quality of input than that required to
achieve the desired outputs or results.
An audit of economy is therefore concerned with determining whether the most
to achieve given objectives appropriate and lower-cost inputs are chosen to achieve the given objectives. It
will deal with issues such as whether:

• the audited entity acquires the appropriate type, quality and amount of
resources at the minimum cost;
• the audited entity manages its resources with a view to minimising overall
outlay;
• the intervention could have been designed or implemented in another way
which would have resulted in lower costs.

Chapter 2: The Performance Audit Approach - page 18

 Example of risk - Invalidity pension scheme of the European institutions
With some rare exceptions, the files examined did not contain any indication
that serious action had been taken beforehand with the aim of finding an
alternative solution and suggestions from the medical officer, mainly regarding
a change of post, were practically never followed up. An alternative to an
invalidity pension was seldom offered, or was offered too late. One quarter of
the sample of former members of staff receiving invalidity pensions would
probably have been able to continue working if the institution had detected and
treated their cases at an early stage. This is confirmed by the replies received
in response to the questionnaire: 20 % of the former members of staff thought
that they would have been able to continue working had an alternative solution
been offered. On that basis, it can be estimated that, in principle, savings of
about 10 million euro could be made every year on the net cost of the invalidity
pensions granted in the year, by implementing a policy of early detection and
treatment of repeated or prolonged periods of absence due to illness.

Examples of audit questions related to economy

Auditing performance directly
• Have best prices been obtained for consultancy services for support
programmes for SMEs?
• Is there potential for equitably reducing the cost of sickness absences?
Auditing control systems
• Does the Commission management of the cotton production aid scheme
include consideration and monitoring of the costs, including those of
consumers?
• Are there procedures in place to ensure that the transport costs of food aid
are the lowest available and compare favourably with costs incurred by
other donors? Are these procedures adequate and being properly applied?
Considerations of economy often lead the auditor to examine processes and
management decisions internal to audited entities regarding the procurement
of goods, works and services. The auditor will determine in particular whether
the procurement process has resulted in the best value being obtained. Areas
and often focusing on to be audited will include, for example, the establishment of detailed user
procurement. requirements to determine what is expected to be achieved through the
purchase of goods, works and services, the identification of the quality required
in relation to the required outputs and the determination of the desired timing
for the delivery of the goods, works and services. The auditor will also often
examine the drawing-up and implementation of selection and award criteria.

Chapter 2: The Performance Audit Approach - page 19

  Efficiency

Issues of efficiency arise when an entity or intervention could increase the

amount or quality of outputs or results without increasing the use of resources.
General risks in this area can include:

• leakages, i.e. resources used do not lead to the desired outputs;

• non-optimal input/output ratios (e.g. low labour efficiency ratios);
Making the most of available
resources • slow implementation of the intervention; and
• failure to identify and control externalities, i.e. costs imposed on individuals
or entities falling outside the boundary of the intervention or organisation.
An audit of efficiency is therefore concerned with assessing whether the best
relationship exists between the resources employed and the outputs or results
produced. Efficiency is closely related to the concept of "productivity" and the
to maximise "productivity",
key question is whether outputs or results have been maximised in terms of
quantity, quality and timing for the level of resources available. The audit will
deal with issues such as whether:

• outputs or results have been produced cost-effectively;

• there are any avoidable bottlenecks or unnecessary overlapping.

Example of risk - Forestry measures within rural development policy

Regarding the objective of increasing woodland areas, there are different kinds
of land which might be afforested. The Regulation stipulates that where
support is granted for afforestation of agricultural land owned by public
authorities, it shall cover only the cost of establishment, i.e. no loss of income
compensation or maintenance are to be paid, which is indeed the case for
afforestation of private land. Thus, theoretically, focusing this action on public
land would be much more efficient as it would, with the same funding, allow for
a considerably bigger area to be afforested.

Chapter 2: The Performance Audit Approach - page 20

 Examples of audit questions related to efficiency
Auditing performance directly
• How does the cost per job created by the EU training programme for the
long-term unemployed compare with similar costs per job elsewhere?
• Could the projects have been implemented in a different manner which
would have resulted in improved timeliness and quality?
Auditing control systems
• Are there adequate procedures to prioritise and select transport
infrastructure projects to ensure the maximum impact from EU funds? Are
the criteria used appropriate and are they being implemented as intended?
• Do the EU institutions have and consistently use adequate key
management information about the size, condition, utilisation and cost of
their office space for decision making?
When the audit objective of efficiency considers outputs, it often leads the
auditor to examine the processes by which an organisation transforms inputs
into outputs. The assessment can involve the calculation of unit cost of outputs
and may consider outputs produced (e.g. average cost per hour of training) or labour efficiency ratios
(e.g. number of subsidy applications processed per day) and their comparison
with accepted criteria, which can be derived from similar organisations,
previous periods or standards which the audited entity has explicitly adopted.
When the audit objective of efficiency encompasses results, economic tools
are generally necessary to assess the ability or potential of an audited entity,
operation or programme to achieve certain results at a given cost. As an
example, cost-effectiveness analysis can be used to relate the net effects of an
intervention to the financial inputs needed to produce those effects; the
and/or results.
judgement criteria might, for example, be the cost per unit of result produced
(e.g. cost per job created) which is then compared to that of other interventions
chosen as benchmarks. Depending on the audit approach, the auditors will
either examine the reliability of the analysis performed by the audited body or
carry out such analysis themselves.

Chapter 2: The Performance Audit Approach - page 21

  Effectiveness

Issues of effectiveness arise when an entity or intervention does not produce

the expected outputs, results or impacts. General risks in this area can include:

• faulty policy design, e.g. inadequate assessment of needs, unclear or

incoherent objectives, inadequate means of intervention or impracticability
Achieving the stipulated aims or of implementation; and
objectives,
• management failures, e.g. objectives not being met, management not
prioritising the achievement of objectives.
An audit of effectiveness is therefore concerned with measuring the extent to
which the different types of objectives have been achieved:

• operational objectives: the audit assesses the extent to which the intended
outputs have been produced and normally involves the examination of the
whether operational (outputs),
operations internal to the organisations which are responsible for the
implementation of the intervention;

• immediate objectives: the audit assesses whether the intervention had

clear and positive results for direct addressees at the end of their
immediate (results), participation and normally involves examining monitoring information
produced by the implementing organisations as well as obtaining
information from direct addressees;

• intermediate and global objectives: the examination extends beyond the

boundaries of the audited entity and seeks to measure the impacts of the
public intervention. This requires the audit to take account of exogenous
intermediate or global (impacts).
factors and to produce evidence that the impacts observed are actually
produced by the public intervention concerned and are not the
consequences of such factors.

Example of risk - Support regime for the production of dried fodder

The scheme provides for two rates of aid: a low rate for fodder dried naturally
by the sun and a high rate for machine-dried fodder, to compensate for the
extra fuel costs.
The differentiation in aid rates has had a big impact on how fodder is dried. At
the time of Spain’s accession to the Community in 1986, only 61 000 tonnes of
fodder was dried artificially each year; in 1996/1997, 1 414 000 tonnes was
dried in this way as producers found the higher rate of aid more profitable. The
annual energy consumption of this regime in Spain alone is sufficient to meet
the annual electricity needs of a town the size of Alicante (285 000 citizens).
More than 200 000 hectares of forest are needed to absorb the additional
carbon dioxide produced by the drying process.

Chapter 2: The Performance Audit Approach - page 22

 Examples of audit questions related to effectiveness
Auditing performance directly
• Have the support schemes for the early retirement of elderly farmers and
the setting-up of young farmers had a clear and positive impact on the
modernisation and economic viability of holdings in less-favoured areas?
• Have infrastructure projects contributed to increasing traffic flow while
reducing journey times and improving safety?
Auditing control systems
• Have Member States set up and properly implemented suitable measures
to monitor and mitigate environmental impact in the sugar sector?
• Have the Commission and the Member States carried out an adequate
assessment of needs and possible benefits arising from the Early School
Leavers programme to support the funding decision?
The audit of effectiveness will therefore concentrate on outputs, results or
impacts:

• There can be considerable difficulty involved in assessing the impact of an

intervention, i.e. the extent to which the global and even intermediate
objectives of this intervention have been achieved. The difficulty arises
because the objectives are usually expressed in such broad terms that they
cannot be associated with measurable indicators and the extent of their
achievement is therefore difficult to verify. Likewise, when the objectives
Assessing impact is difficult, are more clearly identified, the collection and analysis of the required audit
evidence would involve disproportionate audit resources if this information
is not readily available within the audited entity. Furthermore, it is difficult to
assess whether the impacts observed are really the effects of the
intervention rather than exogenous factors. In such instances, the audit
approach should therefore first consider whether relevant and reliable
evaluation information is available and can be used as audit evidence.

• A more feasible audit objective will often be to assess the outputs or results
of an intervention, i.e. the extent to which operational or immediate
objectives have been achieved. Provided that the objectives are "SMART" -
but easier to assess outputs or
results.
specific, measurable, achievable, relevant and timely - in accordance with
the Commission's performance and risk management approach, and that
their achievement is monitored by performance indicators, this is likely to
provide a clear and suitable reference basis for assessing effectiveness.

Chapter 2: The Performance Audit Approach - page 23

 2.4 ESSENTIAL QUALITIES OF GOOD PERFORMANCE AUDITS

The performance audit process comprises three phases - planning,

examination and reporting. To ensure as far as possible the successful delivery
of the performance audit, certain qualities need to be present during these
phases.
In order to establish a framework for timely delivery of high-quality audit reports
and to avoid unnecessary activities, performance audits must be undertaken
using the SMARTEST approach:
The auditor should ensure that:
Sound judgement is exercised throughout the audit process.
Methodologies are appropriate and combined to capture a range of data.
Audit question(s) are set which can be concluded against.
Risks to delivering the audit report are analysed and managed.
Important from start to end of audit
Tools are employed to help achieve successful delivery of the audit.
Evidence is sufficient, relevant and reliable to support the audit findings.
Significant / substantive conclusions and recommendations to the final report
are considered from the planning phase onwards.
Transparency - a 'no surprises approach' - is adopted with the auditee.

The essential elements of these qualities can be described as follows:

2.4.1 Sound judgement is exercised throughout the audit process

By nature, performance auditing is wide-ranging and open to judgement and

interpretation; indeed, every facet of a performance audit requires professional
Determines quality of audit work
and report
judgement and individual initiative. The ECA's reputation and credibility, the
cost-effectiveness of the audit, and the quality of the report depend on sound
judgement being exercised throughout the entire audit process.
In particular, sound judgement should be exercised in setting the audit
objectives (also known as the audit questions), defining relevant audit criteria,
establishing an appropriate quantity and quality of audit evidence, deriving
audit findings, drawing conclusions and reporting.

Chapter 2: The Performance Audit Approach - page 24

2.4.2 Methodologies are appropriate and combined to capture a range of data

A methodology is a technique for collecting or analysing data that helps to

provide evidence that enables conclusions to be drawn from the audit work.
Combination increases quality of Ideally, several different methodologies will be employed in order to capture a
evidence range of data and corroborate findings from various sources, thus increasing
the quality and reliability of the audit evidence in support of the audit findings,
conclusions and recommendations.
Examples of methodologies include reviewing files or literature, conducting
interviews, or carrying out surveys. A performance audit will also normally
combine methodologies to obtain quantitative and qualitative data; quantitative
and depends on the audit question
data is numerical in nature, whilst qualitative data is non-numerical. The most
appropriate combination of methodologies should be determined by the
subject matter and the audit questions being addressed.

2.4.3 Audit questions are set which can be concluded against

For performance audits, it is appropriate to set out the audit objectives in the
form of questions that the audit is to answer. In order to provide a proper focus
to the audit and to prevent the audit team from undertaking an overly-ambitious
scope of work, there should be one overall audit question together with a
limited number of sub-questions to be concluded against. The wording of these
questions is decisive for the results of the audit - they are the fundamental
research questions to which the auditors are seeking answers; thus,
ambiguous or vague questions are to be avoided.
The audit questions should then be further converted into lower-level
questions, the lowest level of which can be answered by carrying out specific
audit procedures. All sub-questions in the hierarchy should be both mutually
exclusive and collectively exhaustive (so that together they are sufficient to
answer the immediately preceding higher-level question).
All questions in the hierarchy should be capable of yes/no answers, so as to
enable the audit work to be focused on a specific end-product. However, this
does not mean that the only possible answer to such a question is yes or no;
the answer would obviously be much more developed. Furthermore, the terms
'yes' or 'no' are not intended to be included in the audit report; this approach is
purely a tool to help enhance a disciplined approach to audit questions and to
focus audit work. Audit questions should not, therefore, be formulated in an
inconclusive manner, such as "Assess the extent to which...", as this may
result in audit work that does not have a clearly defined scope and becomes
too extensive and time-consuming.
To facilitate the development of a good hierarchy of audit questions, audit
teams should carry out an issue analysis exercise before writing the Audit
12
Planning Memorandum (APM) .

12
CH 324/11 of 30 September 2011 on Audit Quality management. Detailed guidance on the process can be found in the Audit Guideline on
Issue Analysis and Drawing Conclusions (internal documents).

Chapter 2: The Performance Audit Approach - page 25

2.4.4 Risks to delivering the audit report are analysed and managed

The ECA must ensure that the period from the adoption of the APM to the
13
adoption of the special report does not generally exceed 13 months . The
starting point for this period is the adoption of the APM (or a later date when
the audit is planned to commence, as specified in the APM); the end date is
14
that of the adoption of special report APC
Risks to the timely and quality delivery of the audit report include difficulties in
obtaining data, unavailability of staff, and lack of co-operation by the auditee.
By preparing a risk management strategy, disseminating it to the audit team,
and referring to it as the audit progresses, the audit team is in a much better
position to manage the risks and to respond effectively if problems arise.
At all stages of the audit, the audit team is to identify:
• what could go wrong;
• how likely it is to go wrong;
• what would be the impact of it going wrong;
• what can be done to minimise the chances of it going wrong; and
• how the risk can be managed, should it materialise.

2.4.5 Tools are employed to help achieve successful delivery of the audit

The use of appropriate tools will help to ensure the development of a realistic
plan, and facilitate ongoing monitoring and review of actual achievement
against plan.
The basic planning instrument - the (APM) - is a "contract" agreed between the
responsible Member and the Audit Chamber. The APM defines the audit, the
product to be delivered, the resources to be employed, and the delivery date. It
includes an assessment of the risks to sound financial management, the audit
questions, audit criteria, evidence to be collected or generated, and the
methodology to be employed.
Tools and activities to help achieve successful delivery include:

• Issue Analysis and Drawing Conclusions (IADC) to produce quality reports

that are clear, brief and rigorous, and have an impact;
• clear definition and communication of responsibilities;
• a work plan showing audit team members and the timing of key milestones;
• monitoring tools which assist in keeping the work on track;
• progress reviews, and corrective action where necessary;
• document management established for paper records and ASSYST;
• quality-control procedures that are built into all aspects of the audit
process.

13
The FR. Article 163(1) applicable from 1 January 2016 requires that the Courts special reports are drawn up and adopted within an
appropriate period of time, which shall in general not exceed 13 months.
14
APC – “Après procedure contradictoire” (after adversarial procedure).

Chapter 2: The Performance Audit Approach - page 26

2.4.6 Evidence is sufficient, relevant and reliable to support the audit findings

Evidence collected during the audit provides a factual basis for developing
observations and concluding against the audit questions. It provides
persuasive support for a fact or point in question. As such, it is evidence that
must support the contents of an audit report, particularly all observations and
conclusions leading to recommendations. The audit evidence gathered should
thus be sufficient (in quantity), relevant (to the audit questions) and reliable
(objective and trustworthy).
The quantity and quality of evidence needed depends on the subject matter
and the audit questions. Evidence is stronger when provided by the auditor or
obtained from multiple sources and corroborated.

2.4.7 Possible conclusions and recommendations of the final report are considered from the planning
phase onwards

The audit team needs to assess early on whether clear conclusions and
recommendations are likely to emerge from their work. This will encourage
them to think from the outset about the likely messages to be delivered to the
target audiences and how to maximise the utility and impact of the report.

15
2.4.8 Transparency - a 'no surprises' approach - is adopted with the auditee and other stakeholders

The development of good and proper relations with stakeholders is a key factor
in achieving effective and efficient performance audit results. As performance
audits are not normally conducted on a regular (e.g. annual) basis on the same
audited entities, channels of communication may not already exist. Therefore
auditors should seek to establish and maintain good professional relationships
with all stakeholders involved in an audit, promote a free and frank flow of
information insofar as confidentiality requirements permit, and conduct
discussions in an atmosphere of mutual respect and understanding of the
respective roles and responsibilities of each stakeholder.
The communication process between the auditor and auditee begins at the
planning stage of the audit and continues throughout the audit process, by a
constructive process of interaction, as different findings, arguments and
perspectives are assessed.
Discussion of the audit with the auditee at the earliest possible opportunity
during the planning phase lessens the possibility of disagreement at a later
stage of the audit. It enables the auditee to understand the purpose of the
audit, the audit questions being tested, the criteria to be used and the
methodology to be employed. As there are often no predefined audit criteria for
the subject being audited, a full exchange of views with the audited entity is
necessary. Such discussion will also help to determine at the outset if the topic
is in fact relevant and auditable. Furthermore, early contact also helps to
establish a sense of constructive dialogue, which should be maintained
throughout all phases of the audit process.

15
ISSAI 300 – Fundamental principles of performance auditing, paragraph 29.

Chapter 2: The Performance Audit Approach - page 27

This approach is in keeping with good audit practice and the ECA's 'no
surprises' approach. In particular, the audit objective, questions, criteria, and
scope should be communicated preferably in writing, and insofar as possible
agreed, ideally between the responsible Member and the relevant Commission
Director-General, prior to the APM being presented to the Audit Chamber.
Where important audit findings are made during an audit, these should be
communicated in a timely manner to those charged with corporate governance.
Notwithstanding the above requirement, auditors should not communicate to
third parties, either in writing or orally, any information they obtain in the course
of audit work, except where doing so is necessary to discharge the ECA’s legal
or regulatory responsibilities. Any such communication should be governed by
the ECA’s rules of procedure. Auditors, however, may exchange information
regarding management deficiencies with internal auditors, should this
information not be of a data-security or confidential nature, for the purpose of
ensuring that identified shortcomings are addressed.

2.5 QUALITY CONTROL

Auditing Standard ISSAI 40 sets out the six elements of a system of quality
control:

• Leadership
• Relevant ethical requirements
• Acceptance and continuance of auditee relationships and specific
engagements
• Human resources
• Performance of audits and other work – see below
• Monitoring – in the language of the ECA, this is equivalent to the ex post
annual quality assurance exercise carried out by DQC

The fifth element, performance of audits and other work, comprises two main
functions: supervision and review, and engagement quality control review
(EQCR). Supervision and review refers to the normal hierarchical monitoring of
work conducted within the audit team. Thus, at its most basic level, the head
of task monitors the work of team members, and the reporting member
monitors the work of the head of task. However, the audit team may agree
with the chamber directorate team that additional support be provided to the
supervision and review activity, such as advice and guidance from a principal
manager or subject matter expertise from a policy expert.
The second function of this element of quality control is EQCR, an objective
evaluation of the significant judgments made by the audit team and the
conclusions reached in formulating the report. For performance audits, EQCR
is carried out (as a minimum) at the APM and preliminary observations stages.
It is undertaken by staff independent of the audit team and is designed to
provide additional assurance to the chamber that audits have been performed
in accordance with professional standards and applicable legal and regulatory
requirements, and that the reports issued are appropriate in the circumstances.
Details on how EQCR operates for performance audits are in the Vademecum
of General Audit Procedures – Audit Quality Management Framework (internal
document).

Chapter 2: The Performance Audit Approach - page 28

 PERFORMANCE AUDIT MANUAL

CHAPTER 3
PLANNING THE AUDIT

Chapters in the
TABLE OF CONTENTS
Performance Audit Manual

3.1 Introduction
3.2 The audit planning phase
General introduction 3.2.1 Purpose of audit planning
3.2.2 Preliminary work
3.2.3 Acquire up-to-date knowledge of the audit area
 Identify the objectives and logic of the intervention and related indicators
Chapter 1: Framework for
 Determine the resources made available for the intervention
Performance Audits by the
European Court of Auditors  Determine the respective responsibilities of the various actors
 Identify key management and control processes, including IT systems
 Define information needs for management and control purposes
 Identify the risks to sound financial management
Chapter 2: The Performance
3.2.4 Outline the audit
Audit Approach and the
3 ‘E’s  Take account of previous audits and evaluations
 Consider the potential audit questions, criteria, evidence, methodology, scope
and impact
 Consider the timing and resourcing of the proposed audit
Chapter 3: Planning the 3.2.5 Assess if the audit is realistic, realisable and likely to be useful
audit 3.3 The Audit Planning Memorandum
3.3.1 Purposes and contents of the APM
 Define the audit questions
Chapter 4: Examination  Set the audit scope
Phase  Establish the audit criteria to be used
 Identify the audit evidence required and its sources
 Define the audit methodology to be employed
Chapter 5: Reporting Phase  Consider the potential audit observations, recommendations and impact
 Determine the timetable, resources and supervision and review arrangements
 Communicate with the auditee
3.3.2 Drawing up the APM
3.4 Quick special reports
Annex I: Contents of an Audit Planning Memorandum
Annex II: Evidence Collection Plan
Annex III: Outline audit programme

For any further information, please contact:

European Court of Auditors
Who to contact
Directorate of Audit Quality Control Committee (DQC)
E-mail: ECA-DQC-CONTACT or [email protected]

Chapter 3: Planning the Audit - page 29

 3.1 INTRODUCTION

The varied nature of performance audits places particular emphasis on the

need for good planning. This helps to determine whether the audit is
worthwhile and feasible, sets clear and reasonable objectives, defines a
Good planning is essential,
realistic and robust audit approach, and establishes the resources needed. If
the audit is not well planned, there is a risk that the audit work will not be
efficient or effective.

Audit planning includes:

• Consideration of the significant risks to sound financial management, the
and is the basis of the APM. potential audit objectives, approaches and methodologies, and whether the
audit is realistic, realisable and likely to be useful.
• The APM, which is approved by the Audit Chamber, defines the audit
scope, objective(s) and methodology, the resources to be employed and
key milestones to be achieved in the proposed audit.

The following chart describes the audit planning phase; the audit planning and
APM are described in more detail in this chapter.

Chapter 3: Planning the Audit - page 30

Chapter 3: Planning the Audit - page 31
 3.2 THE AUDIT PLANNING PHASE

3.2.1 Purpose of audit planning

The ECA requires Audit Chambers to develop audit proposals for each topic
16
ranked as a priority . These proposals contain the relevant information
Preliminary work necessary to decide whether or not the audit is to be considered for inclusion in
for the preparation of the APM the Annual Work Programme (AWP). The preliminary work develops and
expands upon this information, which may lead to a reassessment of whether
to carry out the audit as already planned in the AWP.

3.2.2 Preliminary work

The extent of the preliminary work necessary depends on the audit team's
existing knowledge of the audit area, but must be such as to enable the
preparation of the APM. In order to determine whether the audit is realistic,
realisable and likely to be useful, auditors need to acquire up-to-date
Making use of existing information knowledge of the audit area. If the audit area and possible audit questions are
well known, or the audit subject derives from existing financial or compliance
audits, preliminary work may be unnecessary. Detailed audit testing work
should not be carried out at this stage; rather, the emphasis is on considering
the availability of information and the feasibility of methods.

16
ECA work programming: instructions and guidelines – 2016 up-date CA 018/16 (internal document).

Chapter 3: Planning the Audit - page 32

Carrying out the preliminary work

ACQUIRE UP-TO-DATE KNOWLEDGE OF THE AUDIT AREA

 Identify the objectives and logic of the intervention and related indicators
 Determine the resources made available for the intervention
 Determine the respective responsibilities of the various actors
 Identify key management and control processes, including IT systems
 Define information needs for management and control purposes
 Identify the risks to sound financial management

OUTLINE THE AUDIT

 Take account of previous audits and evaluations

 Consider the potential audit questions, criteria, evidence, methodology, scope and impact
 Consider the timing and resourcing of the proposed audit

ASSESS IF THE AUDIT IS REALISTIC, REALISABLE AND LIKELY TO BE USEFUL

Each of these steps is described in detail in the following paragraphs:

Chapter 3: Planning the Audit - page 33

3.2.3 Acquire up-to-date knowledge of the audit area

The emphasis is on building a sufficient understanding of the topic under

examination. The approach to gaining this knowledge will vary, depending on
the nature of the subject and the amount of knowledge that the team
possesses at the outset, including information available in the permanent files
and that already obtained by the ECA's financial auditors. It might involve
simple documentary review and a single brainstorming session when a great
Extent and sources of information deal is already known or a much larger data-gathering exercise when the topic
are varied. is new or complex.
Sources of information may be third parties (legislation; viewpoints from
experts in the field; scientific studies and research; official statistics) or the
auditee (mission statements; strategic and corporate plans; Annual Activity
Reports and Activity Statements of the Commission’s Directorates General
(DGs); organisation charts; internal guidelines and operating manuals; and
discussions with auditee management).
Auditors are advised to weigh the time needed to obtain information, and the
related costs, against its added value to the audit.

 Identify the objectives and logic of the intervention and related indicators
The Commission manages its administrative and operational resources
through Activity-Based Management (ABM), with management undertaken
around 'activities' which implement 'policy areas'. The ABM activities form the
principal lines of accountability for the Commission's management of its
activities and budget. The use of ABM as the basis for the Commission’s
approach to sound financial management requires that expenditure is based
17
on SMART objectives. In addition, the implementation of these objectives
18
must be monitored by the relevant DG through RACER indicators on output
Objectives and indicators
regarding the intervention and impact for each policy area and activity, with management required to take
are the starting point of the audit. action to address any identified shortfall against objectives.
An understanding of the objectives and logic of the intervention is the starting
point in planning a performance audit. Diagrammatic representations of the
logic of the intervention (inputs, processes and outputs/objectives) may be
made available by the auditee, or constructed by the auditor to facilitate this
understanding.

 Determine the resources made available for the intervention

The human, administrative and financial resources allocated to the audit area
should be determined to confirm its materiality. This involves analysing
budgetary appropriations allocated, and amounts committed and paid.

17
Objectives must be Specific, Measurable, Achievable, Relevant, Timely.
18
Performance indicators must be Relevant, Accepted, Credible, Easy and Robust.

Chapter 3: Planning the Audit - page 34

  Determine the respective responsibilities of the various actors
A cornerstone of performance audit is holding auditee management accountable
only for that over which they have control; it is therefore fundamental to define
the auditee. This is particularly pertinent in instances of shared or indirect
management between the Commission on the one hand and Member States,
beneficiary states or international organisations on the other. In such cases, the
Define who is responsible for
managing the intervention,
Commission may have little real direct involvement in the on-going management
and implementation of co-financed interventions, although it always has overall
19
responsibility for managing the Budget . In this context, the ECA requires its
performance audits to take as their starting point the Commission's
management, including the methods implemented, to ensure that management
by Member States is appropriate.
The audit team should identify management responsibility for the intervention
as this influences how and where on the part of the Commission, Member States, beneficiary states and
the audit is conducted. international organisations, by way of interview and review of organisation charts
and regulations.

 Identify the key management and control processes, including IT systems

The audit team should determine the key management and control processes
for the activity/activities which concern the potential audit topic. This can be done
by reviewing regulations and internal procedure manuals and by way of
interview. An important consideration in this regard is the IT systems used by the
auditee, and the level of IT internal control.

 Define information needs for management and control purposes

Review monitoring and project The audit team should identify the types of information and reports currently
selection criteria.
used by the auditee for management and control of its activities. This includes
the reports used for overall monitoring of the activity, and, where relevant, the
project selection criteria adopted to determine the basis on which projects are
selected for funding. Particular attention needs to be paid to identify data held in
IT systems and its impact on the audit approach; it is advisable to contact DQC
Consider IT data and its impact on at an early stage to get support in this regard.
the audit approach.

 Identify the risks to sound financial management

Risk is the probability that an event or action may adversely affect the
organisation, such as exposure to financial loss, loss of reputation, or failure to
Inherent risks?
deliver the policy or programme economically, efficiently or effectively. The
information obtained, as described above, provides auditors with a basis for
analysing the most significant risks to the achievement of sound financial
Control risks?
management. The main risks may be inherent risks (the factors that make
sound financial management hard to achieve, no matter how well the entity is
managed) or control risks (how well the entity manages performance).

19
See Article 287 of the TFEU.

Chapter 3: Planning the Audit - page 35

 In addition, the audit team needs to become familiar with the risk analysis and
risk management of the main activities developed by the relevant DG, as is
required by the ICS 6 'Risk Management Process' standard of the
Commission's internal control system. Elements of the ABM cycle - particularly
Information on risks is available
from many sources.
the Activity Statements and Annual Activity Reports - will provide an additional
useful source of information on risk. Each DG is required to systematically
analyse risks in relation to its main activities at least once a year, develop
appropriate action plans to address them, and assign staff responsible for
implementing those plans.
The auditor asks:
• what can go wrong?
• what is the probability of it going wrong?
• what would be the consequences of it going wrong?
• what is the auditee's strategy to minimise or control the risk?
Risk factors will include the:
• nature and complexity of the policy, programme and operations;
• diversity, consistency and clarity of the entity's objectives and goals;
• existence and use of appropriate performance measures;
• availability of resources;
• complexity of the organisation structure and clarity of responsibilities;
• existence and quality of control systems;
• complexity and quality of management information.
In planning the audit, the audit team should analyse the relative significance of
these risks, mapping the likelihood of occurrence against the likely impact, both
quantitative and qualitative. They will usually focus on those risks having both
Focus on major risk exposures.
a higher likelihood of occurrence and a higher impact if they do materialise,
whilst also considering the action taken by the auditee to mitigate such risks
('risk response').

3.2.4 Outline the audit

 Take account of previous audits and evaluations

The audit team should consider previous audits and evaluations undertaken
in the subject area, both in order to avoid duplication of work and to follow up
Consider audit and on significant findings and recommendations that relate to the potential audit
evaluation reports. question. Such audits may include those performed by the ECA or the
Commission's Internal Audit Service (IAS). Any evaluation reports will also
need to be considered.

Chapter 3: Planning the Audit - page 36

  Consider the potential audit questions, criteria, evidence, methodology, scope and impact
For performance audits, it is appropriate to set out the audit objectives in the
form of questions that the audit is to answer; henceforth, only the term ‘audit
questions’ will be used. The aforementioned risk analysis will help to provide
focus for both the potential audit questions and audit scope. When preparing
the APM, it is possible to consider different potential audit questions. The
auditor may interview people with special knowledge of the audit subject, as
well as studying basic documents and other literature.
Generating ideas for potential audit
questions is crucial. The audit team should then define which of these audit questions can be
answered. This is done by considering whether the questions identified are
auditable, i.e. whether audit criteria are available or can be developed, whether
audit evidence exists or can be generated and is accessible by the auditor, and
whether audit methodologies can be successfully employed to collect and
analyse such evidence. To facilitate this process, as well as identifying audit
questions which could more efficiently be addressed in a separate audit, audit
20
teams should carry out an issue analysis exercise before drafting the APM .

 Consider the timing and resourcing of the proposed audit

The audit team should consider the timetable for the audit, including any
preliminary work and the available resources. In this context, it should be
borne in mind that the timeframe for carrying out an audit is 13 months, from
Keep within the required timeframe
of 13 months.
the adoption of the APM (or a later date when the audit is planned to
commence, as specified in the APM) to adoption of the report APC. Experience
indicates that the time needed to carry out each stage of the audit should be
planned as realistically as possible on the basis of past performance.
Consideration also needs to be given to the possible impact of the report on
upcoming changes in legislation. Where possible, reports must be timed to
contribute to such changes and should have regard to the agendas of the
European Parliament and Council.
The audit team should also reflect on the availability of suitably qualified and
experienced auditors to carry out the proposed audit.

3.2.5 Assess if the audit is realistic, realisable and likely to be useful

In order to ensure an efficient planning process, preparatory work should not

result in the presentation of a separate report to the Chamber. Instead,
informal consultation on the preliminary work should be favoured. If
Informal consultation on considered necessary, an oral presentation could be made by the responsible
preliminary work, or oral Member to the Chamber in order to facilitate transparency and inform
presentations if necessary subsequent discussions on the APM.
Preparatory work should be limited to what is essential to research and plan a
realistic, realisable and useful audit and to prepare and present the Audit
Planning Memorandum.

20
CH 324/11 of 30 September 2011 on Audit Quality management. Detailed guidance on the process can be found in the Audit Guideline on
Issue Analysis and Drawing Conclusions (internal documents).

Chapter 3: Planning the Audit - page 37

 3.3 THE AUDIT PLANNING MEMORANDUM

3.3.1 Purposes and contents of the APM

Detailed planning of the audit is set forth in the APM. The APM is the "contract"
between the responsible Member and the Audit Chamber, by which the
Member responsible is committed to deliver a product (the audit results) in
accordance with quality standards, within the established deadlines, and in
Standard format of APM, exchange for the resources made available by the Audit Chamber. The APM
should identify in a clear and concise manner the audit work to be performed,
the resources and timeframes required, and the anticipated impact of the audit.
It should be submitted by the Member responsible to the Audit Chamber for
decision, in the format set out in Annex II.
The APM should show (e.g. by way of an Evidence Collection Plan - see
Annex III) how evidence will be obtained and analysed to answer the audit
questions. It should also contain an outline of the audit procedures required for
collecting and analysing the necessary information to allow the auditors to
21
to include
reach valid conclusions (see Annex IV Outline Audit Programme). The latter
Evidence Collection Plan and does not need to be developed to a very detailed level, as certain tests may
Outline Audit Programme. only be properly determined once audit work gets underway, and the detailed
tests required may change during the course of the audit.
In order to promote efficiency in planning, and creativity and flexibility in
conducting the performance audit, the audit team should as a rule avoid
preparing excessively detailed or sophisticated APMs.

The audit testing should not start until the APM has been approved by the
Audit Chamber. It is only once the APM is approved that the resources are
No audit work until APM approved.
formally committed to the audit task. The audit must be designed so that it can
be delivered within the timetable and with the resources agreed.
The issues to be addressed during the detailed planning of the audit, the
results of which will appear in the APM, are set out in the following figure.

21
See DQC Intranet for practical examples of Evidence Collection Plans and Audit Programmes for performance audits.

Chapter 3: Planning the Audit - page 38

Setting up the APM:

DETAILED PLANNING OF THE AUDIT - ISSUES TO BE ADDRESSED

 Define the audit question and sub-questions

 Set the audit scope
 Establish the audit criteria to be used
 Identify the audit evidence required and its sources
 Define the audit methodology to be employed
 Consider the potential audit observations, recommendations and impact
 Determine the timetable, resources and quality control arrangements
 Communicate with the auditee

ASSESS IF A SUFFICIENT BASIS HAS BEEN DEVELOPED FOR THE AUDIT

DRAFT APM

The key considerations regarding each of these aspects are detailed below:

Chapter 3: Planning the Audit - page 39

  Define the audit questions
22
The APM should define the audit questions based on the results of the Issue
Analysis exercise. The wording of the audit questions is of great importance for
Ensure that the audit questions can
be answered,
the audit, and is to be based on rational and objective considerations. Unless
care is taken in this area, it may prove difficult to gather sufficient, relevant and
reliable audit evidence to answer the questions.
In order to ensure that the audit objectives are thematically related,
complementary and mutually exclusive, audit questions may be presented as a
pyramid of questions with one overall audit question and a limited number of
sub-questions, which concentrate on one topic, and clearly identify the audit
subject (e.g. programme, policy or DG) and the performance aspects to be
that they concentrate on one topic,
audited. Ideally, the subject of the audit will, as far as possible, comprise
individual policy areas or components thereof, such as one or more activities or
actions as defined under Activity-Based Budgeting (ABB) and Activity-Based
Management (ABM), or individual operations such as an EU Agency. This will
facilitate the audit and help to ensure that the report is practical and focused.
In defining the audit questions, the following are to be considered:
Is the topic material? Will the potential impact of the audit be
Relevance
material? Is it important for stakeholders (including the
of the
discharge authority and the general public)? Are there risks to
questions
sound financial management?
are relevant and auditable,
Can all the questions be answered? Can an audit be carried
out and a conclusion reached in light of the availability of the
Auditability
necessary information, audit methodologies, resources and
audit skills? Are conditions appropriate in terms of timing?

with an appropriate focus on The focus of the audit questions may be either the examination of control
control systems and/or
performance. systems or the direct examination of performance, or a combination thereof:

• the audit of control systems: most performance audits will include an

examination of:
 the formulation of the objectives of the intervention, so as to conclude
on whether they are realistic, relevant and meaningful;
 the indicators used, in order to determine if they properly measure
progress towards achieving those objectives;
 the IT systems which support the management of programmes,
projects, etc., in order to determine if these systems deliver actual,
accurate and essential data and information, and to assess whether
such data and information are properly used.
 the underlying data, to determine reliability; and,
 the project selection criteria used to allocate resources.
• the direct examination of performance focuses on the achievement of
the auditee’s objectives. Objectives, if found to be properly conceived, are
the basis against which to judge the performance of the auditee. Similarly,
the indicators, if properly conceived, can be used to assess progress.

22
DQC has Guidelines for developing audit questions (internal document).

Chapter 3: Planning the Audit - page 40

 From audit question to sub- The audit question can then be broken down into sub-questions, which are in
questions.
turn broken down into sub-sub-questions. Usually, there are four such levels,
from the main audit question (Level 1) down to the detailed questions which
are answered by carrying out specific audit procedures (Level 4); the latter
form the basis for the sources of evidence. The sub-questions at each level
must be different from one another (mutually exclusive), but together cover the
main aspects of the question (collectively exhaustive) at the immediately higher
23
level .
24
Breaking down each audit question will form a pyramid . This helps to impose
a logical disciplined pattern on one's thinking and to ensure that all aspects of a
question or sub-question are considered.

How to break down the main audit-questions:

LEVEL 1
Audit question

LEVEL 2
Sub-question

LEVEL 3
Sub-sub question

LEVEL 4

Audit Procedures

Sources of Evidence

23
More detailed guidance on the development of audit questions and sub-questions can be found in the Audit Guideline on Issue Analysis
and Drawing Conclusions and the Guideline on Developing Audit Questions.
24
Minto B., The Minto Pyramid Principle, Ed. Minto International, Inc., 2003.

Chapter 3: Planning the Audit - page 41

Example of a developed pyramid of questions, adapted from the Audit Planning Memorandum on Devolution

Level 1: Audit
Is the Commission managing the devolution process successfully?
question
Level 2 Level 3 Level 4

1.1.1 Clear definition of the functions to be carried out by devolved delegations?

1.1.2 Definition of the functions consistent with the underlying principle?

1.1 Were needs in
delegations properly 1.1.3 Definition clearly communicated to, and understood by, staff?
assessed in preparing for
1. Have 1.1.4 Analysis of existing resources in delegations prior to devolution?
devolution?
delegations been Clear assessment of the resources needed to meet the future activities and aims of
well prepared for 1.1.5
delegations after devolution?
devolved
management? 1.2.1 Needs of delegations addressed successfully, on time and within budget?
1.2 Were needs in
delegations properly Did the Commission clearly address the question of whether needs (in staffing,
1.2.2
addressed in training, guidance, premises, IT) had been addressed?
implementing
Were difficulties encountered in addressing needs, and were the problems
devolution? 1.2.3
overcome?

2.1.1 Was there a clear definition of the key monitoring and support functions of HQ?

2.1.2 Was the definition consistent with the underlying principle?

2.1 Were headquarter
(HQ) needs properly 2.1.3 Was this clearly communicated to, and understood by, staff in HQ and delegations?
assessed in preparing for
2 Have central 2.1.4 Was there a clear analysis of existing resources at HQ?
devolution?
departments been
well prepared for Was there a clear assessment of the resources needed to meet the future activities
2.1.5
devolved and aims of HQ after devolution?
management?
2.2.1 Were the needs of HQ addressed successfully, on time and within budget?
2.2 Were HQ needs
properly addressed in Did the Commission clearly address the question of whether needs had been
2.2.2
implementing addressed prior to devolution?
devolution?
2.2.3 Were difficulties encountered in addressing needs, and were problems overcome?

3.1 Was devolution after Does the Commission ensure that needs continue to be addressed after sub-
3.1.1
sub-delegation in delegation and that delegations are operating effectively under devolution?
delegations properly
monitored? 3.1.2 Does the Commission ensure that the underlying principle is respected?

3.2 Was devolution after Does the Commission ensure that the needs of HQ continue to be addressed after
3. Does the 3.2.1
sub-delegation in HQ sub-delegation?
Commission have
effective properly monitored? 3.2.2 Does the Commission ensure that the underlying principle is respected?
procedures for
monitoring Were project management tools used to manage the overall devolution process and
3.3.1
performance of were they used appropriately and effectively?
devolved
Were the lessons learnt from the first and second waves of devolution applied to the
management? 3.3.2
3.3 Was project second and third waves?
management efficient?
3.3.3 Was there a clear and accurate estimate of the costs of devolution?

3.3.4 Do estimates compare with actual costs?

3.3.5 Were costs properly monitored and reported?

Chapter 3: Planning the Audit - page 42

  Set the audit scope
The scope defines the boundary of the audit, and is directly tied to the audit
questions. In particular, the auditors need to define:
the programme(s), ABM activity and action(s) & budget lines
 WHAT
to be audited

the Commission departments and/or other entities to be

 WHO
covered by the audit

 WHERE the geographical scope of the audit

 WHEN the time period to be covered
The auditor will need to consider the rationale for the scoping decisions. As it is
neither practical nor efficient to cover all possible aspects in a single audit, the
nature, extent and timing of audit procedures should be restricted to a limited
number of matters of significance. These are matters that pertain to the audit
Limit scope to ensure audit can be questions, can be carried out with the resources and expertise available, and
delivered
are critical to the achievement of the intended results of the audit subject.
When laws, regulations, and other compliance requirements pertaining to the
audit entity have the potential to significantly impact on the audit questions,
then the audit should be designed to address these issues in order to conclude
on the audit questions.

 Establish the audit criteria to be used

Audit criteria are standards against which the actual performance (adequacy of
systems and practices and the economy, efficiency and effectiveness of
Audit-appropriate criteria are activities) is compared or evaluated. They are required to assess existing
essential to assess performance conditions and produce audit findings (what is compared to what should be). It
is important that audit criteria are as objective as possible, so that room for
subjective interpretation is minimised.
As the general concepts of economy, efficiency and effectiveness need to be
interpreted in relation to the subject matter, audit criteria will vary from one
performance audit to another, and the choice is normally relatively open and
formulated by the auditor. However, audit criteria should be devised from
recognised sources, and be objective, relevant, reasonable and attainable.
Sources of audit criteria determine the effort needed to assure their suitability:
Criteria : objective, relevant, • criteria based on legislation, regulations or recognised professional standards are
reasonable and attainable,
among the most uncontroversial. Generally accepted criteria can also be obtained
from recognised sources
from sources such as professional associations, recognised bodies of experts and
academic literature;

• the other main sources of criteria for performance audits are the standards,
measures and results commitments adopted by auditee management, including
specific targets or requirements set by the Commission in the context of ABB/ABM;

• if criteria are not available from the above sources, the auditor can focus on
performance achieved in comparable organisations, best practices determined
through benchmarking or consultation, or standards developed by the auditor
through an analysis of activities.

Where the entity has adopted meaningful and specific measures for assessing
its own performance, those relevant to the audit should be reviewed to ensure
that they are reasonable and complete.

Chapter 3: Planning the Audit - page 43

 Where criteria are not self-evident and are open to dispute by auditee
management, they should be agreed insofar as possible in terms of their
relevance and acceptability. This approach recognises that the audit is not
and should be agreed with the simply searching for deficiencies to report. If suitable criteria cannot be
auditee insofar as possible. determined and agreed, the detailed audit question may need to be
reconsidered. In the event that disagreement persists, the audit report needs to
explain the criteria used. Under no circumstances can an audit be carried out
using criteria that could lead to biased or misleading audit results.

 Identify the audit evidence required and its sources

The audit evidence needed to answer the audit questions should be identified,
as well as the sources from which such evidence is to be obtained and whether
it is in a form which can be easily collected and analysed.
Identify sufficient, relevant and This evidence needs to be:
reliable evidence…
 SUFFICIENT to enable the main audit question to be fully answered
 RELEVANT to address the audit question being asked
 RELIABLE in terms of its impartiality and persuasiveness
Special attention should be given to audit tasks where personal data is used
… and remember data protection as audit evidence. According to Article 25 of Regulation No 45/2001, a data
notice when collecting personal controller (i.e. the person who is responsible for the treatment of personal data)
data is obliged to give prior notice to the Data Protection Officer. The APM should
25
make a reference to the treatment of personal data .
If in doubt, consideration will need to be given to the potential impact on the
audit if the evidence cannot be obtained at reasonable cost and whether
Sources of evidence. alternative sources of evidence need to be considered. If there is a high risk
that it will not be possible to obtain the necessary evidence, the audit question
will need to be revised.

 Define the audit methodology to be employed

Performance audits can draw upon a large variety of methods, commonly used
in the social sciences, to gather and analyse evidence, such as surveys,
interviews, observations and written documents. In choosing such methods,
auditors are to be guided by the purpose of the audit and the specific questions
to be answered. Clear, robust and practical methodologies should be
identified in order to obtain sufficient, relevant and reliable audit evidence so as
to be able to draw conclusions with reasonable certainty.

25
For example: “It is confirmed that no personal data will be treated during the audit”; or “It is confirmed that personal data from X (e.g.
describe the data population in receipt of an EU co-financed subsidy) will be used during the audit. A notification to the DPO has been
made on (date), which was registered as Data Protection treatment no XXX. The DPO accepted the notification with no remarks/with the
following remark(s) XXX.”

Chapter 3: Planning the Audit - page 44

 Different methodological approaches may be employed at different audit
phases and for different purposes (see various audit guidelines). Qualitative
techniques are particularly useful at the early stages of an audit to identify the
significant issues, develop preliminary ideas and build hypotheses. These
Qualitative and quantitative
techniques
techniques are also particularly well suited to complex problems. Quantitative
analysis, involving an examination of numerical data, is one of the most
powerful audit tools for developing evidence-based conclusions. Such analysis
adds considerable value to the audit work, as it can provide clear measures of
costs, benefits and performance.
A good performance audit will normally combine different methodologies to
capture a range of data and corroborate findings from different sources, and
combine qualitative and quantitative data. This combination of methodologies
The combination of methods is necessary to provide solid evidence to support the conclusions and
depends on audit questions recommendations, with the quantitative data giving the ECA the means to
demonstrate the significance of its observations and recommendations. It may
be necessary to pilot-test certain methods to ensure that they can provide the
evidence required to answer the audit questions.

 Consider the potential audit observations, recommendations and impact

The likely audit observations and recommendations should be considered.
This will not only help in providing an outline report structure, but also in
determining if the audit questions as defined are likely to result in constructive
recommendations. The audit team needs to be clear from the outset that the
Anticipate results.
audit has the potential to produce practical recommendations, and is
encouraged to look forward constantly and consider at what stage in the
Commission's programme-planning cycle the audit report will be published.
This will help to determine the likely impact and usefulness of the final report.

 Determine the timetable, resources, and supervision and review arrangements

Auditors need to make a realistic assessment of the human and financial
resources that the audit will require, ensure that the required knowledge and
experience is available within the audit team and, where necessary, anticipate
the need for external expertise. The APM should plan for a stable core audit
team for the duration of the audit, with a permanent team leader, deputy team
leader and responsible Member. The potential for efficiencies and reducing the
audit duration by increasing the team size should be considered.
A core audit team The audit should be planned and managed as a single project from APM to
APC stages using the ECA’s audit management and documentation tools
(Audit Management System (AMS) and ASSYST II). A realistic plan should be
established, indicating resources and responsibilities involved for each main
task throughout the audit lifecycle. The plan should detail the quantity of
resources required; competencies and expert knowledge of the audit team in
the subject matter, and possible external expertise if needed. It should also
And realistic plan is needed
give detailed timetables for each phase of the audit process (including report
publication), with reporting milestones established for each phase, and a focus
on external deadlines where relevant (e.g. the issue of new regulations in the
audited area).

Chapter 3: Planning the Audit - page 45

 As a general rule, no more than five Member States (or beneficiary countries)
should be visited on the spot in the course of an audit. The choice of Member
States should automatically include a smaller Member State (in terms of
population, geographic size or financial support received). Where deviations
from these general rules are considered necessary, they should be explained
in the APM.
The APM should set out the full timetable for the audit starting with any
preliminary work considered necessary to the planned publication date of the
report. In this context, the timeframe for an audit, from APM approval to the
scheduling 13 months from APM adoption of the special report (after the adversarial procedure) is 13 months.
approval to report adoption. Experience indicates that the time needed to carry out each stage of the audit
should be planned as realistically as possible on the basis of past
performance. In exceptional cases, where a longer timeframe is required, this
should be justified and approved by the Audit Chamber.
Significant risks to successful implementation of the audit and how these risks
can best be managed are to be identified. In planning the critical path of the
audit, it can be helpful to identify possible high-risk points where difficulties are
that considers the risks to delivery, more likely to arise, resulting in delays or putting quality at risk. Foreseeable
constraints on the availability of auditors (who may be required for other audit
tasks) and the consequences of late availability of audit results and the audit
report always need to be considered.
There may be value in accelerating audit fieldwork by employing a large team
to collect data quickly, whereas drafting and clearance of facts (especially the
adversarial procedure) might be undertaken by a much smaller team. By
reviewing the planned weeks for each auditor for reasonableness, there is
greater assurance that the number of weeks allocated is feasible in light of the
proposed timetable.
Quality control arrangements (see Audit Quality Management Framework –
VGAP – internal document) should be established for the assignment and
and builds in quality control,
understanding of responsibilities concerning the direction, supervision and
review of audit work, provision of timely feedback, and regular monitoring.
In order to allow for appropriate reflection by the Audit Chamber concerned,
and to add value by considering their different perspectives, provision should
be made for a progress report, to be submitted orally or in writing by the
Member responsible to the Audit Chamber. The timing of this report may vary
including a progress report. somewhat from one audit to another, but ideally is planned to take place upon
completion of a key milestone (e.g. the end of the first ‘block’ of missions). This
report is to include, whenever possible, an indication of the main observations
and conclusions already evident from the audit and a draft outline of the final
report.
Arrangements for documenting and managing audit processes should be
referred to in the APM including quality control procedures.

 Communicate with the auditee

Continuous dialogue and mutual understanding between auditors and auditee
personnel, emphasised in the ECA's 'no surprises approach', is essential in
order to gain acceptance for the conclusions and recommendations in the audit
report.

Chapter 3: Planning the Audit - page 46

 Regular contact with auditee is Contacts with auditee personnel should be planned to take place throughout
essential
the audit, in order to keep them continuously informed of audit progress.
Standard points at which contact might be made with the auditee, and matters
about which the auditee is to be informed, include:
CONTACT POINT Purpose
Auditors explain to management staff of the audited
entity the reasons for carrying out the audit; the
proposed questions, scope, criteria and methodologies;
Start-up the timetable; and work procedures. It may also be
useful to explicitly clarify what will not be audited so as
to help reduce misconceptions or false expectations by
the auditee.
Auditors explain to auditee staff the purpose of the
Prior to missions mission, information that may be required, meetings to
be arranged, and the timing of the mission.
Auditors discuss the audit procedures with auditee staff,
From start to end
keep them updated on progress and conduct a wrap-up
of mission
meeting to inform them of facts that have arisen.
Statement of Findings arising from the audit are documented and
Preliminary communicated to the auditee (see chapter 4).
Findings
Pre-adversarial & To ensure agreement of audit findings (see chapter 5).
adversarial
procedure

3.3.2 Drawing up the APM

The responsible Member should assure him/herself that a sufficient basis has
been developed for proposing that a realistic, realisable and added-value audit
be undertaken.
The audit questions and audit criteria should be communicated and, insofar
as possible, agreed, ideally between the Member responsible and the relevant
Commission Director-General, prior to the APM being presented to the Audit
Chamber.
The APM submitted to the Chamber is subject to EQCR, part of the ECA’s
framework for quality control (see section 2.5). Detailed procedures for how
EQCR operates are set out in the Vademecum of General Audit Procedures –
Audit Quality Management Framework - EQCR and in practical arrangements
26
issued by the Chambers .

26
Internal documents

Chapter 3: Planning the Audit - page 47

 3.4 QUICK SPECIAL REPORTS

Tackling narrowly defined audit It may be necessary at times to carry out audits that are not included in the
subjects
Annual Work Programme at short notice and at an accelerated pace. These
audits requiring immediate examination and reporting may arise due to
changing circumstances, for example, a request from the European Parliament
or from the Council. They are intended to tackle narrowly defined audit
subjects within a tight time-frame with prompt communication of the results to
auditees and stakeholders. The objectives and scope of the audit task should
therefore be limited in nature (e.g. to a single issue).
Exceptional products Given their specific origins “quick special reports” should be considered as an
exceptional product among the ECA’s range of special reports and should not
be regarded as falling within the ECA’s normal audit process. Due to their ad
hoc nature, the decision to produce a quick special report would most likely
require rescheduling other tasks and should be taken by an Audit Chamber or
the Court.
do not require an APM or detailed Due to their expeditious nature, no presentation or reporting on the results of
preliminary work
preliminary work or of an APM is required. Instead, an engagement letter
should be presented to the Audit Chamber by the Reporting Member setting
out in a clear and concise manner the need for the expeditious treatment, the
audit work to be performed, the resources and timeframes required, and the
anticipated impact of the audit. The engagement letter should also indicate
which administrative arrangements, if any, are different for this audit and
whether these arrangements have been discussed in advance with the
relevant Commission departments.
but notified to the auditee An overview of the planned scope and timing of the audit should be
communicated to the auditee and effective two-way communication ensured
27
during the audit .
No audit missions Missions to Member States or third countries are unlikely to be feasible under
a quick audit process. However, this would ultimately depend on the specific
circumstances of the audit.
As the carrying out of quick audits requires close monitoring, the use of
ASSYST and AMS tools is also mandatory for these audits. Similarly, EQCR is
applicable as with other audits but it may commence while audit work is
ongoing.

Quick reporting of results Audit findings are not formally communicated to the auditee. However, before
adopting the report, the auditee should be given the opportunity to comment
on the audit findings, conclusions and recommendations. Findings are formally
cleared through adversarial procedures or other channels and the Preliminary
Observations and Special report should be adopted by the Audit Chamber (or
the Court, if necessary).
Drafting of report In order to avoid any unnecessary delays, the reporting phase should be pre-
scheduled with the DQC Directorate. A drawing conclusions meeting should
be held between the audit team, Director and staff of the Reporting Member’s
Private Office (or Reporting Member), and possibly DQC to define the key
messages to be featured in the report.

27
ISSAI 100/43 Effective communication throughout the audit process.

Chapter 3: Planning the Audit - page 48

 Adoption Quick special reports may be adopted by a written procedure without recourse
to a formal meeting of the relevant chamber. The period after which the
28
Chamber’s decision becomes definitive (“emergency-brake procedures” ) is
also shortened to two days.
Publication Quick special reports may be published on the internet in the drafting language
as soon as adopted and when the auditee’s definitive replies have been
received. They are not usually published in hard copy.
It is estimated that the total required time for a quick report from approval of
the engagement letter to publication is between six and eight months.

28
The term “emergency brake procedure”refers to article 26(4) of the Court's Rules of Procedure.

Chapter 3: Planning the Audit - page 49

 ANNEX I: CONTENTS OF AN AUDIT PLANNING MEMORANDUM 29

Executive Summary

The one-page Executive Summary updates and expands upon the Audit
Proposal, prepared for the Annual Work Programme, with new information
or insights gleaned during the APM preparation stage. It summarises the
reasons for carrying out the audit, including the background of the audit, the
audit question, approach and scope, the expected impact, the resources
planned, and the reporting calendar.

What is the area we want to examine, and why?

Description of the audit area The reasons for selecting the audit subject are clearly stated. Relevant
background information is briefly presented on the audit subject (e.g. policy,
programme, DG), which may include the main activities, financial
information, laws and regulations, the objectives of the audit subject (which
may be set out in a Programme Logic Model), and the roles and
responsibilities of the major actors.
Materiality and risks to sound The monetary amounts involved are stated, and the main risks to sound
financial management
financial management identified at the audit planning stage.
Relevance Interest in the subject matter from the Parliament, Council, Commission, the
public, media or other interested parties is identified, as positive change is
more likely to result from the audit if stakeholders are engaged with the
topic.
Potential impact Potential impacts to be identified may include the influencing of future
policies and programmes, potential cost-saving opportunities, and
highlighting of good practice.

What are the audit questions and audit scope?

Audit questions The audit questions are defined as precisely as possible so as to provide
the focus for the audit, avoid unnecessary and expensive work, and allow
the audit team to conclude thereon.
The audit questions are identified and, if there is only one audit question,
translated into immediate sub-questions. Reasons for selecting the audit
questions, and for excluding other potential audit questions, are briefly
described.
Audit scope The scope statement defines and explains the parts of the
organisation/programme/policy that are the subject of the audit, and
identifies the time period and geographical areas to be covered.
Potential areas considered for inclusion in the audit scope, but rejected (e.g.
due to being too time-consuming, not offering sufficient focus) are also
noted.

29
The APM template is available on the DQC Intranet.

Chapter 3: Planning the Audit - Annex I - page 50

 How will we get the answers?

Audit approach The audit approach is clearly stated, i.e. the degree of emphasis to be
placed on auditing performance directly, with an initial focus on outputs and
outcomes, versus auditing the control systems, with an initial focus on
systems and controls.
Audit criteria The audit criteria, against which the actual situation is to be judged, are
clearly stated, indicating the relevant legislation or other sources from which
such criteria are sourced.
Audit methodology A short paragraph is devoted to describing how each data collection and
analysis method is to be used in the context of the audit. Detailed
information regarding the methodology may be set out in an annex to the
APM.
Likely outcome of the audit The likely outcome identifies areas in which findings may be identified,
conclusions drawn and recommendations made. It addresses the audit
questions, and should not be too detailed or give false hopes of far-reaching
effects of the audit.

How will the audit be resourced, supervised and monitored?

Resources, costs and timetable The audit team is identified by name, audit grade and time allocated, and
30
the budget, including consultant and mission costs, is given. The timetable
sets out the dates for starting and ending the audit, including the dates and
location of missions; completion dates for all key milestones (with realistic
timeframes being set for each, and taking account of holidays, training
courses, etc.); the date of the progress report; and the date of final report
publication.
It is obligatory to use the ECA’s electronic audit management system
(Audit Management System - AMS) in planning and reporting on the
sequence of tasks for each part of the audit process. This tool presents a
graphical representation of the duration of tasks against the progression of
time, and the relationships between different parts of the process. It also
provides a critical path analysis to calculate the minimum length of time in
which the audit can be completed, and to identify those activities that must
be prioritised in order to enable the audit to be completed on time.
Risks to delivery within time The major risks to delivering the audit report at the time required and within
and budget
the forecast resources and cost are identified, together with the likelihood of
each major identified risk occurring, the potential impact if the risk were to
materialise, and proposals for managing each risk.
Quality Control arrangements A progress report from the responsible Member to the Audit Chamber upon
completion of a key milestone is provided for in the APM. In addition,
reference is made to the reporting of the progress of the audit task through
the Audit Chamber's reporting system, and the documentation of the audit
findings and working papers in ASSYST II.

30
Requests for budget provisions for external experts should be made to DQC as soon as they are planned.

Chapter 3: Planning the Audit - Annex I - page 51

 Has the auditee been informed?

Reference to discussion Reference is made as to whether the audit objectives, questions, scope and
criteria have been discussed with auditee management in preparing the
APM and whether their reaction has been duly considered. In addition,
planned contacts with the auditee and external experts throughout the
course of the audit (as well as expected presentations of reports to
Parliament, Council and the media) should be included in the APM in the
form of a brief communication plan containing information on who will be
responsible for each communication; what the communication will be, and
when is it likely to occur.

Conclusion

We propose that the audit be undertaken. We hereby ask the Audit

Chamber for permission to proceed with the audit in the manner outlined
above, and with the resources and timeframes indicated.

Chapter 3: Planning the Audit - Annex I - page 52

 ANNEX II: EVIDENCE COLLECTION PLAN

Audit Level 2 Level 3 Level 4 Criteria Evidence Evidence Data collection Data analysis
Questions questions questions questions sources methods methods
WHAT WHAT WHERE ARE
HOW ARE WE WHAT WILL WE DO
STANDARD DO EVIDENCE WILL WE GOING TO
WHAT DO WE WANT TO KNOW? GOING TO GET THE WITH IT ONCE WE
WE MEASURE ANSWER THE GET THE
EVIDENCE? GET IT?
AGAINST? QUESTION? EVIDENCE?

- Legislation, - In person
regulations, (observation,
professional examine documents,
standards - Facts interviews, focus
(numerical - Quantitative
- Standards, groups)
- Answers can evidence; - The entity, other evidence (e.g. trends,
measures or - By post, telephone,
be yes, no, descriptive public entities, comparisons, ratios)
results e-mail (request
yes but, or no evidence, published - Qualitative evidence
commitments of documents,
but. qualitative research, (coding, matrices)
auditee questionnaires)
information) beneficiaries,
- Answerable - Performance of - Systems analysis
suppliers, interest - Sample surveys
- Logical comparable - Experiences / groups (e.g. flowcharts)
(which could be either
organisations, Perceptions / - Case studies
in person or by post,
best practice, or Opinions
e-mail)
standards
developed by - Benchmark against
auditor comparable entities

Chapter 3: Planning the Audit - Annex II - page 53

 ANNEX III: OUTLINE AUDIT PROGRAMME

Audit task:

Prepared by: Reviewed by: Approved by:

Date: Date: Date:
Audit questions

Audit procedures Remarks WP Reference

Chapter 3: Planning the Audit - Annex III - page 54

 PERFORMANCE AUDIT MANUAL

CHAPTER 4
EXAMINATION PHASE

Chapters in the
TABLE OF CONTENTS
Performance Audit Manual

4.1 Introduction
General introduction 4.2 Delivering sufficient, relevant & reliable audit evidence
4.2.1 Purpose of and approach to the examination phase
4.2.2 The nature of evidence in performance audits
Chapter 1: Framework for 4.2.3 Sufficiency, relevance and reliability of evidence
Performance Audits by the 4.2.4 Sources of evidence
European Court of Auditors
4.2.5 Types of evidence
4.3 Collecting & analysing data
4.3.1 Purpose of and approach to data collection and analysis
Chapter 2: The Performance
4.3.2 Data collection process
Audit Approach and the
3 ‘E’s 4.3.3 Using the work of others
4.3.4 Data collection and ethics
4.3.5 Data analysis
4.4 Deriving valid audit findings
Chapter 3: Planning the
audit 4.4.1 Need for a sound basis
4.4.2 Drafting audit findings
4.5 Communicating audit findings
4.6 Documenting the audit
Chapter 4: Examination
Phase 4.6.1 Purpose and approach
4.6.2 Audit documentation referencing
4.7 Audit management and quality control arrangements
4.7.1 Audit management, including supervision and review
Chapter 5: Reporting Phase
Annex I: Data collection and analysis methods

For any further information, please contact:

European Court of Auditors
Who to contact
Directorate of Audit Quality Control Committee (DQC)
E-mail: ECA-DQC-CONTACT or [email protected]

Chapter 4: Examination Phase - page 55

 4.1 INTRODUCTION

The audit examination phase commences at the start of audit work, following
approval of the APM, and continues until the drafting of the final report
From APM approval to clearing
findings,
commences. It thus includes carrying out audit procedures to collect and
analyse data, evaluating facts against pre-determined criteria, and drafting and
clarifying audit findings, a process which is graphically represented below.
Audit examination work takes place on the basis of the audit planning already
undertaken and the planning documents thereby developed (APM, Evidence
Collection Plan and outline Audit Programme). The plan should be followed
insofar as possible, in terms of the work to be performed, resources,
timeframes and quality. However, some parts of the audit may need to be
reconsidered during the examination stage if the auditor encounters difficulties
in gathering evidence. In general, the organisation of the audit should also
31
satisfy the requirements of good project management .
With objectivity and judgement It is critical that auditors consider from different perspectives the activity being
essential.
audited and keep an unbiased attitude to information presented while being
open-minded to different views and arguments.
The exercise of sound professional judgement is particularly required in
assessing whether the quantity and quality of evidence will enable sound
conclusions to be drawn regarding the audit questions, and in determining the
significance of audit findings.

31
Audit Standards ISSAI 300, para. 37, ISSAI 3000, para 96-97 and ISSAI 3200 good project management and submitting the plan to
supervisors and SAI management, para 56-60.

Chapter 4: Examination Phase - page 56

Chapter 4: Examination Phase - page 57
 4.2 DELIVERING SUFFICIENT, RELEVANT & RELIABLE AUDIT EVIDENCE

4.2.1 Purpose of and approach to the examination phase

A search for evidence The purpose of the examination stage is to gather sufficient, relevant and
reliable audit evidence to allow the auditor to conclude on the audit questions
and to support all the statements made in the audit report.
During the audit examination phase, audit procedures are carried out to collect
and analyse data; the resultant evidence or fact ('what is') is evaluated against
the pre-determined audit criteria ('what should be') in order to derive audit
findings; and the causes and effects of these findings are determined. Audit
to arrive at an audit finding findings therefore consist of the evidence compared to the standard
(expressed in the form of an audit question or criterion) and an analysis
thereof. The findings are communicated to the auditee, whose written
response is required, whether indicating agreement or reasons for
disagreement. .
The conduct of audit work comprises both an analytical and a communicative
aspect. The analytical process concerns the collection, analysis and evaluation
of data whereas the communicative process, initiated at the time when the
audit is first presented to the auditee, continues as and when different findings,
arguments and perspectives arising during the course of the audit are
assessed.

4.2.2 The nature of evidence in performance audits

Data, information and audit evidence are interrelated, as follows:

CHART: From Data to Audit Evidence

DATA
if evidence
insufficient, compiled
& analysed
collect more data

AUDIT EVIDENCE INFORMATION

used to support
a point

The nature of the audit evidence required is exclusively dictated by the subject
matter and the audit questions, which tend to vary significantly in performance
audits. In addition, such audits are more judgement-based, with the result that
Nature of evidence varies and
tends to be persuasive
audit evidence tends to be more persuasive ("points towards the conclusion
that...") than conclusive ("right/wrong") in nature. The combination of these
factors requires auditors to be creative and flexible in their search for the right
type of evidence.

Chapter 4: Examination Phase - page 58

 Detailed assessments of information needs should be carried out at both the
audit planning and examination phases so that the auditors are not swamped
by excessive data. As well as facilitating the elimination of incidental details
and irrelevant approaches, this will also contribute to the sorting and
structuring of data collected. It might be useful to hold discussions in advance
with experts regarding the nature of the data to be obtained and the way in
which it will be analysed and interpreted by the auditor, in order to reduce the
risk of misunderstanding and potentially speed up the process.

4.2.3 Sufficiency, relevance and reliability of evidence

Sufficient, relevant and reliable evidence should be obtained in order to

conclude against the audit questions and to support the audit findings and
conclusions, thereby ensuring that the contents of the audit report stand up to
critical review. The concepts of quantity (sufficiency) and quality (relevance
and reliability) in relation to audit evidence must be considered together, as an
Evidence must support the audit
report's contents
inverse relationship exists between them. Thus, high-quality evidence can lead
to a reduction in the need for a large quantity of evidence; a large quantity of
evidence can sometimes, but not always, be persuasive, even though
individual pieces of evidence are not of high quality. The exercise of
professional judgement is essential here as there are no precise guidelines to
measure the degree of proof required.
Audit evidence is sufficient if there is enough of it to persuade a reasonable
person that the audit findings and conclusions are valid, and that the
and be of the required quantity and recommendations are appropriate. Audit evidence is relevant if it is clearly and
quality logically related to the audit questions, audit criteria and audit findings. Audit
evidence is reliable if the same findings arise when tests are carried out
repeatedly or when information is obtained from different sources.
Whilst the following concepts are often useful in assessing the quality of audit
evidence, certain constraints may exist in their practical application:
• original evidence is superior to photocopies, facsimiles, etc.;
• documentary evidence is preferable to oral evidence. However,
correspondence, memos and reports may be incomplete, ambiguous or
even incorrect, whilst interviews can provide an in-depth understanding not
only of facts, but also of constraints and the environment. Nevertheless,
evidence collected from interviews needs to be corroborated from other
Practical constraints on evidence
sources;
• third-party evidence is better than that generated within the auditee
organisation. However, in performance audits, limited use may be made of
third-party confirmations, as information may only be available within the
entity being audited. Furthermore, strong internal controls within the
auditee organisation can improve the quality of information obtained;
• evidence generated through the auditor's direct observation, inspection and
computation is superior to evidence obtained indirectly.

Chapter 4: Examination Phase - page 59

 In assessing the quantity and quality of audit evidence, the auditor needs to
consider the following:

a higher standard is required for

Criteria to help determine if
the purpose evidence supporting audit findings
evidence is sufficient,
for which the evidence will be used than for background information
relevant and reliable
provided in the audit report
in general, the higher the level of
the level of materiality
materiality or significance, the higher
in monetary terms or the significance
the standard of evidence that is
of the audit finding
required
greater reliance can be placed on
the degree of independence
evidence which emanates from
of the source of the evidence
independent sources
the cost at some point, the cost of obtaining
of obtaining additional evidence more evidence will outweigh the
relative to likely benefits in terms of improved persuasiveness of the total
supporting findings and conclusions body of evidence
the greater the risk of legal action,
the risk
controversy or surprise from reporting
involved in making incorrect findings
an audit finding, the higher the
or reaching invalid conclusions
standard of evidence needed
the care taken including the extent of the auditors'
in collecting and analysing the data skills in these areas

In performance audits, important facts are often not of an individual nature, but
rather comprise several interrelated facts. In assessing the quantity and quality
of evidence, the auditor must take into account that the strength of the
combined facts may be as important, or even more so, than the strength of the
individual facts.
Furthermore, the auditor must satisfy him/herself that the quantity and quality
of evidence minimises the risk of arriving at invalid or inappropriate findings,
conclusions or recommendations. If the evidence-collection process does not
produce sufficient, relevant and reliable evidence, then audit findings and
conclusions must not be drawn.

Chapter 4: Examination Phase - page 60

4.2.4 Sources of evidence

Different sources should be employed in collecting evidence, in order to

Multiple sources provide better corroborate such evidence, thereby making the related audit findings more
evidence reliable, and to ensure that different perspectives are taken into account. There
are three broad sources of information for performance audits:
Sources

through interviews, questionnaire surveys, focus groups,

direct inspection and observation.
generated directly The auditors can determine the methods that will provide
by the auditors the best quality of evidence for the particular audit.
However, their skills in designing and applying the
methods will determine the quality of the evidence.
such as information from databases, documents, activity
statements and files (e.g. reports of the Internal Audit
Capabilities, impact assessments and ex-post
evaluations).
provided by the
auditee Auditors must determine the reliability of data that is
significant to the audit questions by review and
corroboration, and by testing the auditee's internal
controls over information, including general and
application controls over computer-processed data.
which may have been verified by others or whose quality
is well known, e.g. national statistical data.
provided by third The degree to which such information can be used as
parties audit evidence depends on the extent to which its quality
can be established, and its significance in relation to the
audit findings.

Chapter 4: Examination Phase - page 61

4.2.5 Types of evidence

Audit evidence derived from the above sources can be of four types - physical,
documentary, oral or analytical - which can be obtained and documented as
follows:

AUDIT
PROCEDURES
CONSIDERATIONS DOCUMENTATION
TO OBTAIN
EVIDENCE

Whilst usually the

most persuasive
evidence, the
auditor must be Direct inspection Notes, photographs,
PHYSICAL aware that his/her or observation of charts, maps,
presence may people, property drawings, samples, or
distort what would or events audiovisual material.
normally occur, thus
reducing the quality
of the evidence.
This evidence may
be in electronic or
Review of Performance
hard-copy format.
documents, management reports,
However, useful
reports, manuals, policies &
DOCUMENTARY information may not
literature, the procedures, system
always be
internet, postal or descriptions, letters,
documented, thus
web-based contracts, survey
also necessitating
surveys. results.
the use of other
approaches.
Oral evidence is
generally important
in performance
audits, as
information
obtained in this
Enquiry or
manner is up-to-
interviews of Summary of
date and may not
ORAL auditee staff or information obtained
be available
third parties focus through these
elsewhere.
groups, expert methods.
However,
panels.
information should
be corroborated and
statements
confirmed if they
are being used as
evidence.
Such evidence is Summary of
obtained by using Analysis through analytical data,
professional reasoning, including ratio
ANALYTICAL judgement to reclassification, analysis, regression
evaluate physical, computation and analysis,
documentary and comparison. benchmarking and
oral evidence. coding.

By collecting evidence through a number of these methods, the quality of audit

Combine different types of evidence is strengthened considerably. A broad range of such methods
evidence for more persuasive
argument should be used, insofar as is consistent with the audit questions and the
subject matter, whilst bearing in mind cost and time considerations.

Chapter 4: Examination Phase - page 62

 4.3 COLLECTING & ANALYSING DATA

32
4.3.1 Purpose of and approach to data collection and analysis

Although data analysis follows data collection in chronological terms in this

process, auditors need to know what analytical techniques they will use before
designing their strategy for data collection, e.g. when using surveys.
Otherwise, they may find that the data collected cannot be analysed. Analytical
techniques to be employed may be quantitative (e.g. trend analysis, regression
analysis) or qualitative (e.g. analysis and interpretation of interviews or
documents).
Both quantitative and qualitative data may be collected for different purposes,
whether as part of the learning process to understand the audit subject, to
Understand, assess, document.
assess and measure performance, or to document errors or problems already
known (e.g. from the planning phase).
The character of data collection usually changes as the audit proceeds.
Initially, e.g. during the planning phase, the auditor is interested in more
general information; as the audit proceeds, the data needed will be more
specific.
Prior to data collection, a general assessment of the internal control system,
Assess internal control system. including IT, should be performed in order to identify risks that could
compromise the integrity of the data.

4.3.2 Data collection process

Evidence collection follows an iterative decision-making process, whereby

Iterative process, auditors obtain data, examine it for completeness and appropriateness,
analyse it, and make decisions on whether additional evidence is required.
Data collection methods (see Annex I) range along a continuum from, at one
extreme, those giving an overall picture of a situation or population (e.g.
surveys) to, at the other extreme, the in-depth exploration of a small number of
with methods determined by audit
subject & questions,
items (e.g. case studies), with other methods such as interviews, documentary
reviews and focus groups in between. The particular methods to be used in
any given performance audit will depend on the audit subject, the audit
questions being addressed, and the resources and time available.
Auditors are encouraged to use computer-assisted audit techniques (CAATs)
using CAATs when possible. for collecting and analysing audit evidence whenever their use will increase the
efficiency of the audit.

32
See DQC Intranet for guidelines on individual data collection and analysis methods.

Chapter 4: Examination Phase - page 63

4.3.3 Using the work of others

Use only if relevant, Performance auditors may rely on the work of others whenever possible, when
relevant to the audit questions. The ECA’s auditors may use the data and
findings generated by the Commission’s Internal Audit Service (IAS) and by
the Commission’s evaluation reports.
When the ECA contracts work out to other parties (either auditors or experts), it
should communicate its ethical policies and quality control procedures to such
parties and seek confirmation that they have an effective quality control
system. It should also check that they have the necessary competencies
required to perform the work, and that they are subject to appropriate
confidentiality arrangements.
When the work of internal audit or evaluators is used to support particular audit
findings, the work on which the auditors intend to rely should be assessed and
corroborated, to determine if it meets the standards for sufficient, relevant and
reliable audit evidence. This may be done by assessing the reputation,
qualifications and independence of those performing such work, as well as by
evaluated and corroborated.
reviewing their reports and working papers. The nature and extent of the
review depends on the significance of the work in relation to the audit
33
questions and the extent to which the auditors will rely on it . When such
matters are included in the audit report, the source of findings should be
indicated.
In addition, external experts may be engaged to perform technical work which
is outside the auditor's area of expertise or which the experts can perform more
economically. The appropriate procedures in engaging such experts should be
followed, such as: assessing their independence, objectivity and professional
competence prior to their engagement; ensuring the terms of reference and
scope of work are appropriate; and evaluating and corroborating the specific
work on which the auditors intend to rely as audit evidence. An ongoing
dialogue with the expert during the course of his/her work makes it easier for
the auditor to continually keep up to date with any issues arising.

4.3.4 Data collection and ethics

In the course of audit work, the auditor may obtain or come across sensitive
information. Such information should be treated in a confidential manner, and
data protection regulations observed.

33
See Guidelines on Evaluation (DQC Intranet) for further information.

Chapter 4: Examination Phase - page 64

 Auditors should discuss among themselves and be alert for situations, control
weaknesses, errors and unusual transactions or results that could indicate
illegal acts or abuse, such as fraud, impropriety, corruption or irregularities.
When performing risk-assessment procedures and related activities, they
should determine how and where fraud might occur and the extent to which
34
such acts affect the audit result . In the occurrence of such an event, the
standard Court procedures in respect of irregularities and fraud should be
followed.
The auditor should have a questioning mind and maintain professional
scepticism. Notwithstanding the auditor’s past experience of the honesty and
integrity of the entity’s management, the auditor should recognise that a
situation of irregularity or fraud could exist. When planning the audit, some time
and resources should be provided for unpredictable events.

4.3.5 Data analysis

Data requires analysis to explain what has been observed, and to make the
connection between cause and effect. Auditors need to be aware that
collecting data serves no useful purpose if it cannot be properly analysed.
Allow time for analysis,
Attention must therefore be given to setting aside the time and resources
necessary to carry out analysis and to assess the results. Computer assisted
audit techniques (CAATs) are often an essential part of such analysis.
using many techniques available. The term ‘data analysis’ is generally used to include both the compilation
(coding and tabulation) and analysis of data. Data analysis, either quantitative
or qualitative, involves considering the results from different perspectives or
together with other data. Quantitative analysis may employ simple techniques
(e.g. frequency counts) or more sophisticated techniques (e.g. trend analysis,
regression analysis or variance analysis) - see Annex II. Qualitative analysis
may be used to analyse and interpret interviews or documents, or to identify
descriptive material that may be used in the audit report.
The final stage in data analysis involves combining the results from different
types of sources, e.g. combining results from surveys with those from case
studies, etc. There is no general method for doing this, but it usually involves
weighing up arguments and consulting experts where necessary.

4.4 DERIVING VALID AUDIT FINDINGS

4.4.1 Need for a sound basis

The auditor uses the information gathered to make an objective assessment of

actual performance against the audit criteria. Where performance does not
Evaluate evidence against criteria,
meet the criteria, further investigation is needed to gain assurance that any
resultant audit findings and conclusions are significant, fair and well-founded.

34
Auditing standard ISSAI 1240 (ISA 240, paragraphs 15 – 24).

Chapter 4: Examination Phase - page 65

 Gathering additional evidence or discussing the matter with auditee
management may be necessary so as to: determine whether the deficiency is
an isolated instance or represents a systematic problem; identify the cause of
the deficiency; determine whether the problem can be addressed by the
auditee or is outside its control; and assess its potential effect. In many cases,
the effect of a finding may be quantifiable, e.g. expensive inputs or processes,
and understand deficiencies.
unproductive facilities, time delays, etc. However, qualitative effects, as
evidenced in a lack of control, poor decisions or a lack of concern for service,
will also be significant and need to be considered.
Active dialogue should be maintained with the auditee and potential audit
findings discussed as they arise. Constructive discussion of initial findings with
the auditee helps to establish the quantity and quality of evidence.

4.4.2 Drafting audit findings

Audit findings should be set out in a clear and logical framework so as to allow
for an easy understanding of audit criteria applied, facts established by the
evidence, and the analysis by the auditor of the nature, significance, and
Clear logical framework
causes of the problem or the better-than-expected performance. The impact in
terms of economy, efficiency and/or effectiveness must also be considered, as
this provides the basis to demonstrate the need for corrective action.
In stating the audit finding, the auditor must assess the degree of confidence in
the audit finding, based upon the strength of the evidence. The assessment
must be clearly reflected in the wording of the finding, with qualifying words
(e.g. generally, frequently) used.
Performance audits should focus on providing a balanced view of the topic,
presenting not only deficiencies but also, when appropriate, positive findings
and indications of good practice. The overall emphasis is to formulate audit
findings in a constructive and balanced way.
Furthermore, the auditor will need to determine auditee management's
with constructive, balanced awareness of the issue; if management is aware of the problem and already
findings. taking corrective action, this needs to be recorded and taken into consideration
for reporting purposes.

4.5 COMMUNICATING AUDIT FINDINGS

As mentioned in section 2.4.8, the communication process between the auditor

and auditee begins at the planning stage of the audit and continues throughout
the audit process, by a constructive process of interaction, as and when
different findings, arguments and perspectives are assessed. Furthermore, as
stated in the previous section, the auditor needs to make the auditee aware of
the preliminary audit findings to determine and verify whether corrective action
has already been taken and, if so, take this into consideration for reporting
purposes. The communication of preliminary findings (standards, facts, and
analysis) also provide confirmation that the facts and findings are accurate.
Such confirmation (e.g. by the Member State authorities) may well provide
additional audit evidence or information that was not available or understood
during the audit visit. Preliminary audit findings need to be assessed in light of
the auditee's responses. The results of this communication and confirmation
should be documented as part of the audit process.

Chapter 4: Examination Phase - page 66

 In principle, there should remain no disagreement on the factual content
following this process. Furthermore, issues which may lead to controversial
discussion during the subsequent adversarial procedure with the Commission
must be duly identified and carefully analysed. Differences of view expressed
regarding the audit criteria applied or the analysis carried out have to be
considered carefully, and final audit findings must always state the reasons
why the auditee's arguments have not been accepted. Only those findings
which have been communicated to the auditee should be included in the final
report.

4.6 DOCUMENTING THE AUDIT

4.6.1 Purpose and approach

A guiding principle in documenting audit evidence is that audit documentation

should enable an experienced auditor, who has had no previous connection
Provide understanding of evidence, with the audit, to establish and understand the evidence that supports the
auditors' significant judgements and conclusions. All audit work should be
documented in ASSYST.
Proper documentation of evidence is vital and should be completed before the
audit findings are cleared, thus helping to ensure that the audit findings are
evidence-based. Audit files and working papers need to contain information
about the approach and work undertaken to answer the audit questions, and to
be structured logically so as to provide ready access to the audit evidence.
Irrelevant or unnecessary documentation should not be included in audit files.
Good documentation of audit evidence helps to ensure that:
• a defensible basis exists for the audit findings and draft final report
contents (which is particularly important during the adversarial procedure);
is the basis for report contents,
• the audit findings, conclusions and recommendations can be explained;
• an appropriate basis exists for quality control both during the audit and for
subsequent reviews (Quality Assurance, peer reviews).
The auditor must prepare minutes for all meetings involving auditee staff that
the auditor intends to rely on for evidence purposes. It is a matter of judgement
as to whether a particular meeting will require minutes to be signed off by the
auditee in order to improve its quality as evidence, as well as the extent of
and includes minutes where
relevant
detail to be recorded. In many instances, a single note summarising the key
points of several meetings is sufficient. However, it is recommended that the
auditor, at the beginning of the audit, inform the auditee that the record of
certain meetings might be used as audit evidence and that the approval of
minutes by the auditee will be necessary in such cases.

4.6.2 Audit documentation referencing

As most performance audits produce a mass of evidence on paper, it is

important to have a documentation referencing system in place that links the
work done to the resultant findings. A simple trail of evidence should exist.
Links work to findings The key item of this trail is a cross-referencing of audit findings to the evidence,
prior to their submission to the reviewer/approver. This could be supplemented,
if desired, by a short summary explaining how the audit methodology was
employed, the nature and extent of evidence collected, and the analyses to

Chapter 4: Examination Phase - page 67

 which it was subjected. This summary could be in matrix form, grouped around
each of the main findings.
Key documents should be recorded and cross-referenced, including major
decisions influencing the audit work and its management; key correspondence
and other contact with the auditee; the main items of evidence, their sources
and the analysis undertaken; and evidence of supervisory reviews.

4.7 AUDIT MANAGEMENT AND QUALITY CONTROL ARRANGEMENTS

4.7.1 Audit management, including supervision and review

Audit management and the Head of Task should ensure that:

• audit work is appropriately supervised;
• audit work is carried out to the required quality and properly documented;
• progress is monitored against the timetable and resources allocated; and
• timely corrective action is taken where necessary.
It is also crucial that they be aware of changed circumstances (e.g. new
legislation, major reform proposed) in order to critically re-assess the relevance
of the audit, the validity of the audit approach and, in certain circumstances,
the merits of pursuing the audit.
Supervision involves overseeing the work of staff assigned to the audit, in
order to ensure that the audit questions are answered. Elements of supervision
include providing staff members with sufficient guidance, staying informed
Staff to be properly supervised. about significant problems encountered, and reviewing the work performed.
With experienced staff, supervisors may outline the scope of work and leave
details to the staff. With less experienced staff, supervisors will specify audit
procedures to be performed, and techniques for collecting and analysing data.
Monitoring of audit progress is necessary on an ongoing basis to ensure as far
as possible that audit work is completed within the scheduled time and with the
Monitoring at various levels. resources allocated. The Head of Task and responsible Member perform such
monitoring at a detailed level using the Audit Management System tool (AMS).
The Audit Chamber also has a role to play in monitoring performance.
Taking corrective action. Corrective action is taken when monitoring reveals that the audit work is not of
the required quality, or risks not being completed within the timetable set and
with the resources allocated.

Chapter 4: Examination Phase - page 68

 ANNEX I: DATA COLLECTION AND ANALYSIS METHODS

Data collection methods Data analysis methods

Interviews Ratio analysis

Case studies Frequency counts

Surveys Regression analysis

Documentary review Comparative analysis

Focus groups Coding and abstraction

Benchmarking Variance analysis

Inspection Trend analysis

Observation

Enquiry

Chapter 4: Examination phase - Annex I - page 69

 PERFORMANCE AUDIT MANUAL

CHAPTER 5
REPORTING PHASE

Chapters in the
TABLE OF CONTENTS
Performance Audit Manual

5.1 Introduction
5.2 Report quality
 Objective
 Complete
 Clear
 Convincing
General introduction  Relevant
 Accurate
 Constructive
 Concise
Chapter 1: Framework for 5.3 Planning the report
Performance Audits by the 5.3.1 Consider the addressees
European Court of Auditors 5.3.2 Approach to planning the report
5.3.3 Prepare a drafting plan
5.4 Drafting the report
5.4.1 Approach to writing the report
Chapter 2: The Performance 5.4.2 Structure and layout
Audit Approach and the  Executive Summary
3 ‘E’s  Introduction
 Audit scope, approach and methodology
 Observations
 Conclusions and recommendations
Chapter 3: Planning the 5.4.3 Logic and reasoning
audit  A focus on the audit questions
 Use of examples
 Naming of third parties in the ECA's reports
5.4.4 Types of information and data to be included
Chapter 4: Examination
5.4.5 Style
Phase
5.4.6 Use of non-textual information
 Tables, charts and graphics
 Maps and pictures
Chapter 5: Reporting Phase  Numbers and percentages
5.5 Reviewing the report
5.6 Clearing the report
5.7 Distributing the report
5.8 Report follow-up
5.8.1 Follow-up of report recommendations and findings
5.8.2 Follow-up of how the audit report has been perceived

For any further information, please contact:

European Court of Auditors
Who to contact
Directorate of the Audit Quality Control Committee (DQC)
E-mail: ECA-DQC-CONTACT or [email protected]

Chapter 5: Reporting Phase - page 70

 5.1 INTRODUCTION

The reporting phase of a performance audit begins with drafting of the

preliminary observations and ends, typically, with the publication of a “Special
35 36
Report” or, on occasion, the issuance of a President's Letter . It thus
includes drafting, approval of the preliminary observations by the Audit
Chamber, the adversarial procedure with the Commission, adoption of the final
37
report by the Chamber, translation, presentation to the discharge authority
and publication. A graphical representation of the reporting phase is included
below.
Effective communication The purpose of a special report is to report the results of the audit to the
discharge authority, the auditee and the outside world. The key to a good
report is effective communication, with the report clearly and objectively setting
out the main findings and conclusions on the audit questions, allowing the
reader to understand what was done, why and how, and providing practical
recommendations. Note that the audit questions presented in the special report
should be those questions which give rise to the report’s conclusions. These
questions need not be exactly the same as the original audit questions as set
38
out in the APM .
which effects change Publication helps to ensure that the ECA's work results in real change and
provides transparency in terms of the management of EU funds. The
reputation and credibility of the ECA in the area of performance audit is earned
largely through the publication of clear, useful and timely special reports, which
contribute to improving the economy, efficiency and effectiveness of EU
spending. As the ECA's reports are read within the European institutions and
by the outside world through the press, they are the basis on which the
legislative and budgetary authorities, as well as the general public, judge the
ECA's output.
based on a good audit. The audit report is the end-product of the entire audit process. A properly
conceived and implemented audit provides the basis for a good report, whilst
conversely a poor audit is unlikely to result in a good report.

35
A special report is the reporting vehicle generally used for communicating the results of performance audits.
36
A President's Letter comprises the report, with a covering letter from the President of the Court, which is sent to the auditee. It is not
necessarily translated or published. The decision to publish a President's Letter is taken on a case-by-case basis by the Court.
37
The European Parliament, through its budgetary control committee - the CONT. Special reports are also presented to the Budget
Committee, and may be presented to other specialised committees of the Council and European Parliament.
38
The audit work is carried out with the aim of answering the APM audit questions. However, it is often the case that those questions are not
the most suitable basis for presenting the Court’s conclusions in a special report. In these circumstances, for the purposes of reporting in
special reports, audit teams are not required to adhere strictly to the APM questions.

Chapter 5: Reporting Phase - page 71

Chapter 5: Reporting Phase - page 72
 5.2 REPORT QUALITY

Based on the INTOSAI Auditing Standards and guidelines for performance

auditing,(as well as the report writing audit guidelines), the ECA's reports
should be objective, complete, clear, convincing, relevant, accurate,
constructive, and concise. An effective quality control system is required to
help ensure that the reports exhibit these qualities, as set out below:

 Objective
Audit reports need to be written from an independent unbiased viewpoint, with
actual performance judged against objective (and preferably agreed) criteria.
Balanced, neutral, fair
The report should be balanced in content and neutral in tone, be fair and not
misleading, with the audit results put into context.
Objective reports give due recognition to positive aspects of performance, and
are representative of what was actually found, rather than over-emphasising or
exaggerating deficient performance. Interpretations need to be based on
insight and understanding of the facts and conditions. This can help ensure
improved acceptance of the report by the auditee.

 Complete
This requires that the report contain all information and arguments needed to
answer the stated audit questions and to promote an adequate and correct
understanding of matters and conditions reported. The relationship between
All relevant information
the audit questions, criteria, observations and conclusions should follow a
logic which aids understanding, with a clear link between the findings,
conclusions and recommendations.

 Clear
Clarity requires that the report be easy to read and understand; employing
straightforward and non-technical language as far as possible; explaining
Clarity of message acronyms and any technical language deemed necessary, and avoiding
ambiguity. The main messages should be clear, relevant and easily
identifiable ('clarity of message'), and not be susceptible to misunderstanding.
The logical organisation of material, and accuracy in stating facts and drawing
conclusions, are essential to clarity and understanding. Effective use of titles
and headings makes the report easier to read and understand. In order to
increase the likelihood that the ECA’s reports are easily identified by internet
Easy to understand search engines, thus increasing their visibility and impact, all reports should
include, wherever possible, the words “Europe” or European” in their titles.
Visual aids (such as pictures, graphs, charts and maps) can be used to
illustrate and summarise complex material. Well-selected examples also help
to clarify the text.

Chapter 5: Reporting Phase - page 73

  Convincing
It is imperative that the audit results conclude against the stated audit
questions; the observations are presented persuasively and are supported by
sufficient information and explanations for the reader to understand the extent
Convince reader of content
and significance of these observations; and the conclusions and
recommendations follow logically from the facts and arguments presented. The
information given should convince the reader of the validity of the findings, the
reasonableness of the conclusions, and the benefit of implementing the
recommendations.
To develop a logical, convincing argument, one approach is to use inductive
logic. This connects the findings to the recommendations, by asking why the
recommendation is being made, as follows:
Recommendation Related findings
- Noise enters office when windows are
closed
We need reinforced
- Draughts enter office when windows are
and properly sealed WHY?
closed
glass windows
- Fumes enter office when windows are
closed

 Relevant
The report's contents must pertain to the stated audit questions, be of
importance and interest to the report's users, and add value, e.g. by saying
something new about the topic. An important aspect of relevance is timeliness;
Timely and value-added. to be of maximum use and to help contribute to change, the audit report
should provide relevant up-to-date information in time to respond to users'
needs. Auditors must plan for the timely issuance of the report and conduct the
audit with this in mind.

 Accurate
The evidence presented should be true and all findings correctly portrayed.
This is based on the need to assure readers that what is reported is credible
Evidence must be accurate.
and reliable, as one inaccuracy in a report can cast doubt on the validity and
credibility of the entire report and divert attention from its substance. In
addition, inaccuracies can damage the ECA's credibility and reduce the impact
of its reports.

 Constructive
The report should assist management in overcoming or avoiding problems in
the future, by clearly identifying who is responsible for the weaknesses
identified and making practical recommendations for improvement. It is not
Assist and encourage. appropriate to criticise management for issues that are beyond their control.
Balanced reports, which give due recognition to positive aspects of
performance, can help ensure improved acceptance of the report by the
auditee.

Chapter 5: Reporting Phase - page 74

  Concise
The report should be no longer than is necessary to convey and support the
message. Extraneous details or immaterial findings may detract from a report,
No longer than necessary. conceal the real message, and confuse or distract the users. Special reports
that are as concise as the subject matter allows are likely to achieve greater
impact.

5.3 PLANNING THE REPORT

5.3.1 Consider the addressees

The ECA's reports have a wide range of addressees, such as the discharge
39
authority, the auditee and the general public.
Discharge authority. The discharge authority - the European Parliament acting on the
recommendation of the Council - is a critical target group, which uses the
ECA's reports in a direct and practical way to assess financial management,
support its discharge decisions and make observations and requests.
Auditees. Auditees are managers of the budget, and experts in the area. As detailed
findings will have been communicated to the auditee during the audit process,
the report can focus on communicating the overall findings and main
messages.
European citizens. The citizens of the Union are reached in large part through coverage of the
ECA’s reports by the media. This target group is rarely expert in the audit
subject.
In order to meet the addressees' requirements, reports should be drafted for
the attention of an interested but non-expert reader who is not necessarily
familiar with the detailed EU or audit context. This avoids the need for
sensationalism to attract attention and overly detailed explanations of basic
facts, but requires the text to be presented in an interesting way and the
context and impact of findings to be clearly described.

5.3.2 Approach to planning the report

Planning for the report should start at the time of overall audit planning. The
audit questions must be set in a manner that will facilitate a relevant and
interesting report. At the planning stage, the auditor will generally already have
in mind an idea of the report structure and content.
In performance auditing, it is a good discipline to put together, at an early stage
of the audit, a report outline based around the audit questions, which identifies
Sketch report outline early in audit the main findings and provisional conclusions. This report outline, typically
drafted by the Head of Task, needs to be reviewed periodically throughout the
audit.

39
Generally a European institution, agency or body, rather than individual beneficiaries or Member States.

Chapter 5: Reporting Phase - page 75

 As part of the Audit Chamber's monitoring of audits, a progress report, as
envisaged in the APM, should be prepared, generally by the Head of Task,
and presented to the Audit Chamber. This progress report should include,
and present in progress report. when possible, an indication of the main observations and conclusions already
evident from the audit and a draft outline of the final report. This may involve
critical examination of the audit questions concluded upon, the work done,
findings, conclusions and how to communicate the main messages.

5.3.3 Prepare a drafting plan

Audit teams should carry out a “drawing conclusions” exercise between the
40
completion of the audit work and the drafting of the report .
On the basis of the drawing conclusions session, the report outline should be
When audit completed, developed into a more detailed drafting plan. The drafting plan is generally
develop drafting plan
established by the Head of Task, based around the main audit findings and
conclusions. The plan sets out the report's structure, tone and key messages,
concentrating on a limited number of material items. It needs to be brief and
specific.
The drafting plan is based on the audit work undertaken with regard to the
audit questions answered, the evidence obtained, the key conclusions and the
need to present material observations in the most useful and relevant way to
that focuses on key messages the non-expert reader. The report planning process thus helps to identify and
eliminate unsupported conclusions. Key messages must be clearly apparent,
useful, and supported by evidence. Consideration also needs to be given at
this stage to identifying practical, useful recommendations.
The Head of Task and Member should review the drafting plan and approve it,
checking if the observations and conclusions are material, and if the evidence
and is properly approved. supporting the observations, conclusions and recommendations is sufficient,
relevant and reliable. Detailed drafting should begin only following approval of
the drafting plan.

5.4 DRAFTING THE REPORT

5.4.1 Approach to writing the report

An audit report is not a record of all the audit findings; that is the purpose of the
audit files, both electronic and hard-copy. The report must set out the material
Focus on material,
relevant issues,
and relevant observations and conclusions, with a clear link between the two.
This will help with writing the report in a clear way, focused on the main
messages and articulated around the audit questions.
The full report should follow the structure of the drafting plan, although
experience with the actual drafting may require the plan to be changed. It
based on drafting plan,
needs to be remembered that writing is an iterative process, which means that
the draft must be reviewed and changes and improvements made.

40
CH 324/11 of 30 September 2011 on Audit Quality management. Detailed guidance on the process can be found in the Audit Guideline on
Issue Analysis and Drawing Conclusions (internal document).

Chapter 5: Reporting Phase - page 76

 It is the role of the Head of Task to ensure that the preliminary observations
are drafted and the necessary revisions made following input from the
Reporting Member, Chamber Members and the Court.
Before the Preliminary Observations are drafted and in order to ensure that
audit results are of adequate quality, the questions set out in the Vademecum
of General Audit Procedures – Audit Quality management framework checklist
in Appendices 1.2.(a) – (c) should be considered (internal documents).
The draft preliminary observations should be accompanied by a document
which cross-references each observation (per auditee) to the corresponding
and referenced to audit findings. audit findings. Such a document helps the audit team to ensure that all findings
are directly sourced from the audit findings and is necessary in order to
prepare the pre-adversarial meetings.

5.4.2 Structure and layout

Reports should follow a standard layout, in five main sections as shown

below, but within this be structured to help the reader follow and understand
the arguments being presented. In general, this will mean structuring the report
around the audit questions (or sub-questions, if one main audit question was
identified) so as to provide a logical thread between the audit purpose,
observations and conclusions. There must be a logical progression of the
argument, which is clearly signposted by means of the appropriate use of
headings and sub-headings.

The five main sections for the ECA's audit reports are as follows:
1. Executive summary
2. Introduction
3. Audit scope and approach
4. Observations
5. Conclusions and recommendations

 Executive Summary
The executive summary is one of the most critical elements of any report as it
is the most read; in addition, it often forms the basis for the information note
(press release). It is therefore imperative that it make the right impact. The
Reflect report contents,
executive summary should reflect accurately and comprehensively what is in
the report, and guide the reader to the significance of the audit questions and
the answers thereto.
The descriptive parts of the report should be kept to the minimum necessary
to understand the text. The audit scope and approach need only be described
briefly, together with the main observations. The emphasis must be on the
main conclusions of the audit and an outline of the recommendations. For this
with emphasis on main
purpose it should include clear statements such as “The objective of the audit
conclusions and
recommendations, was...”; The audit covered the period...”; The audit examined...”; The audit
found...”, and “The audit recommends...” It is preferable that the text is not too
long (around 2 pages). A fluent and readable style will entice the reader,
avoiding lengthy paragraphs, and using bullet points where appropriate to
present the points being made.

Chapter 5: Reporting Phase - page 77

 In order to avoid unnecessary changes and to reduce the risk of deviation, it is
best to draft the executive summary only when the findings and conclusions
and be balanced have been drafted and reviewed. Particular attention needs to be paid to
ensure that the executive summary is balanced and does not overemphasise
negative findings.

 Introduction
The introduction to the report sets out the context of the audit, helping the
reader to understand both the audit and the observations. It comprises a
description of the audit area, setting out the:
Describe the audit area  objectives of the intervention and its main characteristics;
 principal regulations;
 budgetary arrangements and impact;
 main systems and processes; and
 description of the types of projects and/or programmes financed.
The introduction should not be overly long and detailed. It should contain a
statement but it should not contain audit observations. Where further detail is
succinctly. considered useful for the reader, it can be provided in an annex, and
indications can be given of how the reader could obtain further information
(e.g. internet references).

 Audit Scope, approach and methodology

The audit scope and approach is key to the reader understanding what to
expect from the report, and thereby what use can be made of the results and
conclusions and the degree of reliance to be placed thereon. Different readers
have different needs and expectations. “Front-line” readers such as the
discharge authority and media may not read beyond the executive summary.
However, readers from the audit and academic community usually welcome
Key to understanding more detail, in particular concerning the scope and methodology employed in
the audit work supporting the special report. Therefore, this section within the
main body of the report should set out the following, in concise form, with
unnecessary descriptions avoided: the audit subject; reasons for the audit; the
audit questions to be answered; audit scope; audit criteria; audit methodology
and approach, sources of data, and any limitation to the data used. Detailed
information should then be included by way of annexes.
When providing details, the text needs to focus on what the audit was seeking
to achieve rather than simply what was done. Setting out the audit scope and
what the audit sought to achieve.
approach is particularly important in respect of performance audits, as they
vary much more than financial audits.

 Observations
The observations section represents the main body of the report, containing
the audit findings and audit evidence. The observations should be structured -
as far as possible - around the audit questions, as this provides the focus for
Structure around audit objectives the audit and its conclusions. This reminds readers of the purpose of the audit,
and enables them to have realistic expectations of the report and to be able to
place the observations, conclusions and recommendations in their proper
context.

Chapter 5: Reporting Phase - page 78

 However, within this framework, it is more important that the presentation of
the results is designed to help the reader follow the argument flow. For
example, it may be clearer for the reader if the observations are presented by
and flow of argument,
element of management (e.g. selection and approval of projects), rather than
by administrative level (e.g. Commission, Member State central, Member State
local).
In general, this will allow linked observations to be presented together (e.g.
implementation of the requirements of the regulation), and negates the need to
repeat the regulatory background at each level, thus helping clarity and the
flow of arguments.
When presenting audit observations, the following elements should be
apparent to the reader:
the basis against which the actual situation was
 standard judged - regulatory or normal practice requirements,
or standards set by management or by the auditor;

what was examined and why - the extent and scope of

 work done
whilst identifying these elements. testing

the situation found - including its cause and materiality

 facts
- making apparent the source and extent of evidence

 impact and what the finding means - including the effect on the
consequences EU budget - and why it is important.

41
 Conclusions and Recommendations
The primary purpose of this section is to provide clear answers - conclusions -
to the audit questions, and to make related recommendations on how to
Give clear answers to
audit questions
improve. As such, the conclusions, based on the material observations, must
be presented on the audit questions. The conclusions should provide answers
to the questions set, rather than simply summarising the observations.
The report should include recommendations regarding changes that can be
made to address serious deficiencies reported, where the audit observations
Make recommendations for
the main problems
have demonstrated the potential for significant improvement in operations and
performance. Where corrective action is already under way, it is good practice
to point out this fact.
The ECA’s main auditee is the Commission. However and in particular in
audits undertaken in relation to shared management mode, whenever
and to the right auditees
appropriate the recommendations may be addressed to Member States (or
42
even to a specific Member State or MS authority (see 3.2.3)) .
Recommendations are only to be made when the audit has identified practical
remedies for weaknesses identified. They need to flow from the related
conclusion, and make clear which organisation has the responsibility to act on
that are practical them. Whilst stating what needs to be done, they do not comprise detailed
implementation plans, which are a matter for management. To be constructive,
recommendations must indicate the main components of any changes
required.

41
For more information see additional guidance on writing audit recommendations, QA 012/16 (internal document).
42
More information on addressing recommendations to Member States is provided in DEC 094/15 Recommendations to Member States for
special reports and annual reports (internal document).

Chapter 5: Reporting Phase - page 79

 The recommendations are likely to achieve greater impact where they are
positive in tone and content, are results-oriented (giving some indication of the
and positive intended outcome), bear cost considerations in mind and have been discussed
and agreed with the auditee.

5.4.3 Logic and reasoning

 A focus on the audit questions

The audit questions are the key element of the report, as they set out the
purpose of the audit, and provide the focus for both the structure of the
observations section and the questions to be answered by the conclusions.
They help the reader to understand the audit and its findings. The report
should set out the questions actually answered, i.e. those on which
conclusions were reached, rather than those originally approved in the APM
Report on questions answered
but that were not answered.
Auditors should refer to all significant instances of non-compliance and
43
significant instances of abuse that were found during or in connection with
the audit. Where such instances are not pertinent to the audit questions, they
should nevertheless be communicated to the auditee, preferably in writing, at
the appropriate level.
 Use of examples
Examples are an effective way of illustrating and giving life to technical or
theoretical findings – both positive and negative - thereby helping the non-
expert reader to understand the points being made. However, it is essential
Examples: sound and to the point that they be used sparingly and with care, as they can easily be taken out of
context by selective readers. As they will be closely scrutinised, the examples
should be sound, with findings fully supported by evidence. They also need to
be clearly written, be limited to the point in question, and not be overly detailed.
 Naming of third parties in the ECA's reports
Regarding the naming of third parties in the ECA's reports, the judgement in
44
the Ismeri case was that "the Court might in certain cases be required to give
the names of third parties directly involved in serious malfunctions of the EU
institutions. The assessments made in such cases of the persons concerned
might constitute a fault giving rise to liability on the part of the EU [Community]
Exercise great care
if the facts on which those assessments were based were not accurately
reported or were incorrectly interpreted". Thus, it is imperative that a
heightened duty of care should be exercised in verifying the facts and
interpreting them, in those instances where third parties are either directly
named in the ECA's report or can be easily identified by the reader.

43
Auditing Standard ISSAI 1240/P6 “Abuse involves behaviour that is deficient or improper when compared with behaviour that a prudent
person would consider reasonable...”.
44
Judgement by the Court of First Instance on 15 June 1999 in Case T-277/97 Ismeri Europa v Court of Auditors concerning criticisms made
against Ismeri by the Court in Special Report No 1/96 on the MED programmes.

Chapter 5: Reporting Phase - page 80

 It is important to note that the third party cited also has a right of reply. The
judgement stated that "publication of reports of the Court of Auditors...are
capable of having consequences for those persons such that those concerned
and allow the third party to make
observations
must be enabled to make observations on those points in such reports which
refer to them by name, before those reports are definitively drawn up". Auditors
should ensure that the third party is given the opportunity to make such
observations prior to adoption of the report.

5.4.4 Types of information and data to be included

An audit report should only present data and information that is important for
the reader to understand the context of the audit or its results. Data is not to be
given as a matter of completeness, but to illustrate a specific issue highlighted.
Only include if essential to If data is given, then it needs to be described and analysed in the text so the
understanding, reader knows its purpose. Modern technology and better and more transparent
accounting by the Commission, has greatly increased readers’ access to data.
As such, it will often be sufficient to provide references (e.g. internet links) to
detailed data, rather than providing the data in the report itself.
It is important that the audit is put into context with information on budgetary
expenditure (commitments and payments) and the scope and coverage of the
including budgetary expenditure. audit. The data must not be overly detailed, must be presented in such a way
that it can be linked back to its source (e.g. budgetary nomenclature) and be as
up-to-date as possible.

5.4.5 Style

A well drafted report helps to ensure that the findings are taken seriously,
whereas a poorly presented report will distract the reader, and may prompt
questions about the quality of the findings. Court reports should be interesting
Be accessible,
and easy to read, and provide a positive image of the ECA’s work. Whilst their
technical nature is inevitable, it is more likely that the casual reader will be
encouraged to read further if the reports are accessible.
Consistency is important: a report written in different styles in terms of
approach and expression is difficult to read. It is recommended that one person
consistent, be designated as responsible for ensuring consistent text throughout the
report, even if different individuals are involved in drafting different parts of the
report.

Chapter 5: Reporting Phase - page 81

 The style must be clear and unambiguous. The meaning needs to be
unambiguous, immediately apparent from the text, and not require the reader to interpret what
is being said. In particular:

• long paragraphs are intimidating, and must be avoided;

• long complicated sentences, with many clauses and sub-clauses, are
difficult to read and understand, and to translate. Write in short sentences
where possible;
• the use of active verbs and real subjects helps to avoid ambiguity as
regards who is doing what;
• bullet points – rather than continuous text - can be used (but not overused)
to present lists of items;
• proper use of punctuation can help the reader and avoid misunderstanding;
• the language used needs to be professional, whilst avoiding jargon.
However, the text has to remain sufficiently precise to be understood and
acted upon at the working level.

The reader expects Court reports to be conclusive. Assertions must be

affirmative, and not raise questions, supposition or uncertainty. It is important
that statements are supported by evidence, and that phrases such as ‘it may
be’ or ‘it appears that’ are not used, unless completed by an explanation of why
and conclusive.
the ECA cannot arrive at a definitive conclusion. When an assertion represents
the ECA’s opinion, this must be disclosed, together with the basis for that
opinion. When the assertion comes from another source, such as an
evaluation report, then this needs to be explicitly recognised.

5.4.6 Use of non-textual information

Carefully chosen diagrams, graphs, data and pictures can improve the
appearance of a report and help the reader to understand the background and
Improve appearance of report. findings. Technical assistance in the use of graphics is available through DQC,
whilst recent reports, both from the ECA and national Supreme Audit
Institutions, can be used to stimulate ideas.
 Tables, charts and graphics
Tables and charts are used to reinforce important messages or to present
complex information, such as organisational or financial relationships, in a
simple manner.
Present complex information. When a report requires the relationship between two or more variables to be
explained, this is typically best done with graphs, allowing the relationship to be
illustrated visually. Graphs must be clearly labelled and not overloaded with
data and variables. It is important that graphs within the same report be
presented in a consistent and comparable manner.

Chapter 5: Reporting Phase - page 82

  Maps and pictures
Reports can be made more user-friendly through the use of pictures and maps
where appropriate. Whilst some reports may lend themselves to attractive
illustration or interesting cartography more readily than others, all reports can
be made more attractive and appealing to the reader by the inclusion of such
devices. Indeed, photographs can provide a visual theme running through a
More attractive and appealing.
report, as well as being used to illustrate specific points. Non-textual
information such as pictures and maps form part of the report and therefore
such material should be included in all approval and adoption stages of the
preliminary observations and special report, including the adversarial
procedure.
 Numbers and percentages
Numbers and percentages must be presented with an appropriate level of
precision, and be consistent within the same sentence or paragraph. In
general, rounded numbers are easier to read. Specific guidelines are issued by
DQC on such matters.

5.5 REVIEWING THE REPORT

The Reporting Member is responsible for submitting high-quality reports to the

Chamber. To this end, the Member should assess whether the report is
Reporting Member, audit team and
Chamber EQCR,
clearly drafted, fair, balanced, and supported by evidence; decide whether the
report faithfully presents the results of the audit; and determine whether the
draft places the performance of the auditee in its proper context.
Effective supervision and review within the audit team (section 2.5) is
particularly important in supporting the Reporting Member in achieving a high-
quality report.
The first time a full draft of the audit report – known as the ‘preliminary
observations’ – is submitted to the chamber, it is subject to EQCR. In this case,
in addition to initial self-assessment by the audit team, responsibility for EQCR
is split between the Chamber Director (on matters relating mainly to evidence)
and DQC (on matters relating mainly to presentation).
Detailed procedures for these quality control arrangements are contained in
the Vademecum of General Audit Procedures – Audit Quality Management
Framework – EQCR (internal document).
The Audit Chamber should then review and approve the draft preliminary
Audit Chamber.
observations.

Chapter 5: Reporting Phase - page 83

 5.6 CLEARING THE REPORT 45

The 'clearance' process covers the period from the time the draft preliminary
From first reading to final adoption, observations are first submitted to the Audit Chamber, through the adversarial
procedure with the Commission, until final adoption by the Chamber.
Once the Chamber has approved the draft preliminary observations, this
document is then forwarded to and discussed with the auditee. To ensure as
far as possible that the process of clearing the draft report with the auditee is
efficient and effective, it is essential to have:

• good communication throughout the audit, based on the ECA's 'no

surprises' approach; and
• timely and effective communication with the auditee on the preliminary
audit findings.
Article 163 of the Financial Regulation (FR) requires that “the Court of Auditors
shall transmit to the institution or the body concerned any observations which
are, in its opinion, such that they should appear in a special report. Those
observations shall remain confidential and shall be subject to an adversarial
procedure. The institution or body concerned shall inform the Court of Auditors,
within six weeks of transmission of those observations, of any replies it wishes
to make in relation to those observations”.
The six weeks period includes what is generally known as the adversarial
procedure, during which a meeting is arranged between the institution
concerned and the ECA. The primary purposes of this meeting are to resolve,
including the Adversarial firstly, any disagreements over the facts and, secondly, any differences of
Procedure opinion between the ECA and the Institution (or other body) over interpretation
of the evidence. The meeting is also used to finalise the institution's replies to
the observations. DQC acts as an intermediary in arranging and chairing such
meetings.
In order to ensure a smooth adversarial procedure meeting, it is good practice
and pre-adversarial meeting,
to hold a pre-adversarial meeting between the audit team and the auditee.
The changes made to the text of the draft special report as a result of the
adversarial procedure must be apparent in the final draft report to be submitted
for Chamber reading, and the reasons given for the main changes.
The same FR Article also states that "The Court of Auditors shall ensure that
the special reports are drawn up and adopted within an appropriate period of
th 46
time, which shall, in general, not exceed 13 months . The special reports,
together with the replies of the institutions or bodies concerned, shall be
with procedures outlined in the transmitted without delay to the European Parliament and the Council, each of
Financial Regulation. which shall decide, where appropriate in conjunction with the Commission,
what action is to be taken in response. The Court of Auditors shall take all
necessary steps to ensure that the replies to its observations from each
institution or body concerned, as well as the timeline for the drawing up of the
special report are published together with the special report.”

45
Additional guidance is available in the Vademecum of General Audit Procedures : Clearing audit findings for performance auditors (internal
document).
46
See also DEC 116/15 FINAL Measuring the 13 month target (internal document)..

Chapter 5: Reporting Phase - page 84

Once the definitive version of the special report has been adopted by the
Chamber, it is submitted to the ECA’s Translation and Language Services
Directorate for translation into the official languages.

Chapter 5: Reporting Phase - page 85

 5.7 DISTRIBUTING THE REPORT

Special Reports are published and distributed in accordance with the ECA's
procedures. The latter include making the report available on the ECA's
website, in addition to issuing an 'information note' for the press. The Reporting
Member normally presents the report to a subsequent meeting of the CONT
(the European Parliament's Budgetary Control Committee), and holds a press
conference if (s)he deems it appropriate.

5.8 REPORT FOLLOW-UP

5.8.1 Follow-up of report recommendations and findings

Assessing and measuring the impact of the ECA's performance audit reports is
a necessary element in the cycle of accountability. The recommendations
made in special reports should be followed up in order to establish and assess
the measures taken. Indeed, the very existence of the follow-up process can
encourage the effective implementation of report recommendations by
auditees.
Following up report recommendations serves four main purposes:
 increasing the effectiveness of audit reports - the prime reason for
following up audit reports is to increase the probability that
recommendations will be implemented;
 assisting the legislative and budgetary (including discharge) authorities
Following up recommendations
serves several purposes.
- following up recommendations may be valuable in guiding their
actions;
 evaluating the ECA's performance - follow-up provides a basis for
assessing and evaluating the ECA's performance; and
 creating incentives for learning and development - follow-up activities
may contribute to better knowledge and improved practice.
The follow-up work takes the form of “limited reviews” carried out by the Audit
Chambers, which assess the extent to which the auditee (generally the
Commission) has addressed the findings and recommendations contained in
the ECA’s special reports. It does not however, assess the effectiveness of
those actions taken by the auditee, as this would require a detailed audit
enquiry. Accordingly, a detailed examination of a specific special report may be
carried out by Audit Chambers as an ‘in-depth’ follow-up audit, if considered
necessary.
Follow-up will normally take place three years after publication of the special
report. The starting point could be the regular follow-up reports submitted by
the Commission and other institutions to Parliament when these are available,
as well as the Commission’s follow-up database known as RAD
(Recommendations, Actions, Discharge). The ECA reports on the follow-up of
its special reports in an annual follow-up report or by way of separate special
reports. The annual follow-up report contains the results of the work carried out
by the ECA to assess the corrective actions of the auditee in response to the
ECA’s audit findings and recommendations.

Chapter 5: Reporting Phase - page 86

Selection of reports for follow-up During the AWP process, Audit Chambers select the special reports in
consultation with Chamber V which they intend to follow up as part of this
47
annual task . The reports are selected on the basis of two criteria: whether
more than two to three years have elapsed for the auditee to address the
recommendations, and whether the recommendations are still relevant.
Planning the work The work, which is less than that required for a reasonable assurance audit, is
performed on the basis of a brief planning document rather than a detailed
planning memorandum. Under normal circumstances, following up one special
report should not require more than 10 auditor-weeks.
The review as performed by the Audit Chambers, follows up the audit findings
(weaknesses) and recommendations included in special reports, and includes
the following stages:
1) a review of the Commission’s management database (RAD) as a
preliminary source of data, in respect of the audit reports being reviewed as
well as a review of the ECA’s relevant databases;
2) a documentary review of annual reports, action plans, policy documents,
specific reports and their analysis;
3) obtaining and analysing evidence on the specific actions taken by the
Commission to implement recommendations and address weaknesses
identified;
4) sending clearing letters containing preliminary findings drafted according
to a standard template.
The review by the audit teams should ascertain the state of implementation of
the ECA’s recommendations (fully implemented; implemented in most
respects; implemented in some respects; not implemented; no longer relevant;
could not be verified).
In reviewing the implementation of the ECA’s recommendations, the audit team
may wish to review the status of the Commission’s Action Plans in relation to
Scope and approach the requests of the Discharge Authority (as required by the Commission’s
Internal Control Standard No 9) as a source of complementary information.
Chamber V is responsible for coordinating the task either in a form of a
contribution to the annual report chapter or as a separate special report. The
Reporting Member is a permanent Member of Chamber V.
The detailed procedure for the follow-up can be found in the Vademecum of
48
General Audit procedures (internal document) .

5.8.2 Follow-up of how the audit report has been perceived

When a performance audit has been completed, there are various

opportunities for obtaining information on how it has been received, for
instance by observing reactions from auditees and Parliament and in the
media. In addition, external experts could be asked to scrutinise performance
audit reports or to give their opinions on the quality of the work.

47
DEC 006/10, CH 234/10 and.DEC 44/16 (internal documents).
48
Note: Court’s methodology for the follow-up is currently being further developed (as indicated by DEC 44/15 – internal document) in particular in
relation to recommendations addressed to the Member States and the upcoming ECA strategy 2018-2020.

Chapter 5: Reporting Phase - page 87

 As most performance audits produce genuine learning points, both for the
teams involved and the whole Court, it can be a valuable exercise for the audit
team to carry out a review after the report has been published, in order to
identify:
 what worked well and why
 what was less successful and the reasons
Acts as a learning tool.
 lessons for the future, and possible wider applications for all
performance audits

Chapter 5: Reporting Phase - page 88

 PERFORMANCE AUDIT MANUAL

DETAILED TABLE OF CONTENTS

OVERVIEW Error! Bookmark not defined.

GENERAL INTRODUCTION ERROR! BOOKMARK NOT DEFINED.

Purpose & Content of the Performance Audit Manual Error! Bookmark not defined.
Structure Error! Bookmark not defined.
Glossary of concepts and technical terms Error! Bookmark not defined.

CHAPTER 1 FRAMEWORK FOR PERFORMANCE AUDITS BY THE EUROPEAN COURT OF

AUDITORS ERROR! BOOKMARK NOT DEFINED.
1.1 Introduction Error! Bookmark not defined.
1.2 Performance audit and sound financial managEment in the European Union Error! Bookmark not defined.
1.2.1 Performance Audit Error! Bookmark not defined.
1.2.2 Sound financial management: Treaty & Financial Regulation Error! Bookmark not defined.
1.2.3 Management methods to implement the budget Error! Bookmark not defined.
1.2.4 Internal control system to achieve sound financial management Error! Bookmark not defined.
1.2.5 Relationship of performance audit & financial and compliance audit Error! Bookmark not defined.
1.2.6 Relationship between performance audit & evaluation Error! Bookmark not defined.
1.3 THE ECA's mandate & objectives for performance audits Error! Bookmark not defined.
1.3.1 The ECA's legal obligations Error! Bookmark not defined.
1.3.2 The ECA's objectives Error! Bookmark not defined.

CHAPTER 2 THE PERFORMANCE AUDIT APPROACH AND THE 3 ‘E’S ERROR! BOOKMARK NOT DEFINED.
2.1 Introduction Error! Bookmark not defined.
2.2 An audit approach focusing on performance achieved Error! Bookmark not defined.
2.2.1 Auditing performance directly Error! Bookmark not defined.
2.2.2 Auditing control systems Error! Bookmark not defined.
2.3 How to apply the 3 ’E’s Error! Bookmark not defined.
2.3.1 The use of logic models in performance audits Error! Bookmark not defined.
2.3.2 Application of the concepts Error! Bookmark not defined.
 Economy Error! Bookmark not defined.
 Efficiency Error! Bookmark not defined.
 Effectiveness Error! Bookmark not defined.
2.4 Essential qualities of good performance audits Error! Bookmark not defined.
2.4.1 Sound judgement is exercised throughout the audit process Error! Bookmark not defined.
2.4.2 Methodologies are appropriate and combined to capture a range of data Error! Bookmark not defined.
2.4.3 Audit questions are set which can be concluded against Error! Bookmark not defined.
2.4.4 Risks to delivering the audit report are analysed and managed Error! Bookmark not defined.
2.4.5 Tools are employed to help achieve successful delivery of the audit Error! Bookmark not defined.
2.4.6 Evidence is sufficient, relevant and reliable to support the audit findings Error! Bookmark not defined.
2.4.7 Possible conclusions and recommendations of the final report are considered from the
planning phase onwards Error! Bookmark not defined.
2.4.8 Transparency - a 'no surprises' approach - is adopted with the auditee and other
stakeholders Error! Bookmark not defined.
2.5 Quality control Error! Bookmark not defined.

CHAPTER 3 PLANNING THE AUDIT ERROR! BOOKMARK NOT DEFINED.

3.1 Introduction Error! Bookmark not defined.
3.2 The audit planning phase Error! Bookmark not defined.
3.2.1 Purpose of audit planning Error! Bookmark not defined.
3.2.2 Preliminary work Error! Bookmark not defined.
3.2.3 Acquire up-to-date knowledge of the audit area Error! Bookmark not defined.
 Identify the objectives and logic of the intervention and related indicators Error! Bookmark not defined.
 Determine the resources made available for the intervention Error! Bookmark not defined.
 Determine the respective responsibilities of the various actors Error! Bookmark not defined.
 Identify the key management and control processes, including IT systems Error! Bookmark not defined.
 Define information needs for management and control purposes Error! Bookmark not defined.
 Identify the risks to sound financial management Error! Bookmark not defined.
3.2.4 Outline the audit Error! Bookmark not defined.
 Take account of previous audits and evaluations Error! Bookmark not defined.
 Consider the potential audit questions, criteria, evidence, methodology, scope and impactError! Bookmark not defin
 Consider the timing and resourcing of the proposed audit Error! Bookmark not defined.
3.2.5 Assess if the audit is realistic, realisable and likely to be useful Error! Bookmark not defined.
3.3 The Audit Planning Memorandum Error! Bookmark not defined.
3.3.1 Purposes and contents of the APM Error! Bookmark not defined.
 Define the audit questions Error! Bookmark not defined.
 Set the audit scope Error! Bookmark not defined.
 Establish the audit criteria to be used Error! Bookmark not defined.
 Identify the audit evidence required and its sources Error! Bookmark not defined.
 Define the audit methodology to be employed Error! Bookmark not defined.
 Consider the potential audit observations, recommendations and impact Error! Bookmark not defined.
 Determine the timetable, resources, and supervision and review arrangements Error! Bookmark not defined.
 Communicate with the auditee Error! Bookmark not defined.
3.3.2 Drawing up the APM Error! Bookmark not defined.
3.4 Quick special reports Error! Bookmark not defined.
Annex I: Contents of an Audit Planning Memorandum Error! Bookmark not defined.
Annex II: Evidence Collection Plan Error! Bookmark not defined.
Annex III: Outline audit programme Error! Bookmark not defined.

CHAPTER 4 EXAMINATION PHASE ERROR! BOOKMARK NOT DEFINED.

4.1 Introduction Error! Bookmark not defined.
4.2 Delivering sufficient, relevant & reliable audit evidence Error! Bookmark not defined.
4.2.1 Purpose of and approach to the examination phase Error! Bookmark not defined.
4.2.2 The nature of evidence in performance audits Error! Bookmark not defined.
4.2.3 Sufficiency, relevance and reliability of evidence Error! Bookmark not defined.
4.2.4 Sources of evidence Error! Bookmark not defined.
4.2.5 Types of evidence Error! Bookmark not defined.
4.3 Collecting & analysing Data Error! Bookmark not defined.
4.3.1 Purpose of and approach to data collection and analysis Error! Bookmark not defined.
4.3.2 Data collection process Error! Bookmark not defined.
4.3.3 Using the work of others Error! Bookmark not defined.
4.3.4 Data collection and ethics Error! Bookmark not defined.
4.3.5 Data analysis Error! Bookmark not defined.
4.4 Deriving valid audit findings Error! Bookmark not defined.
4.4.1 Need for a sound basis Error! Bookmark not defined.
4.4.2 Drafting audit findings Error! Bookmark not defined.
4.5 Communicating audit findings Error! Bookmark not defined.
4.6 Documenting the audit Error! Bookmark not defined.
4.6.1 Purpose and approach Error! Bookmark not defined.
4.6.2 Audit documentation referencing Error! Bookmark not defined.
4.7 Audit management and quality control arrangements Error! Bookmark not defined.
4.7.1 Audit management, including supervision and review Error! Bookmark not defined.
Annex I: Data collection and analysis methods Error! Bookmark not defined.

CHAPTER 5 REPORTING PHASE ERROR! BOOKMARK NOT DEFINED.

5.1 Introduction Error! Bookmark not defined.
5.2 Report quality Error! Bookmark not defined.
 Objective Error! Bookmark not defined.
 Complete Error! Bookmark not defined.
 Clear Error! Bookmark not defined.
 Convincing Error! Bookmark not defined.
 Relevant Error! Bookmark not defined.
 Accurate Error! Bookmark not defined.
 Constructive Error! Bookmark not defined.
 Concise Error! Bookmark not defined.
5.3 Planning the report Error! Bookmark not defined.
5.3.1 Consider the addressees Error! Bookmark not defined.
5.3.2 Approach to planning the report Error! Bookmark not defined.
5.3.3 Prepare a drafting plan Error! Bookmark not defined.
5.4 Drafting the report Error! Bookmark not defined.
5.4.1 Approach to writing the report Error! Bookmark not defined.
5.4.2 Structure and layout Error! Bookmark not defined.
 Executive Summary Error! Bookmark not defined.
 Introduction Error! Bookmark not defined.
 Audit Scope, approach and methodology Error! Bookmark not defined.
 Observations Error! Bookmark not defined.
 Conclusions and Recommendations Error! Bookmark not defined.
5.4.3 Logic and reasoning Error! Bookmark not defined.
 A focus on the audit questions Error! Bookmark not defined.
 Use of examples Error! Bookmark not defined.
 Naming of third parties in the ECA's reports Error! Bookmark not defined.
5.4.4 Types of information and data to be included Error! Bookmark not defined.
5.4.5 Style Error! Bookmark not defined.
5.4.6 Use of non-textual information Error! Bookmark not defined.
 Tables, charts and graphics Error! Bookmark not defined.
 Maps and pictures Error! Bookmark not defined.
 Numbers and percentages Error! Bookmark not defined.
5.5 Reviewing the report Error! Bookmark not defined.
5.6 Clearing the report Error! Bookmark not defined.
5.7 Distributing the report Error! Bookmark not defined.
5.8 Report follow-up Error! Bookmark not defined.
5.8.1 Follow-up of report recommendations and findings Error! Bookmark not defined.
5.8.2 Follow-up of how the audit report has been perceived Error! Bookmark not defined.

DETAILED TABLE OF CONTENTS 89