Supplier Assessments / Audits
CONCEPT
HEIDELBERG
GMP Compliance for
Computerized Systems Validation
January 16 - 17, 2003 at Istanbul, Turkey
Supplier
Assessments / Audits
Dr.-Ing. Guenter Generlich
[email protected]
Computerized Systems Validation Supplier Assessment 1
Dr. Guenter Generlich
Audit / Auditing
The independent examination of a sample of records,
activities and/or systems to assess the state of governance,
to ensure compliance with established controls, policies and
procedures, and to recommend control improvements
where judged necessary to reduce risks.
Role performed by Internal and External Auditors and, for
computer systems, Computer Auditors.
Computerized Systems Validation Supplier Assessment 2
Dr. Guenter Generlich
1
Supplier Assessments / Audits
What are the Reasons for an Audit?
An audit will be An audit can help
• expensive reduce the validation
• labor intensive risk - the risk that
• not wanted by an IT supported
our management business system
• not necessary does not comply
so what’s the point? with regulatory
requirements.
Computerized Systems Validation Supplier Assessment 3
Dr. Guenter Generlich
Assessment Goals
• Ensure compliance to technical, commercial, and regulatory
requirements and directives
• Develop relationships with supplier, clarify expectations, and
identify misunderstandings and risks
• Learn how the supplier´s organization works
• Local managers and professionals should want to improve
their own operation
• Identify major problems before they create unjustified costs
• Enroll the supplier´s opinion leaders in the change process
Computerized Systems Validation Supplier Assessment 4
Dr. Guenter Generlich
2
Supplier Assessments / Audits
Typical ?
Management is often so focused
on finding solutions
that it fails to define the problems
* * *
“I don’t want to hear your problems,
I want to hear your solutions”
Computerized Systems Validation Supplier Assessment 5
Dr. Guenter Generlich
Who Does What
Acceptance
User Requirements
Testing
application
driven
Functional System
Design + Spec Testing
Technical Integration
Design + Spec Testing
technology
driven
Supplier Programs
Unit Testing
Developer Development
Computerized Systems Validation Supplier Assessment 6
Dr. Guenter Generlich
3
Supplier Assessments / Audits
Suppliers We Need to Assess
• Standard packages
• Custom systems / bespoke solutions
• Customized / configurable systems
• Technology products
• Service providers
Audit new vendors, and don’t forget old vendors!
Computerized Systems Validation Supplier Assessment 7
Dr. Guenter Generlich
Types of Audits
• 1st party audits
- internal audits by the pharmaceutical manufacturer
on itself
• 2nd party audits
- by the user company on its suppliers (which can be an
internal or an external organization)
• 3rd party audits
- of suppliers by an organization acting independently
of the pharmaceutical manufacturer(s)
Computerized Systems Validation Supplier Assessment 8
Dr. Guenter Generlich
4
Supplier Assessments / Audits
Joint Audits
• Reduced time and effort to both users and suppliers
• Increased co-operation between user companies
• Summing-up the expertise
• Better base to realize changes, improvements
• Progress towards common auditing standards
Computerized Systems Validation Supplier Assessment 9
Dr. Guenter Generlich
Maturity Scale
1 Pharmaceutical companies auditing infrequently or at the
wrong time
2 Individual pharma companies maintaining their own audit
schedule and performing independent audits
3 Multiple companies visiting a single supplier for the same
product
4 Pharma companies pooling resource to audit a supplier, but
with each producing their own report
5 Pharma companies requesting a third party to audit a supplier
and produce a single report
6 Pharmaceutical user groups in conjunction with suppliers
manage joint audit process
Computerized Systems Validation Supplier Assessment 10
Dr. Guenter Generlich
5
Supplier Assessments / Audits
Supplier Audit Cycle
Preliminary Assessment initial evaluation , pre-qualification,
pre-audit questionnaire
Detailed Audit in-depth, full quality, pre-contract (!)
Follow-up Audit re-audit, monitor audit
Surveillance Audit periodic audit
Computerized Systems Validation Supplier Assessment 11
Dr. Guenter Generlich
GAMP Software Categories
Categ. Software Type Validation Approach
1 Operating System Record version, including service pack; operating system chal-
lenged indirectly by the functional testing of the application.
Non-configurable firmware: record version; configurable firm-
2 Firmware
ware: record version and configuration, calibrate instruments,
verify operation against requirements; manage custom/bespoke
firmware as categorie 5 software.
3 Standard Software Record version (and configuration of environment) and verify
Packages operation against requirements; consider auditing the supplier
for critical and complex applications.
Configurable Record version and configuration, verify operation against re-
4
Software Packages quirements; normally audit the supplier for critical and complex
COTS applications; manage any custom/bespoke programming as
categorie 5 software.
5 Custom (Bespoke) Audit supplier and validate complete system
Software
Computerized Systems Validation Supplier Assessment 12
Dr. Guenter Generlich
6
Supplier Assessments / Audits
Audit Planning
• Prerequisites
• Preparing for the Audit
• GAMP or PDA Technical Report 32
• Conducting the Audit
• After the Audit
• Audit Report
• Follow-up
Computerized Systems Validation Supplier Assessment 13
Dr. Guenter Generlich
PDA - ARC
PDA Parenteral Drug Association www.pda.org
ARC Audit Repository Center www.auditcenter.com
Advantages Issues
• Highly professional contents • Less contact with supplier
• Cost savings • Relevant aspects covered?
• Immediately available
• Reliably suppliers
Computerized Systems Validation Supplier Assessment 14
Dr. Guenter Generlich
7
Supplier Assessments / Audits
Prerequisites
• Objectivity - The auditor must be unbiased
- Gather factual evidence
- Avoid personal judgements
- Keep an open mind
• Independence - No conflict of interest
- Avoid the appearance of conflict
• Courtesy - Be prompt and timely with all communication
- Behave like a guest
- Observe all site safety and security provisions
- Establish a professional report
Computerized Systems Validation Supplier Assessment 15
Dr. Guenter Generlich
Preparing for the Audit
• Pre-work requirements
• Audit team selection
• Planning and scheduling
• Preparing tailored audit criteria
Computerized Systems Validation Supplier Assessment 16
Dr. Guenter Generlich
8
Supplier Assessments / Audits
Pre-Work Requirements
• Supplier profile - commercial focus, market position
- financial stability
- organizational set-up & complexity
- focus quality management
• Product profile - description: what is it
- market it was designed for
- consistence: s/w, h/w, …
- related things: what’s included
- target use
- related risks
• Specific client needs
Computerized Systems Validation Supplier Assessment 17
Dr. Guenter Generlich
Previous Audit Info
• Past performance is a good measure of future
performance
• Good lead for problematic areas
• Communicate intra/inter company
• Check confidentiality agreements if results were provided
by other clients
• Input for surveillance program
Computerized Systems Validation Supplier Assessment 18
Dr. Guenter Generlich
9
Supplier Assessments / Audits
Audit Team
• Experienced lead auditor (certified ?)
• Skill assessment & fulfillment
• Individuals with different skill sets, e.g.
– Users and QA testing procedures, change
members management, adherence to
stated QA/QC standards
– IT professionals product development, software/
and engineers hardware standards, security,
technical documentation
Computerized Systems Validation Supplier Assessment 19
Dr. Guenter Generlich
Planning & Scheduling
• Communication with supplier
- preliminary scheduling: phone, e-mail
- formal letter and schedule
- negotiated best fit for execution
- confirm dates and schedules
• Audit questionnaire
- according to GAMP or PDA Technical Report 32
- lead time at least 2 weeks
• Tailored audit criteria
Computerized Systems Validation Supplier Assessment 20
Dr. Guenter Generlich
10
Supplier Assessments / Audits
PDA Technical Report 32
1. Quality System
2. Project Management
3. Methodology
4. Testing
5. Configuration Management
6. Manufacturing
7. Documentation and Records Management
8. Security
9. Training and Education
10. Maintenance
Computerized Systems Validation Supplier Assessment 21
Dr. Guenter Generlich
PDA Technical Report 32
1. Quality System 6. Manufacturing
2. Project Management 7. Documentation and
3. Methodology Records Management
4. Testing 8. Security
5. Configuration 9. Training and
Management Education
10. Maintenance
Computerized Systems Validation Supplier Assessment 22
Dr. Guenter Generlich
11
Supplier Assessments / Audits
Checklist Example from TR32
# Question Answer Objective Evidence
24. Do testing documents exist for:
24.1 Unit level testing? Y Unit level testing corresponds to the soft-
ware item testing. See procedure “Soft-
ware Item Testing”, SIT002A.
24.2 Integration testing? Y Integration level testing is performed in
the test phase software item test. The
stepwise composition of system compo-
nents up to the complete system is inte-
grated into this test phase. See procedure
“Software Item Testing”SIT02B and C.
24.3 System level testing? Y System level testing covers complete sys-
tem functionality, including system inter-
faces to other systems; it includes the test
phases "System Test" and "Interface Test".
Computerized Systems Validation Supplier Assessment 23
Dr. Guenter Generlich
Conducting the Audit
• Opening - review purpose, scope, objectives
- agree on common goal
- schedule
- supplier presentation (limit 30’, facility tour?)
• Collecting Evidence
- audit guide, criteria checklist, follow-up list
- interviews: establish rapport, confirm/verify
results, suspend judgement, no debate
- document and record findings
• Wrap-up - meeting review
- follow-up list
- preliminary observations list
- obtain supplier acknowledgement
Computerized Systems Validation Supplier Assessment 24
Dr. Guenter Generlich
12
Supplier Assessments / Audits
After the Audit
• Draft Report published within 10 working days
• 10 working days for supplier’s review and formal comment
• 30 days for Response and Commitment Report
• Request for extensions ?
• Supplier status reports
• Metrics?
• Follow-up
- quick hits/easy fixes and extended term
- set due dates
Computerized Systems Validation Supplier Assessment 25
Dr. Guenter Generlich
Report Access & Distribution
• Establish who will receive copy of final report and
source material
• Minimal distribution
• Intranet not recommended
Computerized Systems Validation Supplier Assessment 26
Dr. Guenter Generlich
13
Supplier Assessments / Audits
Content of Report
• Cover Page
• Assessment / Audit History
• Table of Contents
• Introduction: purpose, scope, team members
• Description of Suppliers Business
• Summary: key audit areas, key observations (+ & -)
• Response and Commitment
• List of Evidence: documents reviewed, referenced (no copies)
• Detailed Results: audit checklist
Computerized Systems Validation Supplier Assessment 27
Dr. Guenter Generlich
Some Typical Areas of Concern
• Documentation generally weak
• Testing
– test cases not defined under GxP
– review of results
• Change control system missing
• Subcontractors not under control
• Not familiar with pharmaceutical regulations / guidelines
Computerized Systems Validation Supplier Assessment 28
Dr. Guenter Generlich
14
Supplier Assessments / Audits
Supplier Acceptance or Rejection
Based on the outcome of the audit you may decide
• to use the supplier unconditionally
• to use the supplier for certain products or versions only
• to use the supplier subject to specific corrective actions
• to agree with the supplier on the application of a
documented QMS for the purposes of the contract
• to prohibit the use of the supplier
Computerized Systems Validation Supplier Assessment 29
Dr. Guenter Generlich
The Verification Process May . . .
Look good: Does not look good:
• Validation would be • Do not purchase !!
the same as for in- • Extensive validation required
house developed with large number of test
systems cases
• Consortium with other users
Computerized Systems Validation Supplier Assessment 30
Dr. Guenter Generlich
15
Supplier Assessments / Audits
Vendors with ISO 9000
• Not recognized by regulatory bodies
• ISO 9000 Validation
• ISO 9000 process oriented
• Validation product oriented
an ISO-certified vendor will (most probably)
but
deliver a better product and documentation
Computerized Systems Validation Supplier Assessment 31
Dr. Guenter Generlich
Postal Audits
• No substitute to visiting a supplier
• Part of tendering process - further consideration justified?
• Preliminary info to focus effort on critical areas during the
detailed audit
• Follow-up audit to review outstanding corrective actions
• Means of re-assessing on-going suppliers or service
providers
• Auditing other premises of supplier with same QMS
Computerized Systems Validation Supplier Assessment 32
Dr. Guenter Generlich
16
Supplier Assessments / Audits
Configurable Standard Packages
Base Kernel Implementation
Documentation Documentation Validation
Perform Supplier Kernel Team Owner
Developer User
Approve ROCHE Q ROCHE Q ROCHE Q
Computerized Systems Validation Supplier Assessment 33
Dr. Guenter Generlich
Hints and Tips
for Your Supplier Assessment
• Contact or join user group
• Visit reference customers
• Perform shared audits with other companies
• Operation ? documentation ??
• Plan for pilot installation
• Use your own license template
• Negotiate escrow agreement
• You cannot buy a validated system
Computerized Systems Validation Supplier Assessment 34
Dr. Guenter Generlich
17
Supplier Assessments / Audits
Escrow Agreement
“An Escrow is a deed, a bond, money, or a piece of property
held in trust by a third party to be turned over to the grantee
only upon fulfillment of a condition.”
from http://www.webster.com/
Computerized Systems Validation Supplier Assessment 35
Dr. Guenter Generlich
Auditor Code of Ethics (1)
1 I will be honest, impartial, and candid and will demonstrate
freedom of mind and approach that will ensure objective
viewing of the operation being audited.
2 I will conduct myself in a dignified manner that reflects well
upon my profession and my company.
3 I will inform my company of any personal involvement
(business connections, financial interests, employment
history, or personnel or family affiliations) that might
influence, or appear to influence, my judgement ort
jeopardize my independence in my ability to assess the
suitability of the operation being audited.
Computerized Systems Validation Supplier Assessment 36
Dr. Guenter Generlich
18
Supplier Assessments / Audits
Auditor Code of Ethics (2)
4 I will undertake only those audits compatible with the
degree of training, experience, and proficiency I hold with
regard to the operation being audited.
5 I will issue reports that clearly, factually, and accurately
describe the operation being audited, and that are
constructive in nature.
6 I will not disclose information concerning the business
affairs or technical processes of the client/ supplier without
obtaining prior written consent to do so from the
client's/supplier's management.
Computerized Systems Validation Supplier Assessment 37
Dr. Guenter Generlich
Auditor Code of Ethics (3)
7 I will not disclose any proprietary information or
confidential data provided by a company being audited
without obtaining consent to do so from that company's
management.
8 I will strive to contribute to the development of improved
audit techniques and methods within the quality audit
profession and the PDA Process Model.
Computerized Systems Validation Supplier Assessment 38
Dr. Guenter Generlich
19
Supplier Assessments / Audits
Hot Buttons of CSV
Essential key points that characterize
a sound validation approach, such as:
? Four eyes principle ? Documentation of results
? Team approach? ? Operation ? documentation
? Owner is responsible ? Development method
? Validation plan ? Ongoing evaluations
? Predefined test results ? Supplier assessment not delegated
? Independent approval ? Archive well organized
? Change management ? Risk assessment
? Ongoing training ? Standard Operating Procedures
? Requirements traceable ? System access defined
? Expert judgement ? Never touch a standard source code
Computerized Systems Validation Supplier Assessment 39
Dr. Guenter Generlich
Frequent Misconceptions of CSV
? Long-time use = validation
? One-off activity
? Not needing documentation
? Documentation will always be voluminous
? Just software testing
? Not necessary to know requirements/ user needs
? Just paper generation
? CSV is a job for IT or QA/QC
? Regulatory bodies don’t care about IT systems
? We bought a validated system
Computerized Systems Validation Supplier Assessment 40
Dr. Guenter Generlich
20
