ANNEX - POLICY

PROTOTYPES

INSTRUCTIONS FOR
USE & DISCLAIMERS
CONTENTS
1. Instructions For Use 1
2. Instructions For Use 2
3. Instructions For Use 3
4. Disclaimer 1
5. Disclaimer 2
INSTRUCTIONS FOR USE 1

Use Case : AI software for detection of

eye pathologies based on eye retina
images
INSTRUCTIONS FOR USE 1

Contents
1. Introduction 1
2. Who we are 1
3. Product description, intended purpose, intended users 1
4. Clinical benefits for patients 2
5. Characteristics, capabilities, and limitations 2
5.1. Characteristics and capabilities 2
5.1.1. Specifications for input data & information on ... 2
5.1.2. Expected output 2
5.1.3. Accuracy, robustness and performance 3
5.1.4. Installation and use 3
5.1.5. Image Capture Protocol 4
5.1.6. Uploading and Analysis 4
5.1.7. Interpretation of results and related actions 4
5.1.8. Retention periods 4
5.1.9. Cybersecurity, safety and best-practice tips 5
5.2. Limitations/ Risks to health and safety 5
6. Human oversight: Organizational and technical measures 5
6.1. Organizational Measures 5
6.2. Technical Measures 6
7. Maintenance and care instructions 6
7.1. Expected lifetime 6
7.2. Maintenance and care measures 7
8. Troubleshooting 7
9. Logs 7
10. Reference to other legal instruments 8
10.1. Other relevant policies and documents 8
10.2. ISO Standards and Best Practices for Medical Devices 8
11. Support and contact details 8
1. Introduction

These Instructions For Use (hereinafter “IFU”) aim to provide users with information and
guidance about our product –including its capabilities, benefits and limitations, as well as
a comprehensive explanation of usage, maintenance and oversight instructions.

2. Who we are

• Provider: [Company X]
• Product: [Retina] AI Software
• Address: Rue X, Brussels 1000, Belgium
• Email: [email protected]
• Phone: +32 XXX.XX.XX.XX

3. Product description, intended purpose, intended users

Product description: [Retina] AI Software is a cutting-edge software tool developed

by [Company X] which incorporates AI-based algorithms to evaluate and analyze eye
retina images in order to assist and support ophthalmologists and healthcare teams in
early detection of specific diseases, such as diabetes-related eye pathologies. Images
are captured via retina cameras in hospitals and private practices. These images are
uploaded to the web portal/platform (SAAS) where the analysis is performed. The
outcome is then sent to and can be reviewed by the ophthalmologists, healthcare
providers and respective healthcare teams.

Intended purpose: The AI-based evaluation and analysis of retina images by our software
assists ophthalmologists, healthcare providers and respective healthcare teams in early
detection of specific diseases, such as diabetes-related eye pathologies. The [Retina] AI
Software results help the healthcare provider decide whether to refer for further screening
and treatment. It is intended to be used in a medical setting by responsible professionals
who are aware of these instructions for use.

Intended users and beneficiaries:

• Ophthalmologists: primary users, end responsible for using and interpreting
the AI-supported analysis of retina images when conducting medical
diagnosis.
• Healthcare providers (incl. nurses): secondary users, responsible for assistance
in the capture, upload and (preliminary) review of retina images analysis
results and (possibly) taking a decision whether to refer the patient to
ophthalmologists for further screening.
• Adult) patients: The beneficiaries of accurate and timely AI-enabled
diagnostics (e.g. detection of diabetes-related eye pathologies), also
responsible for consent provision for retina image analysis.

1
4. Clinical benefits for patients

[Retina] AI Software enables a quick, affordable and early detection of diseases, such as
diabetes-related eye pathologies. For people with diabetes, autonomous AI systems used
by the healthcare provider have the potential to improve earlier detection of diabetes-
related eye pathologies, and to immediately refer the patients to an ophthalmologist for
further diagnosis and treatment and thereby lessen the suffering caused by blindness
and vision loss.

5. Characteristics, capabilities, and limitations

5.1. Characteristics and capabilities

5.1.1. Specifications for input data & information on training data

Input data from the user

• Original high-resolution retina images in JPEG, PNG, or TIFF formats. It is

required to use [Retina] AI Software with high quality, in-focus, unmodified
retinal color images (45 degree), which include both the macula and optic
disc, and were captured by a non-mydriatic color retinal camera. Unmodified
means: original exports of the camera, no screenshots or annotated images
One image per eye is sufficient. [Retina] AI Software is not intended to give
results with other images of the retina, other tissue, or random objects.
See part Error! Reference source not found. and following parts for the full
operation instructions.
• The wearing of contact lenses, spectacles or other vision-correcting devices
by the person under examination during image acquisition will negatively
impact device performance or may even inhibit device operation at all.

Data source: Adult diabetes patient

Training data: our software was trained on training data (i.e. retinal images) which
features data labels regarding ethnicity, age, gender and other special categories of
data.

Data source: Commercially available data, where the source warrants both the presence
of required consents and effective de-identification measures to safeguard individual
privacy.

5.1.2. Expected output

A suggestion regarding the presence of certain diabetes-related eye pathologies with an

associated confidence score.
[Retina] AI Software is only designed to detect certain diabetes-related eye pathologies.
It is not intended to detect any other ophthalmic diseases or any other systemic diseases.
Patients should not rely on [Retina] AI Software for detection of any other disease.

2
Therefore, the existence of diseases not screened by [Retina] AI Software cannot be ruled
out by screening. In case the results of [Retina] AI Software are unclear, patients should be
referred to an ophthalmologist.

5.1.3. Accuracy, robustness and performance

Accuracy: [Retina] AI Software has artificial Intelligence diagnostic accuracy with a

sensitivity of >85% and specificity of >82,5%. The use of these metrics (and their reported
levels) are in line with standard market practice and have been checked against fairness
metrics.

Robustness: in order to ensure a robust functioning of our software, we have implemented

measures that can mitigate or deal with foreseeable risks connected to the limitations of
the system (including backups) as well as against malicious actions that may compromise
the security of our AI system.

Performance: the [Retina] AI Software was specifically tested against the following
parameters: [ethnicity, age, gender]. No statistical relevant outliers were detected.
Not respecting these instructions for use will have an impact on the specified levels for
accuracy, robustness and performance. Further information can be found in parts 5.1.1,
5.1.4- 5.1.6.

5.1.4. Installation and use

System requirements
In order to guarantee optimal performance, the user should ensure that the server on
which the software is being installed, meets or exceeds the following minimum system
requirements:
• Operating System: supporting docker
• RAM: 16GB minimum
• Hard Drive: 100GB free space
• Internet Connection: broadband

For ophthalmologists, healthcare providers or technical personnel:

• Use the web portal via the official website.
• Follow the manual or on-screen instructions to complete the image analysis/
screening.
• Register the software using the license key provided.
• Access the platform via the recommended and compatible browsers:
Chrome, Firefox, Safari, Edge (latest versions)
• Register on our web portal using the credentials provided by your institution.
• Retina Camera: consult the list of compatible devices on our website.

For patients:
• No software installation is required for patients.
• Create your account in order to access the results of retina images.
• Access the result of retina images via the platform by using your credentials.

3
5.1.5. Image Capture Protocol

For ophthalmologists, healthcare providers, nurses:

• Verify if the set-up is used in a sufficiently dark/dimly lit room without any
environmental interferences (such as flashing lights, strong vibrations or
sudden changes in temperature)
• Ensure the retina camera lens is clean.
• Adjust the patient’s eye to the camera’s focus point.
• Capture a high-resolution image.
• Save images in recommended formats: JPEG, PNG, TIFF.

5.1.6. Uploading and Analysis

For ophthalmologists, healthcare providers, nurses:

• Login to the web platform.
• Click “Start screening”
• Select the saved retina images from your system.
• Submit the images. The user is responsible for ensuring that the images
submitted (input) to [Retina] AI Software for a patient are correct and
correspond to that patient in order to avoid mistaken identity with respect to
[Retina] AI Software results.
• The platform will automatically begin the analysis.
• Await results – typically available within seconds.

5.1.7. Interpretation of results and related actions

For ophthalmologists, healthcare providers, nurses:

• The results page will indicate potential presence of diseases.
• A confidence percentage accompanies each diagnosis.
• Results are suggestions; clinical verification (from an ophthalmologist) is
required.
• Patients who are subject of a verified result indicating referable
diabetes-related eye pathologies should be immediately referred to an
ophthalmologist for further diagnosis and treatment.
• Patients who are subject of a verified result indicating non-referable
diabetes-related eye pathologies should be strongly encouraged to test
again in a time period according to the guidelines applicable in his/her
respective country

5.1.8. Retention periods

Uploaded retina images and related analyses are retained for a period of [‘5 years after
which the data is archived for 30 years to meet medical liability retention requirements],
after which they are automatically deleted from our servers. Data that is not being used
is being immediately deleted in a secure manner only by the ophthalmologists after
assessment.

4
5.1.9. Cybersecurity, safety and best-practice tips

Our system offers an appropriate level of cybersecurity in line with market practices. To
help sustain this level as a user, consider the following measures or circumstances when
using our product:

• Always prioritize patient’s safety and data protection.

• Always keep software updated for optimal results and to take advantage of
the latest cybersecurity updates.
• Use only for its intended purpose and within the scope of your medical
expertise.
• To prevent unauthorized access to the patient data on [Retina] AI Software
(data input and results), it is strongly recommended that the computer/server
on which [Retina] AI Software is used/installed is password-protected (using
a strong password); free of viruses and malware; with anti-virus software
and firewall installed and activated; and updated with the latest security
patches.
• All patient data and images are encrypted (in transit and in rest) and stored
securely.
• Implement pseudonymisation or anonymisation measures for retina images
prior upload.

5.2. Limitations/ Risks to health and safety

[Retina] AI Software is not intended to be used on images from:

• patients with prosthetic eyes.
• patients not diagnosed with diabetes.
• patients with a history of retinal laser treatment or injections in either eye, or
history of retinal surgery.
• patients contraindicated for imaging with the fundus imaging system used to
obtain retinal images.

6. Human oversight: Organizational and technical

measures

Our system is not solely reliant on AI. Human oversight is crucial:

6.1. Organizational Measures

• If the patient reports sight problems, a negative screening result should not
be the basis for not referring the patient to a qualified physician.
• Patients should be advised to immediately report to an ophthalmologist if
he/she experiences vision loss, blurred vision, floaters or any other symptom
as these symptoms require the immediate attention of an ophthalmologist.

5
 • [Retina] AI Software has limitations that the user should be aware of [Retina]
AI Software performs computerized interpretation of retinal images therefore
[Retina] AI Software will miss diabetes-related eye pathologies in some cases
(false negatives), and erroneously flag diabetes-related eye pathologies in
others where there is none (false positives).
• Analysis results generated by our software are suggestions and need to be
examined and confirmed by an ophthalmologist. Ophthalmologists and
healthcare professionals are required to use professional judgment and
conventional diagnostic methods.
• Ophthalmologists and healthcare professionals using our system are
expected to undergo further professional training and education.
• In case any serious incident occurs in relation to [Retina] AI Software, this
should be reported to your [Retina] AI Software representative (or info@
company.com) and the competent authority of your country.

6.2. Technical Measures

• Each diagnostic suggestion features an associated confidence score. This

confidence score can be an indication for the user of how much caution
must be applied in regard to the output. If the system is unable to generate
an output with a confidence score above 50%, it will provide a signal to the
user and refrain from providing a diagnostic suggestion. It will then provide
suggestions regarding how to improve the confidence score (e.g. verifying
whether the instructions listed in parts 5.1.4 - 5.1.6 are applied correctly)
• If [Retina] AI Software is not able to generate a result on a patient due to
the poor quality of the images, the patient may be retested immediately
after pharmacologic dilation. If dilation is not possible or if [Retina] AI
Software still does not generate a result, the patient should be referred to an
ophthalmologist for evaluation since the patient may have vision threatening
diabetic retinopathy, or other abnormalities including cataract.
• If a user considers to overrule the result generated by [Retina] AI Software, we
advise to follow our overruling policy which lists various factors that may be
considered when making a decision on overruling or subsequent actions. This
overruling policy can be found on your online account.

7. Maintenance and care instructions

7.1. Expected lifetime

As our software is not integrated into a physical product, we do not foresee a determinate
lifetime but intend to maintain our software as long as possible and commercially feasible.

6
7.2. Maintenance and care measures

In order to maintain our system, ophthalmologists, healthcare providers, nurses shall

ensure that:
• the retina cameras are calibrated and cleaned regularly in order to enable
accurate image capture.
• the web portal is accessed using updated browsers for optimal performance.
• the tool is updated to the latest platform version to for improved accuracy
and additional features.

We shall ensure:
• to update this IFU in accordance with new developments and releases or
added functionalities
• to perform regular performance tests
• to provide regular back-ups and software updates to fix any bugs.
• […]

8. Troubleshooting

• Software Not Responding: Restart your computer and relaunch the

application.
• Integration Issues: Ensure that your system is compatible and updated.
• Uploading Issues: Check your internet connection and file format.
• Analysis Errors: Ensure the image is of high quality and properly centred.
• Platform Issues: Clear browser cache or try a different browser.
• Error Messages: Refer to our online knowledge base or contact our support
team.

9. Logs

All activities, such as uploads, analysis requests, and result retrievals, are logged for
security and improvement purposes. These logs are retained on our platform as long as
required by the applicable limitation period and only for the purposes indicated in this
IFU.

You can access these logs by visiting the ‘log history’ tab in your profile on our web
platform. There you can also find guidance on how to read and interpret the logs.

7
10. Reference to other legal instruments

10.1. Other relevant policies and documents

• Privacy policy (incl. data subject rights). Please take into account that
patients have the following rights under data protection legislation:
obtain information, access, rectification, erasure, restriction or objection of
processing of their data, data portability, object to automated individual
decision-making, including profiling. If they wish to exercise these rights,
contact our Data Protection Officer at [email protected].
• Corporate Code of conduct
• […]

10.2. ISO Standards and Best Practices for Medical Devices

Our software complies with the following standards

• ISO 13485, Medical devices — Quality management systems — Requirements
for regulatory purposes
• ISO 14971, Medical devices — Application of risk management to medical
devices
• IEC/TR 80002-1, Medical device software — Part 1: Guidance on the
application of ISO 14971 to medical device software
• IEC 62304/AMD1, Medical device software – Software life cycle processes
• IEC 82304-1, Health software – Part 1: General requirements for product
safety
• WHO/ITU: Good practices for health applications of machine learning:
Considerations for manufacturers and regulators

11. Support and contact details

For any queries, issues, or support:

• Customer Support: +32 XXX.XX.XX.XX
• Email Assistance: [email protected]
• Online Portal: www.company.com/support

8
INSTRUCTIONS FOR USE 2

Use Case : Medical device for cardiac

arrhythmias prediction
INSTRUCTIONS FOR USE 2

Contents
1. Identity & contact details 1
2. Product Description: intended purposes & intended use 1
2.1. Intended purpose 1
2.2. Intended use 1
2.3. Accuracy & performance specifications 1
3. Operating instructions 3
3.1. Installation and setup 3
3.2. Interpretation of results 4
4. Compatibility 5
5. Usage 5
5.1. Limitations 5
5.2. Target population 6
5.3. Overruling procedure 6
5.4. Recommendations 6
6. Human oversight 6
6.1. Purpose 6
6.2. Feedback collection 6
6.3. Interpretation of the results 6
6.4. Emergency procedures 7
7. Security measures 7
7.1. Warnings 7
7.2. Organisational measures 7
8. Expected lifetime & Foreseeable changes 8
8.1. Expected lifetime 8
8.2. Technical measures 8
8.3. Maintenance 8
9. Logs 8
10. Support and supplier information 9
11. Design of the prototype 10
1. Identity & contact details

• Provider:
• Product:
• Address:
• Email address:
• Phone:

2. Product Description: intended purposes & intended use

2.1. Intended purpose

• Intended purpose: The [Name Tool] solution predicts the onset of atrial
fibrillation before it manifests and this up to one year upfront. The output is
a probability score of atrial fibrillation manifestation for the coming year. A
threshold is placed on the output to give a recommendation to the medical
practitioner. Algorithms are applied to an ECG measurement.
• Benefits for user: Using [Name Tool], atrial fibrillation can be predicted
upfront. This enables to detection of high-risk patients more accurately
and objectively. This way, monitoring can be done efficiently on the better-
identified high-risk group.

2.2. Intended use

• The intended use is for adults with known or suspected heart conditions,
adults feeling symptoms, adults undergoing a standard ECG examination
before surgery, adults referred by their GP or cardiologist, and athletes
performing intensive physical exertion. Output results of [Name Tool] are to
be interpreted by a general practitioner or cardiologist.
• Patients at increased risk of atrial fibrillation (VKF) (based on symptoms &
EKG) so that a higher risk of atrial fibrillation can be detected so they can be
monitored (by cardiologists)

2.3. Accuracy & performance specifications (characteristics, capabilities,

limitations)

• Accuracy: [Name Tool] has undergone rigorous evaluation to ensure its

accuracy and reliability. The following performance metrics are derived
from an independent test set comprising x adults, and they highlight the
application’s effectiveness.
• Performance metrics:
• Area Under the Curve (AUC):
• [Name Tool] achieved an AUC of 0.9678, indicating its
exceptional ability to discriminate between health
conditions.
• Confidence Interval: 95% CI [0.9545; 0.9712]

1
 • Overall Accuracy:
• The application demonstrated an accuracy rate of 99.56%,
reflecting its proficiency in correctly identifying health-
related information.
• Confidence Interval: 95% CI [98.76; 99.86]
• Age and Gender Inclusivity: [Name Tool] was tested on a diverse
population, including individuals across various age groups and
genders. Please refer to Table X for specific age range and gender
distribution details, highlighting our commitment to serving a broad
demographic.

• Threshold-Based Analysis: When applying the recommended threshold

of 0.76 to [Name Tool]’s output, the following results were observed:
• Sensitivity:
• Sensitivity measures the application’s ability to detect true
positive cases.
• [Name Tool] achieved a sensitivity rate of 95.56%, ensuring
effective identification of individuals with health concerns.
• Confidence Interval: 95% CI [94.87; 96.89]

• Specificity:
• Specificity assesses the application’s capacity to correctly
identify true negative cases.
• [Name Tool] demonstrated a specificity rate of 84.56%,
indicating its ability to distinguish individuals without health
issues.
• Confidence Interval: 95% CI [83.45; 86.54]
• Patient Characteristics: For a comprehensive understanding of
the evaluation, please consult Table X for details regarding the
characteristics of patients in both the test set and the development
data set.

• Dataset Characteristics:
• The test dataset predominantly represents individuals of
Caucasian ethnicity and middle age.
• ECG measures were meticulously collected using two ECG
devices, namely X and X, to ensure comprehensive data
coverage.
• The prevalence of atrial fibrillation within the test set was XX%,
reflecting the real-world distribution of this health condition.
• Data collection took place across multiple reputable hospitals,
including XXXX, ensuring a wide geographic and medical context.

• Error Rate and Performance Metrics: [Name Tool]’s performance

metrics, including threshold-based error rates and sensitivity, were
meticulously determined on the test dataset, accounting for the diverse
characteristics mentioned above.
• Threshold: The threshold value used for health assessment is
optimized to ensure precision and reliability. Please refer to the
[Name Tool] User Manual for specific threshold details.

2
 • Sensitivity: Sensitivity measures [Name Tool]’s capacity to detect
true positive cases within the test dataset. Our application
achieved remarkable sensitivity, indicative of its ability to identify
individuals with health concerns accurately.
• Specificity: Specificity assesses [Name Tool]’s capability
to correctly identify true negative cases. Our application
demonstrated strong specificity, ensuring accurate identification
of individuals without health issues.

• Comprehensive Dataset Overview: For a detailed understanding of

the dataset characteristics, including ethnicity, prevalence of atrial
fibrillation, age distribution, and ECG device usage, please refer to the
provided dataset documentation.

• Robustness:
• Reproducibility tested by repeated measurements on the same patient
and different ECG devices
• The reproducibility of the prediction algorithms was tested using a test-
retest setup in three independent hospitals that are geographically
distributed. Variations in the ECG measurement device were also
measured. Test-retest reliability measured was XXX.

• Cybersecurity: [Name Tool] uses industry-standard encryption protocols

to secure all products and data in transfer and at rest. FibriCheck uses
industry best practices. Detailed measurement data is only stored on secure,
encrypted servers and not on the device of the user. These databases
contain encrypted user data when at rest or in transit. A continuous backup
system is used to make sure all data is secure.

• Be aware that results from [Name Tool] may differ depending on the
ECG measurement device, whether sports were executed before ECG
examination, etc.

3. Operating instructions

3.1. Installation and setup

• System Requirements:
• Before you begin, make sure your system meets the following
requirements:
• Operating System: Windows 10, macOS, or Linux
• Processor: Dual-core processor or higher
• RAM: 4GB or more
• Storage: 500MB of free disk space
• Internet Connection: Required for initial setup and updates

3
 • Installation steps:
• Download the Application:
• Visit our official website (www.[Name Tool].com) and navigate to
the “Downloads” section. Choose the version of the application
that corresponds to your operating system (Windows, macOS, or
Linux) and click the “Download” button.
• Install the Application:
• Once the download is complete, locate the installation file (e.g.,
“Heart_Felt_App.exe” for Windows) and double-click it to start the
installation process. Follow the on-screen instructions to install the
application.
• Launch the Application:
• After installation, you can find the Heart Felt Application in
your system’s applications or programs folder. Double-click the
application icon to launch it.
• Create or Log in to Your Account:
• Upon launching the application for the first time, you will be
prompted to create a new account or log in with your existing
credentials if you already have an account. This account will
allow you to manage patient data and access predictions.
• Connect External Devices:
• Ensure your ECG machine is connected to the computer and
configured according to their respective manuals. The Heart
Felt Application requires access to these devices for accurate
predictions.
• Patient Data Input:
• Follow FibriCheck instructions to connect and retrieve the data.
• Data Processing:
• The application will process the entered patient data and the
ECG data from the FibriCheck application. It will then generate a
prediction regarding the likelihood of atrial fibrillation.
• Save and Export Data:
• The application allows you to save patient profiles, predictions,
and associated data for future reference. You can also export this
data in various formats (CSV, PDF, etc.) for integration into your
clinic’s electronic health records system.
• Support and Updates:
• For technical support, bug reports, or feature requests, please
contact our support team at support@[Name Tool].com. We
periodically release updates to improve the application’s
performance and accuracy. Ensure you keep the application up
to date by checking for updates in the “Settings” section.

3.2. Interpretation of results

Results are presented in the form of a probability score, categorized into high, moderate,
and low-risk, to assist medical practitioners in making informed decisions.

4
 • Understanding Risk Categories:
• High-Risk: A high-risk prediction indicates a significant likelihood of
atrial fibrillation manifestation. This category is essential for identifying
patients who may require immediate attention and intensive
monitoring.
• Moderate-Risk: A moderate-risk prediction suggests a moderate
probability of atrial fibrillation development. While not as urgent as the
high-risk category, patients in this group may still benefit from regular
monitoring and further evaluation.
• Low-Risk: A low-risk prediction signifies a minimal likelihood of atrial
fibrillation occurrence in the near future. Patients falling into this
category generally have a lower priority for immediate intervention, but
routine check-ups are advisable for ongoing assessment.

4. Compatibility

The Heart Felt application is compatible with the following ECG device models
• GE Healthcare:
• GE MAC 5500 HD
• GE MAC 800
• GE CASE 6.7

• Philips Healthcare:
• Philips PageWriter TC70/TC50/TC30
• Philips Efficia DFM100
• Philips Cardiograph 12

5. Usage

5.1. Limitations

• [Name Tool] is only intended for cardiologists or another medical practitioner

working in a cardiology department.
• The accuracy of the AFib prediction heavily relies on the quality and
accuracy of the input data, the ECG data. If the ECG data is noisy,
improperly calibrated, or affected by artefacts, it can lead to inaccurate
predictions.

5
5.2. Target population: Caucasian, middle-aged, not already diagnosed
with atrial fibrillation before.

5.3. Overruling procedure

In certain situations, the predictions made by the AI application may need to be

overruled based on clinical judgment, patient condition, or other relevant factors. While
the AI algorithm is designed to provide valuable insights, healthcare professionals must
retain the ultimate authority in decision-making.

5.4. Recommendations

• Always ensure the safety and well-being of your patients while using the AI
application.
• Use the AI application in accordance with your clinical expertise and within
the scope of your medical knowledge. While the AI provides predictions,
always rely on your professional judgment and conventional diagnostic
methods to make informed decisions about patient care.
• In order to ensure that patient data and results are not accessed by
unauthorized individuals, it is highly advisable to secure the computer
or device used to access the AI application with a robust password. It’s
essential to confirm that the system is devoid of viruses and malware and
that it’s fortified with up-to-date anti-virus software and an active firewall.
Regularly updating the system with the latest security patches is also
strongly recommended.

6. Human oversight

6.1. Purpose

Human oversight within the [Name Tool] AI application plays a pivotal role in ensuring
the highest standards of safety, accuracy, and reliability. It is integral to various aspects
of the system, including monitoring and evaluation, quality control, error handling, and
emergency procedures.

6.2. Feedback collection

The medical practitioner can provide feedback on the presented results by

• Rating the results
• Open text field to provide suggestions

6.3. Interpretation of the results

An ECG risk score is used to determine the risk of AF. There are three possible outcomes:

1. 0–2: Low probability of AF.

2. 3–4: Intermediate probability of AF.
3. 5–6: High probability of AF.
6
The score is comprised of three P-wave variables: morphology in inferior leads, voltage
in lead 1, and P-wave duration (MVP). The variables and point allocation of the MVP ECG
risk score are based on a critical appraisal of the existing literature and the prevalence
and predictive value of ECG indices for AF (Alexander, MacHaalany, et al., 2017; Alexander
et al., 2016; O’Neal, Zhang, et al., 2016; Tse et al., 2017). Table 1 illustrates the MVP ECG risk
score variables and the assigned weighting.

Variable Value Score

Morphology in inferior leads Nonbiphasic (<120 ms) 0
Nonbiphasic (≥120 ms) 1
Biphasic 2
Voltage in lead I >0.20 mV 0
0.10–0.20 mV 1
<0.10 mV 2
P-wave duration <120 ms 0
120–140 ms 1
>140 ms 2

6.4. Emergency procedures

Healthcare professionals are equipped to take immediate action when necessary,

including bypassing AI recommendations for urgent interventions.

7. Security measures

7.1. Warnings

• Utilize solely for its designated purpose and within the scope of your medical
proficiency

7.2. Organisational measures

• Continuous training and education: Healthcare professionals utilizing our AI

application must undergo regular training and education to stay updated on
the latest advancements in arrhythmia prediction and the utilization of the AI
algorithm.
• Algorithm performance monitoring: The [Name Tool] algorithm will be
regularly monitored to detect deviations or inaccuracies. Protocols have
been developed for tracking outcomes over time to identify areas for
improvement and recalibration.
• Data security and privacy: rigorous measures are in place to protect patient
data. This includes adhering to data protection regulations, anonymizing
patient information, and ensuring secure storage and transmission of
sensitive patient data.

7
 • Reporting incidents and feedback: Any significant adverse events or
incidents linked to the AI application’s predictions must be promptly reported
to both the AI application provider (support@[Name Tool].com) and the
appropriate regulatory authorities in your country.

8. Expected lifetime & Foreseeable changes

8.1. Expected lifetime

[Name Tool] has been designed to provide valuable functionality and assistance for a
defined period of time. Understanding the expected lifespan of the system is crucial for
optimal utilization and performance.

• Expected Lifetime Duration: The expected lifetime of [Name Tool] is

estimated to be X, starting from the date of initial activation or installation.
This estimate is based on factors such as technological advancements,
software updates and the evolving nature of AI technology

8.2. Technical measures

• The AI undergoes routine updates and adjustments, taking into account

input from users
• Data is consistently backed up on a regular basis.
• Unused data is promptly and securely removed.

8.3. Maintenance

• Healthcare professionals using the AI have to:

• Perform routine system checks to verify that their devices and network
connections are functioning correctly. This includes assessing internet
connectivity, hardware performance, and any potential technical issues
that might impact the accuracy of the AI predictions.
• Access the AI application using up-to-date and supported web
browsers to ensure optimal performance and security.
• Keep the AI algorithm up-to-date with the latest version provided by
the application provider. Updates are essential for improving prediction
accuracy, incorporating new medical insights, and adding additional
features that enhance the diagnostic capabilities of the algorithm.

9. Logs

Our AI application may generate logs that capture certain activities and interactions
for the purpose of security, improving user experience, troubleshooting issues, and
enhancing the performance of the application. These logs may include:
• User Interactions: Details of user input and interactions with the AI
application, such as queries, commands, and preferences.
• Technical Information: Information related to the device, operating system,
browser, and network used to access the application.
8
 • Errors and Exceptions: Records of errors, crashes, and exceptions
encountered during the use of the application.
• Performance Metrics: Metrics regarding the application’s responsiveness,
processing times, and resource utilization.
• These logs are primarily used for internal analysis and diagnostic purposes.
We take your privacy and data security seriously, and the information
collected is handled in accordance with our Privacy Policy (add link). Personal
or sensitive information is not intentionally collected or stored within these
logs.

How We Use Logs:

• Improvement: Analyzing logs helps us identify areas for improvement and
refine the AI algorithms to enhance overall performance.
• Bug Resolution: Logs assist our technical team in diagnosing and resolving
any issues that may arise during the use of the application.
• Security: Logs aid in detecting and responding to potential security threats
and anomalies.

We do not share individual log data with third parties, except as required by law or as
outlined in our Privacy Policy.

If you have concerns about the data captured in logs, you can contact our Support Team
for more information.

10. Support and supplier information

If you have any questions, or concerns, or need assistance while using our AI application,
our dedicated support team is here to help. We are committed to ensuring a seamless
experience for our users, and we encourage you to reach out to us with any inquiries you
may have.

Contact Information:
• Email: support@[Name Tool].com
• Phone: +32-XXX-XXXXXXX
• Live Chat: Available on our website during business hours.

Our support team is available from 9:00 AM to 6:00 PM (CMT) Monday through Friday. We
strive to respond to all inquiries within 24 hours.

For general information, updates, and announcements, you can also follow us on social
media:
• LinkedIn: @[Name Tool]

We value your feedback and suggestions as they help us improve our product to meet
your needs better. Thank you for choosing [Name Tool], and we look forward to assisting
you.

Sincerely, The [Name Tool] Support Team

9
11. Design of the prototype

Description of the design of Instructions for Use for the [Name Tool] application:

Feedback Loop: Our digital Instructions for Use (IFU) incorporate a dynamic feedback
loop, e.g. using a permant chat window similar to a chatbot allowing users to provide
instant feedback on the content's clarity and usefulness. This real-time input ensures
continuous improvement.

Heat Map Methods: We have integrated heat map methods to visualize user interactions.
Through intuitive color-coded heat maps, we identify which sections users engage with
the most. This data-driven approach guides content optimization, ensuring that the most
crucial information is readily accessible and easy to comprehend.

Placeholder for Technical Details: To accommodate the evolving nature of technology,

we have incorporated placeholders within the document. These placeholders are
designated sections where the data science team can seamlessly integrate technical
details. This flexibility ensures that the IFU always reflects the latest advancements and
accurate technical specifications.

Drop-Down Menu & Linking: A user-friendly drop-down menu simplifies navigation. Users
can effortlessly access specific sections of interest. Additionally, strategic linking within
the document enables seamless transitions between related topics. This interconnected
structure provides a cohesive user experience, allowing users to explore relevant content
effortlessly.

Concise Language Use: We employ concise language throughout the document, ensuring
clarity and brevity. Complex technical concepts are explained in simple terms, enhancing
user understanding. By eliminating jargon and unnecessary details, the IFU communicates
information effectively, catering to users with varying levels of expertise.

Referencing and Cross-Linking: The digital IFU includes references and cross-links to
external documents, providing users with additional resources for in-depth exploration.
Proper citation and linking enhance the credibility of the content, allowing users to
validate information and delve deeper into specific topics if desired.

---

This design ensures that the digital Instructions for Use of the AI application are not
only comprehensive and informative but also user-friendly and adaptable to ongoing
technological developments.

10
INSTRUCTIONS FOR USE 3

Use Case : HR talent matching tool

INSTRUCTIONS FOR USE 3

Contents
1. Who + Contact Details 1
2. Intended Purpose for the User / Human Role 1
3. Data Transparency 1
4. Time period of data being used to develop the AI system 2
5. Accuracy/robustness/cybersecurity 3
6. Oversight + Logs keeping for users 4
7. Performance 5
8. Worst Case Scenario + Solution 5
9. Evolution and Expected life time of AI System / ... 7
10. FAQ 8
1. Who + Contact Details

• COMPANY

2. Intended Purpose for the User / Human Role

Our primary objective is to facilitate a seamless and highly effective job matching process
by employing cutting-edge AI technologies. Specifically, our platform is designed to
empower users by intelligently connecting candidates to job offers in the most relevant
and personalized manner possible.

• Geographical scope: The geographical setting stipulates that the [Name API
Tool] is designated for the purpose of matching candidates with job offers
within the jurisdiction of the Belgian job market.

• Age Scope: The matching functionality is anticipated to perform effectively

for applicants satisfying the minimum legal age of work on the Belgian job
Market.

• Ethnicity scope: The [Name API Tool] is deemed secure for utilization in
matching applicants of any ethnicity.

• Gender Inclusivity: The [Name API Tool] is designed to be safely employed

for matching applicants of any gender.

3. Data Transparency

• Data used: format, attributes, origin: The information employed is procured

directly from the client, who submits data to the API. Our artificial intelligence
model incorporates diverse parameters to facilitate the efficient matching of
candidates to job offers, encompassing:

• For Vacancies:
• Job Title
• Employer Address
• Job Description
• Expected Skills
• Required Job Experience
• …

• For Candidates:
• Approximate Address
• Reported Skills
• Job Experience derived from Curriculum Vitae
• Self-Declared Interests
Collectively, these parameters serve to augment the precision of our
matching algorithms, thereby guaranteeing a comprehensive and
accurate correlation between candidate profiles and job opportunities.
1
 • Data not used: Attributes that might potentially give rise to discriminatory
practices in the job market, including gender, age, or ethnicity, are
intentionally excluded from consideration in the process of matching
applicants with job opportunities. Nevertheless, a robust testing regimen
has been implemented to thoroughly evaluate and address potential biases
specifically related to gender and age within the overall job-applicant
matching process. It is noteworthy to mention that ethnicity data is not
directly accessible, even to clients.

• Data pre-processing (bias reduction measures, handling incomplete data,

handling obvious bad data; e.g. spelling corrections): Prior to employing
the matching model, a series of pre-processing tasks are systematically
conducted. These tasks are delineated as follows:

• For Candidate Data: Extraction of standardized titles and skills from the
Curriculum Vitae (CV), ensuring a uniform representation of candidate
attributes.

• For Job Data: Extraction of skills from the job description, establishing
a comprehensive understanding of the requisite qualifications and
competencies.

• For Both Candidate and Job Data: Geolocation based on addresses

is performed, a critical factor in calculating the distance between the
candidate's residence and the job location, thereby enhancing the
contextual relevance of the match.

Semantic embedding of titles and skills mentioned in both the job

offer and candidate CV is undertaken. This process involves encoding
these textual elements into a meaningful numerical representation,
contributing to a more nuanced analysis and facilitating a
sophisticated matching process.

4. Time period of data being used to develop the AI

system

The temporal scope of data utilized for the development of the AI system spans from
[start date] to [end date].

• Data size and representativity

• The system's training dataset comprises a set of [number] job offers
and [number] candidate CVs, meticulously curated to encompass
dimensions frequently implicated in discriminatory contexts, including
gender, age, and origin. This comprehensive approach ensures a
representative dataset that reflects diverse attributes.
• The data is strategically employed in the following ways:
• Utilization of skills mentioned in CVs as a crucial factor in the
matching process.

2
 • Incorporation of [specific aspect] in CVs for a more nuanced
understanding.
• The matching process involves the consideration of four
dimensions, namely [dimension 1], [dimension 2], [dimension 3],
and [dimension 4], ensuring a multifaceted analysis for optimal
outcomes.

• Common definitions
• To enhance clarity and consistency, common definitions are established
to provide clear interpretations of technical terms and jargon used
within the system:

5. Accuracy/robustness/cybersecurity

• Accuracy:
• [Name API Tool] functions by providing job recommendations to
applicants based on their demonstrated interest, inferred from their
engagement with posted jobs. The accuracy and recall metrics
are subsequently assessed by examining the extent to which our
recommendations align with the applicants' actual selections.
Additionally, scrutiny is applied to clicks made by recruiters and
employers on applicant profiles.
• Accuracy is gauged within the context of the first-N recommended
jobs.

• Robustness:
• The robustness of our model is rigorously assessed through various
examinations. For instance, the skill/title embedder undergoes
meticulous checks for resilience against spelling variations, diverse
language usage, and instances such as "nurse" or "male nurse,"
demonstrating our commitment to accommodating nuanced
differences.

• Security:
• Security measures are integral to [Name API Tool]'s deployment on
a secure cloud platform. Access to critical cloud resources, including
client databases and the source code, is meticulously controlled and
limited solely to designated members of the [Name API Tool] technical
team.
• The external-facing components consist of the API itself and a singular
Bastion host, exclusively accepting SSH connections from authorized
[Name API Tool] developers.
• Access to the API is facilitated through a robust system of
authentication and authorization, anchored in client-specific
usernames and passwords, ensuring a controlled and secure
environment wherein each client possesses both admin and regular
user identities.

3
6. Oversight + Logs keeping for users

• Technical Measures to Enhance User Comprehension:

• Job Match Explanation: Our system provides a detailed breakdown
of the job match, elucidating the specific criteria that led to the
recommendation. This encompasses a thorough explanation of how
the candidate's attributes align with the job requirements.
• Matching on Title: When the system matches based on job titles, users
receive a clear explanation detailing the relevance of the title to the
job opportunity. For instance, if a job seeks a "Project Manager," the
user is informed that the match is made due to the alignment of the
candidate's title with the job requirements.
• Skills Matching Explanation: For skills-based matches, a detailed
account is provided, outlining the specific skills the job demands and
the corresponding skills present in the candidate's profile. This ensures
transparency and enables users to comprehend the basis for the
match.
• Expertise Level Clarification: When expertise level is a contributing
factor, the system articulates the proficiency level required by the job
and explains how the candidate's demonstrated expertise meets
or exceeds those expectations. This empowers users to gauge the
suitability of the match with a nuanced understanding of expertise
levels.

Example Scenario:
Consider a job that necessitates expertise with a forklift.
The system would explicitly convey to the user that the match is
primarily attributed to the candidate's experience with a forklift,
providing a tangible example of how specific attributes contribute
significantly to the job recommendations.

Example Airbnb

These measures collectively enhance the transparency of our system,

enabling users to not only receive job recommendations but also
comprehend the rationale behind each recommendation, fostering a
more informed and empowered user experience.

• Identifying Anomalies:
• Imagine our system as a diligent assistant always working to connect
you with the most fitting job opportunities. To make sure everything
runs smoothly, we've set up ways to spot any unexpected or irregular
situations—those are what we call "anomalies." In simple terms,
anomalies are like red flags that tell us if something unusual or not
quite right is happening in the job matching process. When you see
something that doesn't seem to match your expectations, it could be
one of these anomalies, and we'll be on top of it to make things right.

4
 • Human-Machine Interaction (Premarket):
• Before our system goes into action and starts suggesting jobs to you,
we've put in place a way for humans and machines to work together,
sort of like a team preparing for a big game. This happens before
everything gets out into the public, in what we call the "premarket"
phase. During this time, we set up the rules and strategies to ensure the
best possible job matches. While we're not actively taking feedback
from users during this phase, we've carefully considered how the system
should operate based on what we know about your needs. So, just like
a team getting ready for a game, we've done the preparation work
to make sure that when our system starts suggesting jobs to you, it's
well-prepared and ready to provide the most relevant and beneficial
recommendations.

7. Performance

• Expectations for Lower Performance:

• Consider our [Name API Tool] as your personal job matching advisor.
While it usually does a fantastic job, there are certain situations where
its performance might not be as accurate. For instance, if a CV is on the
shorter side or uses specific words that the system finds challenging to
connect, it might lead to slightly less accurate recommendations.

• Specific Scenario:
• We've observed that our system might offer lower-quality
recommendations for applicants falling under a specific situation,
described as follows: <...>

• Human Oversight and Final Decision:

• In such situations, it's crucial to emphasize that our system is here to
assist and enhance the job matching process, but the ultimate decision
lies with you, the user. Human oversight is key, and we encourage
users to carefully review recommendations, especially in cases
where the system might have challenges, such as ethnicity-related
considerations. Your judgment and insights are invaluable, and we
believe that your final decision should align with your preferences and
expectations.`

8. Worst Case Scenario + Solution

Worst Case Scenario 1: Job Has No Matching Applicant

In a rare scenario, our [Name API Tool] might encounter a challenge where it fails to find
suitable matches for a particular job, leaving the user with no potential candidates. This
could be due to various factors, such as a highly specialized job role or specific criteria
that are uncommon in the candidate pool.

5
 • Tackling the Problem:
• User Notification: Our system will promptly notify the user that no
matching candidates were found for the given job. This transparency is
crucial in keeping users informed about the outcome.
• Analysis of Criteria: Our technical team will conduct an in-depth
analysis of the job requirements to understand the specific criteria
that led to the absence of matches. This involves assessing whether
the criteria are too niche or if adjustments can be made for better
alignment.
• User Consultation: A consultation with the user, especially if they are
experts like public employment service recruiters, will be initiated to
gather additional insights. Understanding the intricacies of the job and
its requirements from the user's perspective is invaluable in refining the
matching process.
• System Optimization: Based on user feedback and technical analysis,
the system will undergo optimizations. This may involve fine-tuning
algorithms, revisiting pre-processing tasks, or adjusting parameters to
ensure a more effective matching process in the future.
• Continuous Improvement: The incident will be treated as a learning
opportunity, and measures will be implemented to prevent a
recurrence. Our commitment to continuous improvement ensures
that the system evolves to handle a diverse range of job scenarios
effectively.
• User Empowerment: In line with our philosophy of user empowerment,
the final decision always rests with the user. Even in situations where the
system encounters challenges, we believe that the user's expertise and
judgment are paramount. Users are encouraged to actively engage
with the system, provide feedback, and make informed decisions based
on their unique insights and requirements. This user-centric approach
ensures that our system not only meets but exceeds user expectations
over time.

Worst Case Scenario 2: Applicant has No Matching Jobs

In rare instances, our [Name API Tool] may encounter a scenario where it fails to
provide suitable job recommendations for a specific applicant. While this occurrence is
infrequent, we want to be transparent about the possibilities.

Typically, our system strives to present job suggestions tailored to individual profiles.
However, there are situations where we might not find an ideal match. In such cases,
we undertake an additional step by pruning jobs that fall below a certain match quality
threshold. This is part of our commitment to offering recommendations that align closely
with your qualifications and preferences.

If, despite our efforts, you find yourself in a situation where no matching jobs are
presented, the most common solution is for you to enhance your profile by providing
additional details. By filling in more information about your skills, experiences, and
preferences, you increase the chances of our system making more accurate and
personalized recommendations.

6
However, if after profile enhancement the situation persists, it may be indicative of
a genuine lack of available jobs in our database that precisely match your unique
combination of skills and experiences. In these cases, we recommend keeping your
profile updated and checking back periodically, as the job market evolves, and new
opportunities may arise over time.

We understand the importance of this process in your job search journey, and your
insights play a pivotal role. The final decision always rests with you, the user. Our system
aims to assist and enhance, but your judgment, preferences, and expectations are
paramount. Should you encounter challenges, we encourage you to carefully review
recommendations and make decisions aligned with your needs and aspirations.

9. Evolution and Expected life time of AI System/

Maintenance and Care

Our commitment to excellence extends beyond the initial deployment of the [Name API
Tool]. We recognize the dynamic nature of technology and the evolving landscape of
job markets. Therefore, we have implemented strategies for the continuous evolution and
upkeep of our AI system.

• Evolution: The [Name API Tool] is designed to adapt and evolve in response
to changing user needs, technological advancements, and shifts in the
employment landscape. Our development team is dedicated to staying at
the forefront of AI innovations, regularly updating algorithms, and integrating
new features to enhance the overall user experience. These updates are
aimed at improving job matching accuracy, expanding capabilities, and
ensuring that our users benefit from the latest advancements in artificial
intelligence.
• Expected Lifetime: While predicting the exact lifetime of an AI system can
be challenging due to the rapidly evolving nature of technology, we are
committed to providing long-term support for the [Name API Tool]. Our goal
is to ensure its relevance and effectiveness over an extended period. Regular
assessments and updates are scheduled to align with industry trends and
user feedback, ensuring that the system remains a valuable tool for job
seekers and recruiters alike.
• Maintenance and Care: Maintenance is a critical aspect of ensuring the
continued reliability and security of the [Name API Tool]. Our technical
team conducts routine maintenance checks to address any potential
issues, update security protocols, and optimize system performance. These
measures are implemented to guarantee a seamless and secure user
experience.

In addition to technical maintenance, we are attentive to user feedback

and actively seek insights from our community. Your input is invaluable in
identifying areas for improvement, addressing concerns, and shaping the
future development of the [Name API Tool]. We encourage users to share
their experiences, challenges, and suggestions to contribute to the ongoing
refinement of our system.
7
 As we embark on this journey together, rest assured that the [Name API
Tool] is not a static solution. It is a dynamic tool that evolves with the ever-
changing dynamics of the job market and the technological landscape, with
a commitment to serving your needs effectively and efficiently.

10. FAQ

**1. Q: What is the purpose of the [Name API Tool]?**

• A: The primary objective of the [Name API Tool] is to facilitate a seamless
and highly effective job matching process by employing cutting-edge AI
technologies. It intelligently connects candidates to job offers in a relevant
and personalized manner within the Belgian job market.

**2. Q: How is the geographical scope of the [Name API Tool] defined?**
• A: The [Name API Tool] is specifically designed for matching candidates with
job offers within the jurisdiction of the Belgian job market.

**3. Q: Is there an age restriction for using the [Name API Tool]?**
• A: The matching functionality is anticipated to perform effectively for
applicants satisfying the minimum legal age of work on the Belgian job
market.

**4. Q: Can the [Name API Tool] match applicants of any ethnicity?**
• A: Yes, the [Name API Tool] is deemed secure for utilization in matching
applicants of any ethnicity.

**5. Q: Is the [Name API Tool] gender-inclusive?**

• A: Absolutely. The [Name API Tool] is designed to be safely employed for
matching applicants of any gender.

**6. Q: What data is used in the matching process?**

• A: The information used includes parameters such as job title, employer
address, job description, expected skills, required job experience for
vacancies, and approximate address, reported skills, job experience from CV,
and self-declared interests for candidates.

**7. Q: Are attributes like gender, age, or ethnicity used for matching applicants with
jobs?**
• A: No, attributes that could potentially contribute to discrimination, such as
gender, age, or ethnicity, are intentionally excluded from consideration in the
matching process. Rigorous testing has been conducted to address biases
related to gender and age.

**8. Q: How is incomplete or potentially biased data handled in the system?**

• A: Before using the matching model, pre-processing tasks are
performed, including the extraction of standardized titles and skills,
geolocation based on addresses, and semantic embedding of titles
and skills. These measures help in handling incomplete or biased data.

8
**9. Q: What is the expected lifetime of the [Name API Tool], and how often is it
updated?**
• A: While predicting an exact lifetime is challenging, we are committed
to long-term support and regular updates to adapt to technological
advancements, user needs, and changes in the job market.

**10. Q: How can users provide feedback or report anomalies in the system?**
• A: Users can contribute to the evolution of the system by providing feedback
through our platform. If anomalies are identified, we encourage users to
report them, and our team will promptly address and rectify any issues to
ensure a seamless user experience.

**11. Q: What kind of information does the [Name API Tool] use to match candidates with
job offers?**
• A: The [Name API Tool] utilizes a variety of parameters, including job
titles, employer addresses, job descriptions, expected skills, required job
experience for vacancies, and candidate data such as approximate
addresses, reported skills, job experience from CVs, and self-declared
interests.

**12. Q: Can the [Name API Tool] be used by recruiters from the public employment
services?**
• A: Yes, the [Name API Tool] is designed to cater to users with varying levels of
expertise and skills/needs, including public employment service recruiters.

**13. Q: How is user data processed to reduce biases in the system?**

• A: Our system undergoes pre-processing tasks, including the extraction of
standardized titles and skills, geolocation based on addresses, and semantic
embedding of titles and skills. These measures contribute to bias reduction
and enhance the fairness of the matching process.

**14. Q: Is the [Name API Tool] accessible to users outside of Belgium?**

• A: Currently, the [Name API Tool] is designated for matching candidates
with job offers within the Belgian job market, limiting its accessibility to users
outside this geographical scope.

**15. Q: What dimensions are considered for matching candidates and job offers?**
• A: The matching process involves the consideration of four dimensions:
gender, age, origin, and [specific aspect], ensuring a multifaceted analysis
for optimal outcomes.

**16. Q: Can users influence the evolution of the [Name API Tool]?**
• A: Yes, user feedback is highly valuable in shaping the future development
of the [Name API Tool]. Users are encouraged to share their experiences,
challenges, and suggestions to contribute to ongoing improvements.

**17. Q: How is the [Name API Tool]'s accuracy measured?**

• A: The accuracy of the [Name API Tool] is assessed by analyzing how well
its recommendations align with users' actual selections, clicks made by
recruiters and employers on applicant profiles, and the relevance of the
first-N recommended jobs.
9
**18. Q: What happens if I don't receive matching job recommendations?**
• A: In the rare case that you don't receive matching job recommendations,
we recommend enhancing your profile by providing additional details. If the
situation persists, it may indicate a lack of jobs in our database that precisely
match your unique combination of skills and experiences.

**19. Q: How is data security ensured in the [Name API Tool]?**

• A: The [Name API Tool] is deployed on a secure cloud platform, and
access to critical resources is strictly controlled and limited to authorized
technical team members. Security measures, including authentication and
authorization, are in place to create a controlled and secure environment.

**20. Q: What kind of oversight is implemented in the [Name API Tool]?**

• A: Oversight includes ongoing monitoring and logs keeping for users.
Technical measures are in place to enhance user comprehension, explaining
each aspect of the system, such as job match, title and skills matching, and
expertise levels, to make the output understandable for users.

10
DISCLAIMER 1

Use Case: HR Chatbot

DISCLAIMER 1

Use Case Description – background information

This use case concerns a human resources (HR) AI chatbot intended primarily
to communicate with employees (white collar, blue collar) and other people
involved with a particular company (i.e. managers, outsourced staff) in relation to
HR-related queries. Based on a prioritization of ideas and solutions, the Group
developed a scheme/decision tree and disclaimer that could be relied upon (cf.
avatar, interface,…) to implement Article 52 of the Proposal.

Contents
1. Background information regarding article 52 of the proposal 1
2. Prototype User Journey (incl. disclaimer) 3
3. Prototype decision process 4
1. Background information regarding article 52 of the
proposal

According to the explanatory memorandum and recitals of the Proposal, per Article
52 of the Proposal, in essence, (i) natural persons should be notified that they are
interacting with an AI system, unless this is obvious from the circumstances and the
context of use. Also, (ii) natural persons should be notified when they are exposed to an
emotion recognition system or a biometric categorisation system. Such information and
notifications should be provided in accessible formats for persons with disabilities. Finally,
(iii) users, who use an AI system to generate or manipulate image, audio or video content
that appreciably resembles existing persons, places or events and would falsely appear
to a person to be authentic, should disclose that the content has been artificially created
or manipulated by labelling the artificial intelligence output accordingly and disclosing its
artificial origin.

Thus, Article 52 of the Proposal is directed to both providers (situation (i)) and users
(situations (ii) and (iii))1.

The focus of the Group has been on the situation (i) – transparency obligations for
providers of a chatbot, with the main goal of developing disclaimers in accordance with
Article 52 of the Proposal and a related process that allows to decide whether or not to
apply the disclaimer.

In such situation, content of Article 52 of the Proposal can be described as it follows:

• ADDRESSEE: providers
• PROTECTED SUBJECTS: natural persons
• TYPE OF AI SYSTEM: AI system intended to interact with natural persons
(chatbots)
• RULE - REQUIRED ACTION: inform users that they are interacting with an AI
system
• EXECUTION OF ACTION: information must be provided to a natural person
in a clear and distinguishable manner at the latest in the time of the first
interaction of exposure
• GENERAL EXCEPTION FROM THE RULE: interaction between AI system and
users is obvious from the point of view of a natural person who is reasonably
well-informed, observant and circumspect, taking into account the
circumstances and the context of the use
• SPECIFIC EXEMPTION FROM THE RULE: AI systems authorised by law to
detect, prevent, investigate and prosecute criminal offences, subject
to appropriate safeguards for the rights and freedoms of third parties
(exception from exception: unless those systems are available for the public
to report a criminal offence).

Thus, Article 52 of the Proposal applies to AI systems intended to communicate with

natural persons. The rationale of this provision is to ensure that that natural persons are
informed that they are interacting with an AI system, with the exception of AI systems
authorised by law to detect, prevent, investigate and prosecute criminal offences, subject
to appropriate safeguards for the rights and freedoms of third parties.

1 Surely, in some cases, the roles of a provider and a user can be identified.
1
The general rule stemming from Article 52 of the Proposal is that providers must ensure
that the information is delivered to a natural person interacting with their AI system.
In cases where interaction from the AI system is obvious from the point of view of a
natural person who is reasonably well-informed, observant and circumspect, taking into
account the circumstances and the context of the use, the provider of an AI system is not
required to conduct any specific actions with regards to informing the natural person.
In the context of the AI Act, natural person who is reasonably well-informed, observant
and circumspect is a standard that is not explained in the explanatory memorandum or
recitals of the Proposal, but will have to be clarified through future practice and soft law.

From the wording of Article 52 Paragraph 1 of the Proposal, it is reasonable to interpret the
obligation of a provider to ensure an interface of the AI system from which it will in a clear
and distinguishable manner obvious to a natural person who is reasonably well-informed,
observant and circumspect that he/she is interacting with an AI system, thereat taking
into consideration particular circumstances and the context of the use. Otherwise, it
seems evident that the provider must explicitly deliver the information to a natural person
that he/she is communicating with an AI system, also, in a clear and distinguishable
manner at the latest in the time of the first interaction of exposure.

Having all things considered, for AI systems authorised by law to detect, prevent,
investigate and prosecute criminal offences, subject to appropriate safeguards for the
rights and freedoms of third parties, there is no special obligations for providers of AI
system to inform natural persons, unless those systems are available for the public to
report a criminal offence2.

For all other AI systems intended to interact with natural persons, to comply with Article
52 of the Proposal, providers must ensure interface from which the fact that a natural
person is interacting with an AI system would be obvious and/or explicitly provide such an
information. Good practice is most likely to provide such information to a natural person in
writing or through a voice recording in a clear and distinguishable manner at the latest in
the time of the first interaction of exposure.

Taking everything in consideration the Group provided a visualisation of an example

of requirements/user journey and good practice to satisfy transparency obligation
stemming from Article 52 of the Proposal, focusing on a case study of a human resources
AI chatbot intended primarily to communicate with employees (white collar, blue collar)
and other people involved with a particular company (i.e. managers, outsourced staff)
(hereinafter: “the transparency model”). During the development of the transparency
model, the Group identified and considered several needs, fears, and obstacles that
these (and other) stakeholders may have around the use case (privacy concerns,
incorrect information, automation bias, accessibility, presentation). The transparency
model combines use of avatar, graphic representation and text and sound on interface to
ensure that the information on communication with an AI system is delivered to a natural
person using the AI system.

2 In the latter case, such AI system is treated as any other AI system intended to interact with natural
persons
2
2. Prototype User Journey (incl. disclaimer)

START Avatar:
• robot-like
• set on an obvious
place on the interface

Introduction message/ Additional information

disclaimer • clear statement that a
• pops up over avatar chatbot is an AI system
• content of message: "I intended to interact
am not a human, but with natural persons
an artificial intelligence • description of uses
(AI) chatbot you are and purposes of a
interacting with. For particular chatbot (e.g.
additional information information concerning
on how a chatbot employment status
functions. Click here" and relevant legal
frameworks for an
employee or other staff
of a company)
• content of a chatbot
• disclaimers and
limitation of liability
of a provider - pain
points (e.g. erroneous
messages, bias,
privacy concerns)
• complaints and
feedback - information
on how to file a
complaint or give
feedback to providers

Use of a chatbot
END

3
3. Prototype decision process – Decision tree based on
art. 52 AI Act

Further on, the Group also provided a general decision tree for decision-making process
on the need for implementing a transparency model to comply with Article 52, §1 of the
Proposal:

START

AI SYSTEM:
• INTENDED TO INTERACT WITH
NATURAL PERSONS; OR

YES NO

EXCEPTIONS:
• OBVIOUS; OR
• AI SYSTEMS AUTHORISED BY LAW TO
DETECT, PREVENT, INVESTIGATE AND
PROSECUTE CRIMINAL OFFENCES

NO YES

APPLY TRANSPARENCY
MODEL/DISCLAIMER

EXIT

4
DISCLAIMER 2

Use Case: Deep fakes in

documentaries
DISCLAIMER 2

Use Case Description – background information

• Type: Documentary - testimonial of a victim of domestic violence meant
for television or online streaming (larger audience).
• Purpose: Awareness - raising, inspiring story, prevention, opportunity to
educate (pedagogical tool)
• Legal principles:
• In order to preserve anonymity (privacy), the person will be replaced
by an AI generated character (deepfake).
• The purpose of our prototype is to respect the EU AI act requirements
related to transparency and keep the authenticity of the message as
well as the identification process of the audience which is necessary
for awareness - raising and prevention of domestic abuse and
violence.
• Audience: a large adult audience will watch a program on national TV
and the needs of vulnerable viewers are taken into account through
trauma - trained professionals, consideration for visually impaired, deaf/
hard of hearing audience or viewers with low literacy, …
• Permanent watermark: the transparency requirement and icon needs to
be accessible by all audiences at all times.

Contents
1. Prototype decision process 1
2. Prototype: Icon (AI-generated) 1
3. Prototype: Information Notices 2
4. Prototype: Deep Fake Information Policy 3
1. Prototype decision process: Triple A Matrix (AAA) for
decision-making

Based on the audience, the privacy protection of the person and the clear understanding
of the transparency requirements of the EU AI ACT, we created a Triple A Matrix (AAA) with
scoring as a tool to select the best characteristics for our prototype.

The AAA Matrix consists of two sets of elements. On the one hand (vertical axis), there
are tools that can be used to achieve transparency regarding the use of deep fakes
(i.e. an icon, an information policy, an oral warning by a presenter, a QR-code, a trigger
warning and a permanent disclaimer). On the other hand (horizontal axis), there are
three features/characteristics (accessibility, anonymity and apprehension) that these
tools can have which will determine their suitability in particular use cases. By awarding
scores (between 0 and 3) for each feature/characteristic per tool and by adding up these
scores, you can determine which type of tool is most suited to facilitate transparency in
your use case.

AAA Matrix Accessibility Anonymity Apprehension Total Scores

Clear and Safeguard Information
distinguishable rights and about the
manner freedoms of disclaimer shall
third parties be provided in a
clear manner
Icon 2 2 2 6
Information 2 2 2 6
Policy
Audio Presenter 2 2 2 6
QR code 1 2 1 4
Trigger warning 1
0 0 1 1
Permanent 1 2 1 4
disclaimer
Matrix (scores): 0 not relevant, 1 Somewhat relevant, 2 Very Relevant

RESULT: As the Icon, Information Policy and an Oral warning by a Presenter have gathered
the highest score, we decided to further elaborate these tools.

2. Prototype: Icon (AI-generated)

Requirements for the watermark icon prototype for deep fakes

• Part of immutable infrastructure
• The idea is to add a visual icon to the deep fake character itself, to
make sure it is always visible, even when e.g. the format changes,
pieces are cut off or the screen is not wholly visible. In any circumstance
where the audience sees (part of) the deep fake character, the
audience should also be able to see the visual icon.

1 A Meta-Analysis of the Effects of Trigger Warnings, Content Warnings, and Content Notes
1
 • Respectful design
• No political connotation
• Standard template ready for implementation
• Adaptable on all skin types, types of faces (beards…)
• Color contrasts where needed (conformity with accessibility legislations and
standards)2
• Same position, easy to find
• Recommend to standardise icon (e.g. ISO graphical symbols3).

Illustrative example: the icon (the coloured line with dots) could be placed on the face/
chin of the deep fake, using colour contrasts to make the icon visible on any background/
face

3. Prototype: Information Notices

[to be made available in the language(s) of the audience]

1. Oral warning by the presenter prior to each deep fake testimony

• Before we proceed, please be advised: the testimony you're about to witness
uses a computer-generated representation, not an actual person but a so-
called ‘deep fake’. Any resemblance to real individuals is purely coincidental
and unintended. Sensitive viewers, especially those who might recognize
similarities with someone they know or who have lived similar experiences,
should exercise caution. Please visit the link in the credits or scan the QR
code to read more about our use of deep fakes.

2 European Accessibility Act and Web Content Accessibility Guidelines (WCAG)

3 ISO Graphical symbols
2
2. Short warning to be shown next to the deep fake during the testimonial
• This is a visual representation, not the actual witness. For more info, visit
[WebsiteURL.com/DeepfakePolicy]. [or QR code]

3. Notice in the credits

• Some individuals shown in this documentary are visual deepfake
representations to protect their identities. You can recognize such deep fake
presentations by the icon [ICON] attached to the image. Any resemblance
to real individuals is purely coincidental and unintended. Learn more at
[WebsiteURL.com/DeepfakePolicy].

4. Prototype: Deep Fake Information Policy

[to be made available in the language(s) of the audience]

Note: the policy is to be drafted in multiple layers - only text marked as layer 1 is visible at
first - the text expands upon clicking the title

We use deep fakes to protect the identity and the privacy of the persons testifying in
our program. Any resemblance to real individuals is purely coincidental and unintended.
[layer 1]

Here's everything you need to know about our use of deep fake technology. [layer 1]

We Use Deep Fakes to Protect Vulnerable Witnesses [layer 1]

• To keep the identities of some witnesses confidential, we apply deepfake
technology. This strategy allows us to showcase authentic testimonies
without revealing the real faces of those narrating their experiences.

Deep Fakes are entirely computer generated and do not resemble the actual witness
[layer 1]
• Deep fakes are computer-generated visuals that mimic human appearances
and expressions. While they can serve various purposes, our exclusive
intention is protective and truth-preserving.
• Our deep fakes are crafted not to resemble the actual witnesses, ensuring
complete discretion.

Deep Fakes are clearly recognizable [layer 1]

1. Visual Indication: [icon] [layer 1]
• The icon / watermark above accompanies each deep fake representation [in
this documentary], helping viewers discern between actual individuals and
computer-simulated ones.
• For more information about the use of these watermarks, please see here [/
link].

2. Narrative Acknowledgement [layer 1]

• Before a deep fake representation appears, our presenter provides an oral
warning.

3
3. Disclaimer [layer 1]
• Our deep fakes are always accompanied by a short disclaimer.

Our Deep Fakes are ethically generated by [Company/Product] [layer 1]

• We have utilized [technology X] to produce the deep fakes presented in this
documentary. [Technology X] has undergone rigorous scrutiny to ensure
there are no privacy infringements and that its application aligns with ethical
standards.
• For a comprehensive overview of [Technology X], the generation of deep
fakes and its commitment to privacy, please visit [TechnologyXPrivacyURL.
com].

No separate use or modification of our deep fakes

The deep fakes in this documentary are integrated with a specific icon to ensure clarity
and transparency. While technology has its limits, we strictly prohibit any alterations or
removal of these markers. We urge audiences and platforms to respect the authenticity of
our content and not engage in unauthorised modifications or misuse.

Reference List
• European Accessibility Act and Web Content Accessibility Guidelines
(WCAG)
• Accessibility statements generator
• A Meta-Analysis of the Effects of Trigger Warnings, Content Warnings, and
Content Notes
• Latest version of the EU AI Act to date -reference for this policy prototype
• ISO Graphical symbols