Received: 2 February 2024 Revised: 5 March 2024 Accepted: 30 March 2024 IET Communications

DOI: 10.1049/cmu2.12759

REVIEW

Survey of research on confidential computing

Dengguo Feng1,2 Yu Qin2 Wei Feng2 Wei Li2,3 Ketong Shang2

Hongzhan Ma2,3

1
State Key Laboratory of Computer Science, Abstract
Institute of Software, Beijing, China
As the global data strategy deepens and data elements accelerate integrating and flowing
2
Trusted Computing and Information Assurance more rapidly, the demand for data security and privacy protection has become increas-
Laboratory, Institute of Software, Beijing, China
ingly prominent. Confidential computing emerges as a crucial security technology to solve
3
School of Computer Science and Technology, security and privacy problem, and it is also a hot subject of in contemporary security
University of Chinese Academy of Sciences, Beijing,
China technologies. Leveraging collaborative security in both hardware and software, it builds
a trusted execution environment to ensure confidentiality and integrity protection for data
Correspondence in use. This paper provides a comprehensive overview of the development process of con-
Yu Qin, Trusted Computing and Information fidential computing, summarizing its current research status and issues, which focuses on
Assurance Laboratory, Institute of Software, Beijing,
China.
the security requirements for data security and privacy protection. Furthermore, it deeply
Email: [email protected] analyses the common technical features of confidential computing, and proposes a trusted
confidential computing architecture based on collaborative hardware and software trust.
Funding information Then, it elaborates on the research status and issues of confidential computing from four
National Key Research and Development Program
aspects: hardware security, architecture and key technologies, applications, and standards
of China, Grant/Award Numbers:
2022YFB4501500, 2022YFB4501501 and evaluation. Finally, this paper provides a synthesis and outlook for the future devel-
opment of confidential computing. In summary, confidential computing is currently in a
rapidly developing stage and will play an important role in cyber security in the future.

1 INTRODUCTION driven by both government policy and industry. Both in the

technologies and products, confidential computing is also in
1.1 Background of confidential computing a rapid growth stage. According to the ‘Privacy Computing
White Paper’ released by the Privacy Computing Alliance in
With the development and application of new information 2022 [1], it is anticipated that confidential computing and
technologies such as cloud computing, big data, and artificial privacy protection technologies will be commercialized in a
intelligence, data security and privacy protection has become an large scale in the next 5 to 10 years. The report also predicts
urgent and crucial issue. With more and more tenants (including that by 2025, over 60% of large organizations will use one
individuals, enterprises, and government organizations) deploy- or more privacy protection technologies in scenarios such as
ing important data and critical applications to the cloud, trust data analysis, business intelligence, or cloud computing. Cur-
in cloud service providers has become an emergent security rently, confidential computing applications are already involved
issue. While there are many mature security mechanisms for in industries such as finance, communication, government,
data at rest or in transit in the cloud computing, protecting and healthcare [2]. Notably, major global IT companies have
data in use will face the significant technical challenge and invested in the direction of confidential computing technology,
is an open problem. Confidential computing is an innovative and released a number of significant application achieve-
security technology that has been introduced to address this ments, such as Microsoft Azure Confidential Computing,
issue. Google Cloud’s GKE, Alibaba Cloud’s ECS, Ant Finan-
Due to the rapid development of the digital economy, con- cial’s Occlum and KubeTEE, Huawei’s Kunpeng BoostKit,
fidential computing has become a popular technology area and so on.

This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the
original work is properly cited, the use is non-commercial and no modifications or adaptations are made.
© 2024 The Authors. IET Communications published by John Wiley & Sons Ltd on behalf of The Institution of Engineering and Technology.

IET Commun. 2024;18:535–556. wileyonlinelibrary.com/iet-com 535

536 FENG ET AL.

FIGURE 1 The development history of confidential computing.

1.2 Definition of confidential computing or security chips such as TPM/TCM. These chips had been
widely deployed and used in personal computers, servers, and
The Confidential Computing Consortium (CCC) [3] defines other platforms. In the mid-1990s, some foreign computer man-
confidential computing as the protection of data in use by per- ufacturers initiated proposals for trusted computing technology
forming computations in a hardware-based, attested trusted solutions. Rooted in hardware cryptographic modules (such
execution environment (TEE). The Privacy Computing Con- as IBM 4758 cryptographic co-processors) and cryptographic
sortium [1, 2], along with IEEE, Microsoft, and IBM, all have technologies, mechanisms for root of trust, secure storage, and
given their definitions about confidential computing. They are trust chain were established. This technical approach was grad-
similar in the idea of realizing confidential computing based ually accepted and recognized by the IT industry after 1999.
on hardware TEEs, but there are differences in the specific In 2003, the Trusted Computing Group (TCG) published the
understanding. Confidential computing is not just a set of tech- TPM 1.2 technical specifications, leading to TPM’s applications
nology architectures that rely on TEE hardware, but a new type in diverse computing areas. In 2009, the core of these specifica-
of secure computing model covering hardware security, system tions became an ISO/IEC international standard. In 2014, TCG
security, and data security. In our opinion, confidential comput- introduced the TPM 2.0 specification, which supported more
ing is a computing paradigm for protecting and securing data in cryptographic algorithms such as ECC and SHA256. Simulta-
use. The ultimate goal of confidential computing is that in the neously, functions such as authorization management, integrity
future cyberspace, all high-security applications are TEE-based protection, and platform configuration register (PCR) opera-
confidential computing programs. tions were incorporated. At the same time, China developed a
series of standards centered around TCM, and upgraded them
to TCM 2.0 standards in 2022. In 2004, Arm introduced Trust-
1.3 Development history of confidential Zone isolation technology based on CPU extensions, which
computing was widely adopted in smartphones. In this stage, academia
conducted many original researches on trusted CPUs. Stanford
The development of confidential computing, represented by University’s XOM [4] and MIT’s AEGIS [5] utilized enhanced
Trusted Platform Module (TPM)/Trusted Cryptography Mod- modified CPUs to isolate distinct running compartments based
ule (TCM), CPU TEE, and heterogeneous TEE, aims to achieve on programs, providing trusted isolation environments.The
the security application goal of ‘data available but not visi- Bastion architecture [6], proposed in 2010, consisted of pro-
ble’ during the data flow across computing platforms. From a cessor hardware and a small hypervisor, and it provided secure
technical perspective, we can roughly divide the development execution and secure storage capabilities for critical modules
of confidential computing into the following three stages (as on an untrusted software stack. These architectures are more
illustrated in Figure 1). secure and scalable than the TPM.

1.3.1 First stage (2003–2014) 1.3.2 Second stage (2015–2018)

TPM stage. This is the early stage of confidential computing, TEE stage. In this stage, the core technology of confidential
which is mainly guided by the idea of trusted computing, and the computing – TEE – continued to evolve and develop. In 2015,
security functions are isolated in cryptographic co-processors Intel introduced Software Guard Extensions (SGX) hardware
FENG ET AL. 537

TABLE 1 Comparison between trusted computing and confidential computing.

Trusted computing Confidential computing

Hardware ∙ Root of trust, TPM/TCM security chip ∙ Hardware TEE, CPU security support
Technical architecture ∙ An architecturally holistic security solution that ∙ Architecture-based overall security solution with the
includes a global technology stack from hardware, CPU as the trust anchor
software, and network, with the trust anchor being the
hardware root of trust
Security focus ∙ Based on the underlying hardware trustworthiness, it ∙ Confidential computing is based on TEE system
focuses on the trust mechanism of code authentication isolation with a focus on data protection
∙ Trusted computing focuses on static TEEs, which are ∙ Confidential computing focuses on dynamic TEEs,
created at startup with a fixed number of TEE isolation which can be dynamically created and destroyed, and
systems can be dynamically migrated
Data security ∙ Trusted computing has weak protection for data, and ∙ Confidential computing ensures the confidentiality and
its protection is only limited to keys and critical data integrity of large amounts of data, including
algorithmic code
Application scope ∙ Trusted computing focuses on ensuring the security of ∙ Confidential computing requires higher hardware and
terminals and edges, and is applied to PC, IoT, mobile, software performance, and is applied in the cloud
cloud platforms, and other fields, with a wider range of environment, and confidential computing focuses on
applications cloud security
Privacy ∙ Trusted computing has little correlation with privacy ∙ Confidential computing is a technical approach to
computing, and except for static TEE, the DAA privacy computing, which provides support for SMC,
(EPID) of trusted computing belongs to privacy HE, FL, etc., and confidential computing ≈ privacy
computing computing

technology, which could build secure “enclaves” in user process and promoted the rapid deployment of confidential comput-
spaces. Code and data inside enclaves were immune to software ing technologies and standards. In 2020, AMD introduced an
attacks, and memory encryption could prevent certain physical upgraded Secure encrypted virtualization-secure nested paging
attacks. Intel SGX was mainly used on cloud server platforms to (SEV-SNP), Intel introduced Trust Domain Extensions (TDX),
provide confidentiality and integrity guarantees for users’ code and in 2021 Arm also introduced Armv9 confidential compute
and data. This has also led cloud vendors to focus on technolo- architecture (CCA). The development of these technologies laid
gies related to trusted computing. Subsequently, AMD unveiled the foundation for the prosperity of confidential computing.
Secure Encrypted Virtualization (SEV) hardware to provide
VM-level isolation for cloud platforms. During this period, Arm
introduced TrustZone technology into embedded platforms, 1.4 Relationships between confidential
utilizing TrustZone-M to construct trusted embedded systems; computing, trusted computing, and privacy
the TCM specification evolved to TCM 2.0 based on the combi- computing
nation of TPM 2.0, TrustZone, and other technical features. In
Europe, researchers combined TPM with post-quantum cryp- Confidential computing, trusted computing, and privacy com-
tographic algorithms to establish the FutureTPM project [7]. At puting all have their own characteristics, but the goal of
this stage, academia also proposed two classical TEE solutions data security protection is consistent, and effective secu-
on RISC-V: MIT’s Sanctum [8] and Berkeley’s KeyStone [9]. rity complementation can be formed. Confidential computing
focuses on providing a general isolated hardware TEE. Trusted
computing focuses on building a platform operating environ-
1.3.3 Third stage (2019 present) ment trust based on TPM/TCM chips. Privacy computing
focuses on using homomorphic encryption, federated learning,
CC stage. In this stage, hardware TEE technology has become secure multi-party computation, differential privacy and other
relatively mature, and the concept of confidential computing technologies to protect data security and personal privacy infor-
has been officially accepted and started to be rapidly com- mation. The combination of these can provide rich security
mercialized. In 2019, the Linux Foundation announced the solutions for the open sharing and circulation of data. Trusted
establishment of the CCC, with members including CPU hard- computing is the platform basis for confidential computing.
ware vendors like Arm, AMD, and Intel, as well as Internet Both trusted computing and confidential computing include
cloud service providers and system suppliers such as Microsoft, TEE, but trusted computing focuses on non-programmable
Google, Huawei, Alibaba, Baidu, and ByteDance. The CCC static TEE, while confidential computing focuses on hardware-
played a pivotal role in further standardizing TEE hardware based dynamic TEE. A comparison of the two is shown in
and TEE software (such as unified SDK or API interfaces) Table 1.
538 FENG ET AL.

FIGURE 2 Comparison among typical technology roadmaps of confidential computing.

2 TECHNOLOGY ARCHITECTURE tion in cloud servers. In short, the X86 architecture is relatively
mature in confidential computing, and the architectures of Arm
2.1 Technology roadmap and RISC-V are also following up. The system-level TEE solu-
tion has garnered more industry attention and stands as a crucial
Since the concept of confidential computing was proposed, development direction for future applications.
many confidential computing technology roadmaps have Based on the common technologies and characteristics of
emerged, and the mainstream instruction set architectures have various confidential computing architecture roadmaps, confi-
added support for confidential computing hardware. Figure 2 dential computing hardware and software collaborative trust
provides a comprehensive technical comparison of various technology architecture are summarized as Figure 3. This sys-
prominent confidential computing technology roadmaps. tem can be divided into four levels. (1) Hardware–firmware
Through an in-depth comparison and analysis, it can be seen layer, furnishing the hardware security foundation for the
that: (1) while the process-level TEE solution, primarily exem- entire confidential computing platform, delivering necessary
plified by Intel SGX, boasts a smaller trusted computing base hardware security primitives and initial environment security
(TCB), it requires modification of the application and has a large boot. (2) System software layer, managing the hardware
side-channel attack surface. On the other hand, the system-level resource security management for the confidential computing
TEE solution, predominantly represented by Intel TDX, AMD platform, as well as the isolation and secure switching of
SEV, and Arm CCA, features a larger TCB but offers enhanced software components. (3) Security mechanism and service
compatibility and scalability support for confidential comput- layer, presenting a rich set of security mechanisms and trusted
ing applications. (2) These representative technology roadmaps services for confidential computing applications, and delivering
have little differences in security capabilities, but significant a universal confidential computing platform abstraction for
differences in confidentiality and integrity protection imple- upper applications. (4) Interface and application layer,
mentation technology. (3) In terms of confidential computing supplying unified programming interfaces and SDKs for the
applications, the X86 technology roadmap is mainly applied in development of confidential computing applications. It shields
cloud servers, while the Arm and RISC-V technology routes the disparities in underlying confidential computing platform
focus on embedded and mobile intelligent terminals. However, abstractions, hardware architecture, and software component
Arm 64-bit confidential servers have also joined the competi- interface. Moreover, it integrates this confidential computing
FENG ET AL. 539

FIGURE 3 Hardware–software collaborative trust technology architecture for confidential computing.

FIGURE 4 Typical application framework of confidential computing.

capability with various applications, enabling users to swiftly participant roles: platform owner, data owner, code owner, and
develop or integrate confidential computing technologies to consumer. The relationships and security requirements of each
implement various new secure applications for confidential participant role are as follows.
computing.
1. Data owner. This entity is responsible for storing, man-
aging, and controlling large amounts of sensitive data, and
2.2 Application framework has absolute ownership over the data. They determine
who can access the data, which typically contains sensi-
The typical confidential computing application framework for tive information that must not be leaked,tampered with, or
cloud computing (illustrated in Figure 4) mainly involves four used unlawfully.
540 FENG ET AL.

2. Code owner. This entity creates programs, algorithms, mod- nologies like Arm TrustZone and Intel SGX have faced
els, and other code for accessing, computing, and analysing. emerging microarchitecture attacks, such as cache-based
They typically have strong research and development capa- side-channel attacks, controlled-channel attacks, and tran-
bilities but lack actual data. They usually hope to gain as sient execution attacks. Notably, transient execution attacks
much access to data as possible, while hoping that their code represented by Meltdown and Spectre indicate that there
is not illegally tampered with or destroyed. may be more unknown microarchitecture vulnerabilities
3. Platform owner. This entity combines hardware (sourced in CPUs. Consequently, developing defense mechanisms
from hardware suppliers such as CPU manufacturers, server against side-channel attacks in hardware- and CPU-based
manufacturers, and so on), system software (including OS TEE, as well as microarchitecture defense techniques, con-
and hypervisor suppliers), and critical services (such as cloud stitutes essential research areas in confidential computing.
middleware suppliers). They construct a cloud environment These efforts can contribute to fortifying the security of
and establish a confidential computing platform to provide TEE systems themselves.
confidential computing functions and services to external (3) Confidential virtual machines/containers achieved through
entities. the collaborative efforts of hardware and software, are
4. Consumer. This entity has actual demand for the confi- poised to become the main approach for large-scale con-
dential computing platform and services. They rely on data fidential computing applications. This entails dealing with
provided by data owners and code provided by code authors, more complex trust relationships and boundaries, leading to
and use the management functions and software provided by greater challenges in trust establishment. Extending hard-
platform owners to complete their confidential computing ware trust to encompass virtual machines and containers,
needs. ensuring the isolation and integrity protection of work-
loads running within them, and establishing secure channels
To ensure the code and data are secure and trustworthy, data and remote attestation of confidential virtual machines and
owners and code owners only interact with the confidential containers will emerge as pivotal issues in the domain of
computing platforms provided by the platform owner, and confidential computing.
use secure channels to provisioning data and code. Confiden- (4) The trend in confidential computing points toward an open
tial computing platforms only load verified code and properly and interconnected platform architecture. It is a significant
decrypted data. Moreover, the security states of the confidential technical challenge that crafting a multi-level collaborative
computing platforms can be established by providing trustwor- protection architecture for confidential computing covering
thy evidences to the remote party through remote attestation. hardware, firmware, software, network, and more is imper-
In this way, the code owners ensure their code is running in a ative. Currently, there are many confidential computing
TEE, and the data owners only deliver their data after the trust platform architectures, each with distinct underlying hard-
is established. Consumers can also verify the remote computing ware implementations, and programming frameworks and
environment in this way to ensure their computing results come SDKs operate in isolation. This fragmentation significantly
from the intended TEE. affects the widespread adoption and application of confi-
dential computing. It is necessary to improve this situation:
on the one hand, design an open confidential comput-
2.3 Challenges ing platform to provide a unified confidential computing
SDK or API for upper-layer applications. Simultaneously,
(1) The foundational TEE hardware for confidential comput- implementing common security mechanisms like system
ing has not yet been deeply integrated with cryptographic isolation, memory protection, remote attestation, and so on
mechanisms, and its performance needs to be optimized. based on hardware TEE, to ensure data security from the
First, technologies like security bootstrapping, integrity computing environment level.
measurement, remote attestation, and memory encryption, (5) The establishment of a confidential computing ecosystem
which are crucial to hardware TEE, rely on cryptographic is urgently needed, and the standardization and legal reg-
mechanisms. However, their performance is currently ulations of confidential computing need to be improved.
constrained by the efficiency of these cryptographic mech- Confidential computing is a relatively new technological
anisms. The profound integration of efficient and secure field, and the diversity of hardware TEE technology archi-
cryptographic algorithms and protocols with hardware tectures and the complexity of TEE technology urgently
TEE contributes to the expanded application of hardware require the industry, academia, and application parties to
TEE technology. Moreover, by combining confidential establish a confidential computing application ecosystem to
computing with techniques like homomorphic encryption, jointly promote the development of confidential comput-
secure multi-party computation, post-quantum cryptogra- ing. The interoperability issues of confidential computing
phy, and other cryptographic methods, we can effectively platforms require participants in confidential computing
ensure privacy in computing for various stakeholders such to develop unified standards and specifications, and con-
as data owners, platform owners, and code owners. duct rigorous security assessments on the platforms based
(2) TEE is generally vulnerable to side-channel attacks, and on the standards and specifications. Moreover, given the
the security of TEE needs to be strengthened. Tech- involvement of sensitive data and privacy information
FENG ET AL. 541

FIGURE 5 Categories of trusted execution environment attacks.

in confidential computing, potential compliance and reg- escalation attacks and kernel-level rootkits. The typical attack
ulatory issues underscore the need for enhanced legal against system call is the Iago attack [10], where an untrustwor-
frameworks and regulations. thy OS attacks the upper-layer trusted applications by carefully
constructing the return value of system calls. In addition, vulner-
abilities in software implementations may also lead to attacks on
2.4 Primary research area of confidential TEE, such as memory corruption attacks and incorrect param-
computing eter validation attacks (unverified parameters may be passed to
TEE programs through shared memory).
Based on the above technical issues and challenges, this article
will provide an overview of confidential computing in the fol-
lowing four key aspects: confidential computing hardware secu- 3.1.2 Side-channel attacks
rity (Section 3), architecture and technology (Section 4), appli-
cation (section 5), and standards and assessment (Section 6). Side-channel attacks are mainly caused by the sharing of a
large number of system resources between the normal envi-
ronment and the TEE. Most of the side-channel attacks
3 CONFIDENTIAL COMPUTING are cache-based side-channel attacks, typically represented by
HARDWARE SECURITY Evict+Time, Prime+Probe, Flush+Reload, etc. Since TEE usu-
ally assumes that the OS is not trusted, it is also a common way
With the development and popularization of confidential com- to use page tables and their attributes to conduct side-channel
puting technology, the security of confidential computing attacks, typically represented by controlled-channel attacks [11]
hardware has attracted widespread attention. This section sum- and SGX-PTE [12]. In addition, there are also side-channel
marizes and analyses typical TEE attack and defense mecha- attacks that use other system resources, such as SGX-Step
nisms. attacks using interrupts [13], DRAMA attacks using DRAM
[14], side-channel attacks using TLB, and Branchscope attacks
using predictors [15].
3.1 Categories of TEE attacks
Attacks on TEE typically exploit flaws in the architecture design 3.1.3 Transient execution attack
or management mechanism of confidential computing hard-
ware to pose threats to sensitive data. In Figure 5, we have Transient execution attack is an attack method that uses specula-
sorted and categorized the main attacks, which mainly include: tive execution and out-of-order execution mechanisms in mod-
ern CPU architectures to obtain sensitive information. They
are mainly divided into three categories: one is attacks using
3.1.1 System and software attacks branch prediction mechanism, and its representative works are
Spectre attack proposed in 2020 [16] and SGXPECTRE [17],
System and software attacks mainly include OS kernel attacks where SGXPECTRE can completely break the confidentiality
and system call attacks. Kernel attacks mainly include privilege of SGX enclave; The second is attacks using out-of-order
542 FENG ET AL.

execution mechanism, and its representative work is Meltdown Some SGX defense mechanisms focus on rollback preven-
[18] and Foreshadow [19] proposed in 2018, which utilize the tion, data race detection, and side-channel attack detection.
transient instructions caused by out-of-order execution after the Rollback prevention. In order to solve rollback attacks
occurrence of an exception (such as page fault), making it pos- on SGX architecture, Matetic et al. [31] designed the ROTE
sible for applications in user space to break the address space rollback protection system to achieve integrity protection
isolation, and access the kernel data and code or the memory of distributed systems, by storing enclave-specific counters
of other virtual machines; The third is microarchitecture data in a distributed system of collaborative enclaves on distinct
sampling, which allows adversaries to collect data from shared nodes. Since SGX hardware lacks hardware state continu-
CPU microarchitecture resources such as data caches, LFBs, ity protection, making applications vulnerable to rollback
store buffers, and so on, thus leak confidential information and replay attacks, some researchers proposed possible solu-
across security domains, and its representative works include: tions, concluding formal verification of SGX state continuity
(1) RIDL attacks [20] and ZombieLoad attacks [21] using line properties [32], Engraft [33], Narrotor [34], and more.
fill buffers (LFBs); (2) Fallout attacks [22] using store buffers; Data race detection. A data race in SGX can be exploited
(3) CacheOut [23] and SGAxe [24] attacks combining cache to breach the security of the enclave code, to detect the con-
eviction and TSX Asynchronous Abort (TAA). trolled data race vulnerabilities, Chen et al. [35] proposed a
lockset-based binary analysis detection algorithm, and imple-
mented SGXRACER tool to analyse various SGX binaries
3.1.4 Faults injection attacks written in different programming languages.
Side-channel attack detection. Side-channel attack is
Faults injection attacks expose secret information by triggering one of the main challenges for SGX, and side-channel attack
physical or software-based faults in computations. Rowham- detection is an important direction for SGX defense. Chen
mer attack is a typical representative of the software-based fault et al. [36] proposed the concept of verifiable execution con-
injection attacks, which mainly utilizes the mutual influence of tracts, which request the privileged software to provide a
electrons between adjacent memory DRAM cells. Explosive guaranteed execution environment to run SGX enclaves, this
access to DRAM cells will cause bit flips in physically adjacent method can prevent side-channel attacks and OS-dependent
DRAM cells, thereby damaging the integrity of TEE mem- attacks. There are some other efforts in this direction, such as
ory. In addition, there is also a class of fault injection attack HYPERRACE [37], Déjá Vu [38], and so on.
that allow privileged software to adjust device voltage and fre- ∙ Defense mechanisms for Arm TrustZone
quency according to each CPU execution thread, resulting in a Against hardware security problems of TrustZone, David
lower probability that a bit flip will occur. Representative works et al. [39] argued that the defense mechanism of Trust-
include the CLKscrew [25], Plundervolt [26], and V0LTpwn Zone can be addressed from three aspects: architecture,
[27]. implementation, and hardware. The defense mechanism on
the architecture includes building multiple isolated environ-
ments, secure cross-world channels, memory encryption, and
3.2 Defense mechanisms for TEEs trusted computing primitives. The defense mechanism on the
implementation includes managing code runtime, type-safe
Different defense mechanisms are needed to mitigate the vari- programming languages (such as using Rust to implement
ous TEE attacks mentioned above. Next, we will discuss TEE trusted kernel and trusted application), and software verifica-
defense methods for Intel SGX and Arm TrustZone. tion (such as Komodo [40]). The defense mechanism on the
hardware includes the defense of hardware components and
∙ Defense mechanisms for SGX the defense of microarchitecture. For the defense of hardware
Alexander et al. [28] summarized the available defense components, hardware manufacturers tend to encapsulate
mechanisms for attacks on SGX at four levels: microcode more components into SoC chips; for the defense of microar-
patching, system design, compiler and SDK, and appli- chitecture, one method is to carefully implement encryption
cation layer design. CPU vendors can fix many security algorithms in software or use dedicated hardware to pre-
vulnerabilities through microcode patching. Many attacks vent information leakage in cryptographic-related operations;
cannot be fixed through microcode patching, which usu- another approach is to use cache maintenance technology to
ally requires redesigning and implementing the system to prevent information leakage through cache.
fix some underlying vulnerabilities, such as ZeroTrace [29],
which uses oblivious RAM (ORAM) in SGX enclaves to hide
memory access from side channels, thus invalidating many 4 CONFIDENTIAL COMPUTING
side-channel attacks against SGX. In some cases, the defense ARCHITECTURE AND TECHNOLOGY
task can also be left to the developer of the enclave, such as
SGX-Shield [30], which activates the ASLR functionality in 4.1 Confidential computing architecture
the SGX enclave through modifications to the LLVM com-
piler, making it more difficult for adversaries to utilize attacks The confidential computing architecture uses hardware secu-
such as ROP. rity extensions and related components to create a TEE with
FENG ET AL. 543

confidentiality and integrity protection capabilities, laying the also the most widely used confidential computing architecture
foundation for core technologies and applications of con- in the server field to date.
fidential computing. This section summarizes and analyses
typical confidential computing architectures based on different
instruction set architectures and hardware. 4.1.2 System-level TEE on X86

The X86 system-level TEE architecture, represented by the

4.1.1 Process-level TEE on X86 AMD SEV and Intel TDX, combines the isolation mecha-
nism of virtualization technology to build a TEE with virtual
The X86 process-level TEE architecture, represented by Intel machines as nodes. Its advantage is that the system within the
SGX, provides user-level secure enclaves for specific pro- TEE is mature and has strong software compatibility, while its
cesses through hardware instructions. In 2015, Intel released disadvantage is that it usually has a larger TCB than process-
the first generation of SGX, which added a set of instruction level TEE. The idea of system-level TEE architecture is more
sets and memory access mechanisms to the original archi- general than process-level TEE architecture, and its main secu-
tecture. This allows applications to implement an ‘enclave’, rity goal is to provide security protection for the whole isolated
protecting themselves from privileged software adversaries and system based on confidential computing hardware. At present,
providing hardware-based measurement and remote attestation there have been many research achievements for system-level
capabilities. Although it has advantages such as small TCB and TEE architecture, and it is expected to become the mainstream
strong security assumptions, it still suffers from side-channel confidential computing architecture in the future.
and denial-of-service attacks. Additionally, due to the limita-
tions of programming model, software porting is also relatively (1) Intel TDX
difficult. Therefore, research based on the SGX architecture Intel TDX [49] provides hardware-isolated virtual
mainly includes ease-of-use extension research and security machines known as Trust Domains (TDs), which are
enhancement research. designed to isolate virtual machines from hypervisors and
(1) Ease-of-use extension research any other non-TD components on the platform. TDX
The original SGX has limitations such as limited space, diffi- is built using a combination of Intel Virtual Machine
cult programming, and lack of backward compatibility, making Extensions, Instruction Set Architecture Extensions, Multi-
it difficult to be directly applied. Some research focuses on key Total Memory Encryption Engine, and CPU-verified
solving such problems. Solutions such as Haven [41], Graphene- software modules to isolate TDs.
SGX [42], and Occlum [43] introduce a library operating system (2) AMD SEV-SNP
(libOS) inside the enclave to provide more common interfaces AMD-SEV [50] is a solution developed by AMD to
and functional support for applications. Some research focuses build confidential virtual machines. It has evolved several
on the migration and compatibility between SGX and other iterations of development. The earliest SEV technology
confidential computing architectures, for example, vSGX [44] mainly used memory encryption to protect VM’s confi-
uses the security features provided by SEV to achieve security dentiality. Later, SEV-ES further expanded SEV to protect
guarantees similar to those of SGX. VM’s CPU registers. The latest SEV-SNP technology adds
(2) Security enhancement research a strong memory integrity protection.
Aiming at various types of attacks on process-level TEE,
many security enhancement studies have been proposed by ∙ Comparison between system-level TEE and process-
academics. MPTEE [45] proposes a scheme to provide a level TEE
more flexible memory permission control mechanism for SGX
enclave programs based on hardware MPX memory protection Confidential virtual machine is the development direction
extension. SGX-Bouncer [46] proposes a scheme to detect mali- of the next generation confidential computing architecture.
cious enclave programs based on side-channel observation and The latest architectures of the mainstream processors have
interface inspection. proposed the design of confidential computing architecture
Intra-enclave applications can also obtain targeted secu- based on this scheme. Its biggest advantage is that it pro-
rity enhancements through specific implementations. InvisiPage vides hardware TEE security while retaining good software
[47] modifies the SGX confidential computing architecture by backward compatibility. Table 2 compares the advantages
introducing a new Oblivious Page Management (OPAM) struc- and disadvantages of system-level and process-level TEE
ture to achieve memory side-channel protection. Nested enclave architectures.
[48] extends the single enclave of SGX, enabling fine-grained From the comparison of the advantages and disadvantages
hierarchical isolation within the enclave. of process-level TEE and system-level TEE, it can be seen
As a representative solution for process-level TEE archi- that the TCB of process-level TEE is smaller, but its applica-
tecture, Intel SGX has been controversial since its inception, tion compatibility is insufficient, requiring specific development
with various types of attacks constantly appearing. However, its and customization, which is suitable for high-security applica-
unique model design and stringent security assumptions have tions in specific scenarios; The TCB of system-level TEE is
also provided good references for later TEE solutions. SGX is larger, but its application compatibility is better, reducing the
544 FENG ET AL.

TABLE 2 Comparison between system-level and process-level trusted execution environment.

Advantages Disadvantages

Process-level TEE (SGX) ✓ Isolation. Separation from cloud providers and other × Applications require specific development or
tenants customization
✓ Smaller trust boundaries and potential attack surfaces × Frequent calls outside of enclave can impact
✓ Easier code inspection and monitoring performance
✓ Deployable in virtual machines, cloud-native
containers, and bare metal
System-level TEE (SEV and TDX) ✓ Isolation. Separation from cloud providers and other × Larger trust boundaries (guest OS, all applications, and
tenants VM admins)
✓ Reduce the porting effort of existing applications × When the guest OS and hypervisor are updated,
✓ More in line with enterprise deployment requirements re-authentication is required
✓ Can be a simple instance configurator setup (VMM to × The granularity of remote attestation is not granular
manage VMs and k8s to deploy containers) enough

workload of system migration, which meets the needs of that it can reuse the computing resources of general-purpose
large-scale application deployment in enterprises. processors and SoCs. Its disadvantage is that the hardware
resource scheduling is not flexible enough, the switching over-
head between worlds is high, the TCB in the secure world is
4.1.3 TEE architecture on Arm large and the access control is lacking. As the earliest confiden-
tial computing architecture on ARM architecture, TrustZone
The Arm TEE architectures mainly includes two schemes: has developed throughout the history of trusted computing to
ARM TrustZone [51] and ARM CCA [52]. ARM TrustZone, confidential computing, providing standards and references for
proposed in 2004, adopts the idea of REE-TEE dual-world many subsequent confidential computing schemes.
design, and its TEE is suitable for use by vendor-customized
system trusted applications (TAs); while CCA, proposed in ARM CCA
2021, follows the similar design idea of system-level TEE archi- In 2021, ARM proposed a confidential virtual machine CCA
tecture. The confidential VM provided by CCA is called ‘Realm architecture similar to Intel TDX and AMD SEV-SNP. The core
VM’, which is suitable for use by third-party developed TAs and of the ARM CCA architecture is the RME hardware extension
and is more versatile and extensible. and CCA firmware.
Arm CCA adds two new processor states and physical
ARM TrustZone address spaces, namely Realm world and Root world, to the orig-
Ar-TrustZone, first introduced in Armv6K, supports two iso- inal TrustZone architecture (with only secure world and normal
lated software stacks: secure world and normal world. The world), via the RME hardware extensions. The secure world
two software stacks have their own physical memory spaces and Realm world are not trusted by each other. The secure
and runtime contexts, separate page tables and interrupt vec- world is used by OEM device manufacturers to deploy plat-
tor tables, and separate system registers, thus achieving physical form secure use cases, while the Realm world is used for any
isolation between the two worlds. As the earliest widely used third-party confidential computing use cases. The Root world
TEE, TrustZone has a rich ecosystem, academic research, and runs at the highest privilege level EL3, has its own private
industrial solutions. address space, and allows access to all physical address spaces.
The research on the TrustZone architecture focuses on its The core firmware of ARM CCA is Monitor and Realm Man-
own security improvements and the expansion of new secu- agement Monitor (RMM). Monitor firmware runs at EL3 and
rity mechanisms. secTEE [53] improves the ARM TrustZone is in the Root secure state, responsible for switching between
architecture based on on-chip memory (OCM) and memory worlds. RMM firmware resides at EL2 and uses existing hyper-
encryption. Li et al. proposed TEEv [54], which relies on visor technology to manage Realms, but simplifies operations
a simplified hypervisor (TEE-visor) to allow TEE instances such as dynamic resource allocation, scheduling decisions, man-
from different vendors to run in isolation on the same phone. agement interrupts, complex device simulations, and so on.
Komodo [40] decouples the management functions of TEE RMM is only used to protect the confidentiality and integrity of
from hardware functions such as secure memory, address space Realms.
isolation, and attestation. SANCTUARY [55] moves the secure- Currently, Arm CCA is in its early stages, and hardware
world TEEs to an isolated normal-world compartment, and devices supporting ArmCCA are not yet available. Academic
uses TrustZone Address-Space Controller (TZASC) to ensure research based on Arm CCA focuses on architecture extensions
hardware-level bidirectional isolation between TAs and all other and security enhancements. Shelter [56] extended the system-
system components. level TEE architecture of Arm CCA by removing RMM’s
ArmTrustZone isolates hardware into a secure world and trust to reduce the TCB, and created process-level secure
a non-secure/normal world through a bus. Its advantage is enclaves based on CCA’s Monitor firmware. Samsung Islet [57]
FENG ET AL. 545

reimplemented RMM based on Rust to reduce potential mem- RISC-V can provide a good design framework for confi-
ory security issues. Li et al. [58] conducted formal verification dential computing architecture on it, but it also brings some
of Arm CCA firmware. problems. For example, to meet the requirements of high-level
security, many architectures require hardware modifications to
RISC-V, which makes them incompatible with existing hard-
4.1.4 TEE architecture on RISC-V ware design, and most of them only stay at the academic level. In
addition, due to the lack of actual hardware platforms for RISC-
The RISC-V instruction set architecture has its own hardware V architecture, most research can only implement RISC-V
security design, as well as features such as open source and soft cores on FPGA, which is also a major factor restrict-
flexibility, which provides unique advantages for customizing ing the development of confidential computing architecture
security attributes for different security requirements. There- for RISC-V.
fore, many RSIC-V-based confidential computing architectures
have been proposed, with representative work including Sanc-
tum [8], MI6 [59], Citadel [60], Keystone [9], Cure [61], and 4.1.5 Trusted architecture of dedicated secure
Penglai [62]. chip
Sanctum is the first TEE solution on RISC-V proposed by
MIT, providing isolation protection similar to SGX enclave. The trusted architecture of dedicated security chip mainly refers
Based on Sanctum, MIT scholars have conducted research on to architecture that uses dedicated computing chips to iso-
TEE side-channel security protection from the hardware archi- late trusted environments, such as TPM [63]/TCM [64]. This
tecture level, and have achieved many pioneering achievements. architecture separates general-purpose computing from confi-
For example, the MI6 architecture can prevent microarchitec- dential computing environments. Sensitive data is only loaded
ture side-channel attacks by isolating memory physical addresses and computed in trusted hardware secure chips. Its advan-
and providing a purge instruction to clean up private resources tage is the ability to defend against attacks such as software
of secure processes. Citadel further proposes a memory shar- side channels and off-chip physical side channels. Its disad-
ing mechanism that can resist transient execution attacks and vantage is the inability to reuse the computing resources of
microarchitecture side-channel attacks, and improves the per- general-purpose processors.
formance of enclave process through an improved dynamic Unlike the confidential computing architecture based on
cache partitioning scheme. general-purpose processors, the sensitive data and its compu-
Keystone is an open-source framework for customizable tation process in such schemes are not executed on general-
TEE proposed by Standford, which is also the most repre- purpose processors, but are processed in a TEE consisting of a
sentative solution on RISC-V. It relies on the RISC-V PMP separate chip, such as the keys and cryptographic operations in
mechanism to provide isolation, without the need for hardware TPM/TCM. Apple’s Secure Enclave Processor (SEP) [65] can
modifications. It provides security primitives such as memory also fall into this category, but its protection goal is broader and
encryption and dynamic memory management in a modularized includes independent processors and system architectures.
form to optimize and reduce the TCB for different applica- The dedicated security chip is the earliest hardware security
tions. Domestic researchers have proposed Penglai enclave on solution. As a confidential computing architecture, its func-
RISC-V, which realizes scalable fine-grained memory isolation, tionality is limited, and its flexibility often cannot meet the
confidentiality, and integrity protection. needs of today’s confidential computing tasks. However, it can
The basic architecture and isolation mechanism of RISC- be used as a root of trust in combination with other secu-
V TEE and ARM TrustZone are similar, for example, both rity architectures, providing higher security for confidential
systems have four privilege levels, achieve system isolation computing architectures.
through secure hardware, and have system context switching
mechanisms. Therefore, many confidential computing security
mechanisms can be transplanted between the two, and there are 4.1.6 Heterogeneous TEE architecture
many common issues in TrustZone and RISC-V TEE research.
There are also some differences between the two in isolation The goal of implementing heterogeneous TEE architecture is
implementation: (1) different memory isolation mechanisms, to protect the confidentiality and integrity of data on heteroge-
TrustZone relies on the MMU to implement memory isola- neous devices, as well as the security of heterogeneous devices.
tion configured at the EL3 level, while RISC-V uses the PMP Currently, the implementation schemes for heterogeneous TEE
mechanism configured under the M-Mode, and PMP does not can be divided into GPU TEE, FPGA TEE, and heterogeneous
have address translation function compared to the MMU; (2) collaborative TEE, according to the hardware and trust model
the number of isolation systems, RISC-V supports up to 16 iso- used to build the TEE.
lated systems (determined by the number of PMP entries), while
TrustZone supports only two isolated systems (normal/secure (1) GPU TEE
world); (3) application isolation, TrustZone TA’s isolation relies GPU TEE refers to a TEE built on GPU by modi-
on the secure OS, while RISC-V’s application isolation relies fying the GPU hardware design. The modified GPU can
on PMP. ensure its own computing security without relying on CPU
546 FENG ET AL.

TABLE 3 Comparison of heterogeneous collaborative trusted execution environment systems.

CPU secure hardware Heterogeneous chips Application scenarios Hardware modifications

HIX Intel SGX GPU Cloud Y

SGX-FPGA Intel SGX FPGA Cloud Y
HETEE General GPU, FPGA, etc. Cloud Y
StrongBox ARM TrustZone Edge GPU Edge N
Cronus ARM S-EL2 GPU, FPGA, etc. Cloud N
CAGE Arm RME Integrated GPU Cloud, edge N

TEE. For example, NVIDIA H100/H800 series GPUs introduced by Arm CCA to achieve access control of GPU
[66] introduced GPU-based confidential computing secu- memory, providing confidential GPU computing support
rity features in the Hopper architecture. Graviton [67] for the next generation of ARM cloud and edge devices. In
modified the GPU command processor in the GPU hard- addition, HETEE [76] designs a rack-scale heterogeneous
ware to manage and protect sensitive data and computing confidential computing environment that physically isolates
resources of confidential GPU tasks, so that they can run protected devices as a whole, allowing the use of trusted
in an isolated GPU computing environment. peripherals in the TEE.
(2) FPGA TEE
FPGAs have the unique advantage of being hardware
programmable, so many studies have built confidential 4.2 Key technologies of confidential
computing environments based on FPGAs. MeetGo [68] computing
designed a remote attestation, communication, and iso-
lation mechanism to build a TEE on FPGA through a To support data security applications and industry confidential
hardware root of trust. TEEOD [69] used reconfigurable computing applications, confidential computing includes six key
FPGA technology to customize a secure enclave based on security technologies, as shown in Figure 6: integrity and trust
programmable logic for each application instance. In addi- mechanism, attestation and security monitoring, runtime secu-
tion, there are also some studies on protecting FPGA cloud rity, secure communication and trusted channel, trusted storage
security and privacy based on TEE, such as TruFPGA [70]. and data security, and hardware acceleration and optimization.
(3) Heterogeneous collaborative TEE These technologies improve the collaborative protection level
Heterogeneous collaborative TEE refers to a TEE that of software and hardware in confidential computing platforms
is constructed collaboratively by heterogeneous chips and in terms of both basic and application security capabilities.
the main CPU. It usually refers to a design that builds
on the original CPU TEE architecture, incorporates het-
erogeneous chips into the TCB range, and achieves the 4.2.1 Integrity and trust mechanism
attestation of the heterogeneous TEE through software
or firmware. Currently, heterogeneous collaborative TEE Integrity and trust mechanism is one of the basic security capa-
has been applied to various CPU architectures and het- bilities required by confidential computing platforms, which is
erogeneous chips, and is the mainstream design scheme usually implemented by platform hardware, such as hardware
for heterogeneous TEE architectures. Its main design is measurement and root of trust, to provide hardware-based secu-
shown in Table 3. For Intel architecture, HIX [71] extends rity guarantees. The following describes the related research
the protection scope of the original Intel SGX technology from three aspects: trusted boot, measurement, and isolation.
to GPU computation by modifying the bus between the Trusted boot. TCG defines a TPM-based static and dynamic
CPU and GPU, and the GPU driver to ensure an isolated trusted boot scheme in the TPM standard. Intel Trusted Exe-
GPU execution environment; SGX-FPGA [72] designs a cution Technology (TXT) [77] and AMD SKINIT [78] are the
scheme to create a TEE between SGX and FPGA by basic implementations of the dynamic trusted boot scheme.
embedding a CPU controller and FPGA security mon- Measurement. The measurement of TEE is usually imple-
itor in the CPU-FPGA architecture to authenticate and mented by trusted hardware. SGX performs SHA-256 opera-
encrypt communication data between the CPU and FPGA tions on all memory contents during enclave initialization, and
components. For ARM architecture, StrongBox [73] uti- records the results in a preserved area [79]. The mechanism of
lizes traditional Arm TrustZone technology to ensure GPU SEV is similar to SGX, except that its measurement object is the
data security on edge devices; Cronus [74] utilizes ARM encrypted virtual machine.
secure virtualization technology to build an isolated TEE Isolation. There are usually two methods for isolating
for heterogeneous chips such as GPU and FPGA, ensur- TEE: one is based on access control, where the CPU pro-
ing the heterogeneous computation security of Arm cloud vides hardware mechanisms to prevent unauthorized access
servers; CAGE [75] utilizes the RME hardware extension to the corresponding area, such as SGX; the other is based
FENG ET AL. 547

FIGURE 6 Key technologies of confidential computing.

on hardware encryption, where memory encryption prevents [86] proposed an identity derivation mechanism that does not
unauthorized access to the TEE memory without revealing its rely on trusted third parties to establish trust between different
contents, such as SEV. In addition, some research has improved TEEs, eliminating the limitations of hardware root of trust to a
isolation mechanisms, such as SEIMI [80, 81], which proposed certain extent.
an intra-process isolation scheme based on Supervisor-Mode Representative solutions for TEE monitoring include: TZ-
Access Prevention (SMAP) and virtualization technology. RKP [87], Keylime [88], etc. Their main ideas are to monitor
the security of the host system based on TEE, or dynamically
monitor the running status of the TEE environment through a
4.2.2 Attestation and security monitoring remote attestation.

The fundamental purpose of attestation and security monitor-

ing is to transfer the trust chain of the confidential computing 4.2.3 Runtime security
platform to the verifier, enabling the verifier to evaluate the
current security of the platform and TEE. The following Research on runtime security key technology mainly focuses on
introduces its representative technical solutions, as well as the runtime security capabilities of TEE internal code, strength-
ideas and researches on improving the attestation and security ening the security of TEE itself, and defending against memory
monitoring mechanism. corruption, side-channel attacks, etc. Although TEE ensures the
The typical TEE attestation scheme is the IETF remote isolation of the internal environment to a certain extent, its secu-
attestation procedures (RATS) [82], which is currently the ref- rity usually stems from the measurement and attestation during
erence specification for many open-source remote attestation TEE startup and launching; however, TEE internal software is
implementations. For SGX, it includes two remote attestation still affected by its own defects such as memory corruption,
schemes: EPID [83] and DCAP [84]. which can be exploited by adversaries. Therefore, many stud-
Many studies focus on how to improve the attestation ies focus on runtime security after TEE startup. Two typical
and security monitoring technology in the existing confiden- runtime security solutions are described below:
tial computing architecture from the perspectives of privacy
protection and security enhancement. Research on privacy pro- (1) Runtime security solutions based on program graphs.
tection based on the extension of TEE attestation scheme is The main idea is to introduce mechanisms such as control
represented by Opera [85]. In Opera, researchers proposed an flow attestation (CFA) into TEE to ensure the integrity of
open remote attestation platform based on SGX, which can program flow within TEE. Representative work includes
be used as an alternative to Intel’s attestation service. Mage GuaranTEE [89] and SGXMonitor [90].
548 FENG ET AL.

(2) Runtime security solutions based on hardware co- 4.2.6 Hardware acceleration and optimization
processors. Chen et al. [91] proposed a hardware–software
co-processing runtime protection scheme based on the In the confidential computing platform, the computations
RISC-V architecture, which performs runtime instruction that require hardware acceleration can be divided into two
inspection and verification through dynamic information categories: accelerating TEE cryptographic algorithms and
flow tracking co-processors. protocols and accelerating TEE workloads.
Acceleration of TEE cryptographic algorithms and
protocols. The literature [99] proposes a RISC-V system com-
4.2.4 Secure communication and trusted patible with TEE, which has a secure algorithm accelerator and
channel implements acceleration of SHA-3 hash and Ed25519 ECC.
Abdulhandi [100] proposes a flexible and effective hardware
The secure communication process of the confidential com- solution based on FPGA platforms to accelerate the genera-
puting platform is highly relevant with remote attestation, tion of Merkle hash tree. Intel TEE itself also supports AES-NI
and secure communication is the foundation for achieving acceleration instructions and QAT hardware acceleration.
availability of the TEE on the confidential computing platform. Acceleration of workloads performed by TEE. Confi-
Secure communication provides communication capabilities dential computing can be used to provide privacy and data
with remote attestation between TEEs, and between TEE and protection innovative solutions for large models such as Chat-
external parties. RA-TLS [92] integrates Intel SGX remote GPT, for example, Intel Deep Learning (DL) Boost. Intel
attestation mechanism and TLS security protocol. RATS-TLS SGX/TDX can provide support for Intel AMX, DL Boost, and
[93] adds support for heterogeneous hardware TEE on the other computational instructions.
basis of RA-TLS, and supports different cryptographic algo-
rithm libraries and mutual TLS authentication between different
types of TEEs. The EU Confidential 6G project [94] was 5 CONFIDENTIAL COMPUTING
established in 2023 and will develop protocols and security APPLICATION
attestation tools, libraries, mechanisms, and architectures with
quantum resistance capabilities. It aims to integrate the charac- At present, the relevant achievements/results of the confiden-
teristics of confidential computing into the next generation (6G) tial computing platform have been gradually applied in many
communication technology. industry fields, such as finance, government affairs, medical,
communication, and industry. Typical confidential computing
application schemes are summarized in Table 4.
4.2.5 Trusted storage and data security

TEE mainly protects the security of data in use, but the secu- 5.1 TEE-based machine learning
rity of data at rest is equally important. Trusted storage and data
security technology combines confidential computing platform With the development and wide application of artificial intelli-
hardware environments and security mechanisms to provide gence (AI) technology, its data privacy and security issues have
protection for sensitive TEE data in use and at rest, usually become a hot topic in the industry. The attack surface in the
in the form of hardware memory encryption mechanisms and full pipeline of machine learning (ML) is very wide, which poses
secure storage services. a great threat to the data security of model users and own-
For security protection during the data at rest, different ers, such as data poisoning attacks, data reconstruction attacks,
confidential computing platforms use different methods. Arm and so on. Using TEE to protect partial pipeline of ML is
TrustZone and OP-TEE [95] provided users with two data pro- one of the frontier directions of AI security research. Cur-
tection modes, RPMB_FS and REE_FS. Intel SGX introduced rently, the research on using TEE to protect the ML process
the Intel Protection File System (IPFS) library to increase the mainly focuses on the training process of the model and the
security of data at rest. In addition, some research has proposed inference process using the model after deployment. The pro-
improvements to existing secure storage mechanisms, with rep- tection schemes of using TEE for the training and inference
resentative work including DISKSHEILD [96] and SecureFS processes of ML can be divided into four types, as shown in
[97]. Figure 7.
For security protection for data in use, a primary method is
hardware memory encryption, which can ensure the confiden- (1) Machine learning full pipeline protection. The entire
tiality of the data stored in memory within a TEE. Intel SGX inference/training process is carried out in TEE, providing
uses a hardware Memory Encryption Engine (MEE) to apply the highest level of security and protecting against threats
encryption and decryption to data, while AMD SEV introduces other than side-channel attacks. However, there are some
an AES encryption engine inside the System on Chip (SoC) limitations to the current approach. First, most TEE rely
that transparently encrypts and decrypts the data when the data on CPU as the root of trust, which cannot effectively
leaves or enters the SoC, respectively [98]. leverage GPU-based computing acceleration, resulting in
FENG ET AL. 549

TABLE 4 Confidential computing application solution.

Research
Application solution institutes Technology idea Introduction of the solution

Intel Confidential Computing Intel Run isolated applications using Intel SGX 2.0 Based on various security technology stacks,
Zoo (CCZoo) and LibOS, and provide security providing end-to-end security solutions
components such as remote attestation, for different application scenarios
key management, and TLS secure
communication
BigDL PPML Intel One-stop privacy computing solution An end-to-end big data and AI privacy
provided by a combination of SGX, computing solution, developers can easily
LibOS, Spark, federated learning, and create end-to-end distributed AI
other technologies applications using BigDL
Nitro Enclave (NE) AWS (Amazon) Enclave based on virtualization technology, A new feature to Amazon EC2 that allows
using enclave vsock for communication to tenants to create isolated execution
reduce the attack surface environment enclave VMs within Amazon
EC2 instances to run highly sensitive data
applications
Azure Kubernetes Service (AKS) Microsoft Support for adding Intel SGX confidential Run applications packaged as containers,
computing VM nodes as agent pools in a isolating them from other containers
cluster (node cores)
Confidential GKE (Google Google Use confidential VM technology and Leverage dedicated hardware to encrypt data
Kubernetes Engine) node leverage the Secure Cryptographic in use and use it for compute-optimized
Virtualization (SEV) capabilities of AMD C2D virtual machines
EPYC processors
Constellation system Edgeless System The adoption of confidential computing Utilize confidential computing to isolate the
eliminates the security risks of cloud-based entire Kubernetes cluster from the
infrastructure, protecting user data from infrastructure to form a confidential K8S
cloud providers and hackers autonomous system, which ultimately
turns a public cloud into a private cloud
Cloud data security and privacy Anjuna A simple application integration and A solution that safeguards user workloads
solutions deployment method for confidential from snooping and unauthorized cloud
computing, including compilation, tampering. User workloads remain
deployment, operation, and security confidential and trusted during execution,
policies for the full life-cycle application of without the threat of code and data
confidential computing exposure, enabling faster implementation
of confidential computing applications
Ciphertext database scheme Microsoft Including SQL enclave (responsible for Microsoft’s ciphertext database solution,
SQL Server 2019 decrypting data) and DB user enclave providing full ciphertext queries and
(responsible for establishing secure computations to prevent multiple types of
channels) software and hardware attacks
ECS confidential computing Alibaba Cloud Alibaba Cloud servers are equipped with a Alibaba Cloud server g7t instances support
ECS security chip TPM as the hardware root of Intel SGX, ECS security-enhanced
trust, supporting vTPM, Intel hardware general-purpose g7t ECSs support
TEE and TME, and self-developed TPM/TCM chips for trusted boot, and g8i
encrypted computing isolation supports confidential virtual machines
environment enclave, providing a high (TDX).
level of security data protection and a
trusted runtime environment on the cloud
SOFAEnclave Ant Financial Consisting of three parts: the enclave kernel A financial-grade confidential computing
Occlum, the cloud-native confidential middleware to ensure the confidentiality
computing cluster KubeTEE, and the and integrity of data and code for financial
security testing and analysis framework applications

significantly reduced training speed. The development of hardware. Additionally, for multi-enclave solution, it also
GPU-TEE is expected to address this issue. Second, TEE has extremely high requirements on the communication
has limited computational resources, for example, the max- efficiency between enclaves. Citadel [60] is a typical scheme
imum Enclave Page Cache (EPC) size for SGX1 is 128M. for full pipeline protection for model training and inference
When TEE memory usage exceeds this limit, it will cause within TEE, protecting the confidentiality of models and
the memory swapping to reduce the efficiency. Although data. In this scheme, each data owners and model own-
SGX2 addresses this issue, it puts higher demands on server ers have their own enclaves, and data does not leave their
550 FENG ET AL.

FIGURE 7 Trusted execution environment-based protection schemes machine learning.

enclaves. Masking is used to ensure that no model gradient process. DarKnight [105] is an improved scheme based on
information is exposed during aggregation. SLALOM that uses multiple GPUs. It blinds input data
(2) Partial pipeline protection. This method performs part by linearly combining the batch and random noise and
of the pipeline in TEE. The method places sensitive lay- uses redundant GPUs to validate the computation results,
ers of neural networks into a TEE for execution, to protect ensuring the integrity and confidentiality of DNN train-
data in sensitive layers (e.g. the output layer in a model), ing. Goten [106] proposes a scheme that utilizes secret
or conducts rolling protection over the layers of machine sharing to outsource the computation of the linear layer
learning models. While reducing the hardware requirements based on a multiplicative secret sharing protocol between
on servers, this approach requires modifications to the three untrustworthy GPUs, using TEE to share the ran-
model training protocol and requires that the layers of the dom number seed in advance, thus reducing the number of
model be segmented in advance to identify the layers to negotiation rounds for the secret sharing protocol.
be protected. DarkneTZ [101] is a representative scheme (4) Integrity protection. This approach utilizes TEE to
for protecting ML models using TEE, which focuses on monitor the model training/inference process, either by
deploying trained models to edge devices for use. It sup- checking the integrity of the training/inference program
ports encrypting and deploying arbitrary layers to edge or by checking log files. Additionally, mathematical meth-
device TEE to protect model privacy. PPFL [102] is a typ- ods can be used to verify the correctness of computational
ical scheme for rolling protection of ML models, aimed results. Chen et al. [107] propose a privacy-preserving fed-
at protecting data confidentiality in horizontal federated erated learning scheme based on TEE that verifies the
learning scenarios. GradSec [103] proposes an improved integrity of the deep learning process and detects malicious
approach that dynamically protects sensitive layers of ML participants by validating the enclave’s measurement and
models, reducing TCB size and overall training time. signature.
(3) TEE-assisted outsourcing computation. This method
uses TEE to process data before outsourcing it to untrusted
accelerators for computation, ensuring data security while 5.2 TEE-based privacy computing
increasing inference/training speed. This category of meth-
ods requires modifications to the model structure, at the Privacy protection methods such as fully homomorphic encryp-
same time, can reduce the loss of training rate to an tion and secure multi-party computation usually have high
acceptable level while providing confidentiality guarantees. overheads. Using the enhanced trust model of TEE can sim-
SLALOM [104] is a representative scheme for outsourcing plify the privacy computing protocol to a certain extent,
computation protection, which uses matrix multiplication thereby reducing the overhead. In addition, using the secu-
linearity to add blinding factors to data before outsourcing rity capabilities of TEE can further enhance the security of
it to an untrusted GPU for computation with parameter privacy computing.
matrices. The results are then de-blinded in TEE to ensure Brenna et al. [108] utilized the Inter SGX’s guarantee
the confidentiality of input data, while using a complexity of enclave integrity to implement a dependent library that
of O(n2 ) probability algorithm to verify the correctness of performs fully homomorphic encryption in SGX. Fischer
matrix multiplication to ensure the integrity of the inference et al.[109] implemented a dependent library in TEE that can
FENG ET AL. 551

compute encrypted data with side-channel protection, and is Teechan [118], Ant’s CONFIDE [119], Shadoweth [120],
more efficient than using FHE to achieve the same effect. and more.
HYBRTC [110] is a TEE-based hybrid trusted multi-party com-
putation protocol based on the different levels of trust of the
participating parties in multi-party computation. Compared to 5.3.4 Confidential data analytics
traditional protocols, this method has up to three orders of
magnitude improvement in performance, and it also utilizes the VC3 [121] applies confidential computing techniques to the
Melbourne shuffle to defend against side-channel attacks. analysis of trusted data in cloud environments, presenting the
TEE can establish full or partial trust between participants first system to run distributed MapReduce computation in the
in privacy computing. Traditional privacy computing protocols cloud. VC3 [121] relies on the SGX technology to isolate mem-
can leverage this to reduce communication rounds between par- ory regions on individual computers and deploys new protocols
ticipants and optimize the efficiency. There have been many for the protection of distributed MapReduce computation.
studies on applying privacy computing protocols to data min- Opaque [122] is an oblivious distributed data analytics platform,
ing, joint analysis of data, blockchain, and other scenarios, which utilizes Intel SGX to provide strong security guarantees.
but the efficiency issue remains the biggest bottleneck for
practical applications of these studies. Combining these stud-
ies with TEE to reduce overhead is one way to address this 5.4 Confidential computing open-source
bottleneck. applications
With the rise of confidential computing, there have been many
5.3 Other confidential computing open-source projects related to confidential computing, as
applications shown in Table 5. Domestic and foreign enterprise giants and
research institutions open source their confidential computing
In addition to the applications mentioned above in machine components, mainly to build a confidential computing ecosys-
learning, and privacy computing, confidential computing also tem, promote confidential computing technology, and expand
has a wide range of applications in Function Encryption (FE), their influence in this field. Currently, data security and artificial
blockchain, database, etc. intelligence are the two main driving forces for the development
of open-source confidential computing: in order to promote the
development of high-quality digital economy, many countries
5.3.1 Confidential FE urgently need to enhance data security in industry applications,
and there is an urgent need for open-source confidential com-
IRON [111] proposes the first practical and provably secure FE puting projects as technical support; AI has made revolutionary
system that can be instantiated on real trusted hardware based breakthroughs, and its application will grow explosively. How-
on confidential computing, and implements it on Intel SGX, ever, its security and privacy issues have become a global focus,
demonstrating the effectiveness of IRON. and open-source confidential computing projects contribute to
enhancing the trustworthiness of AI.

5.3.2 Confidential database

6 CONFIDENTIAL COMPUTING
EnclaveDB [112] applies confidential computing technology STANDARDS AND ASSESSMENTS
to database to provide confidentiality, integrity, and freshness
guarantees for both data and data queries. Other confiden- Confidential computing is an emerging technical field, and
tial database applications include Microsoft Azure Always its standards and evaluation specifications are still in the
Encrypted (AE) [113], Aliyun Operon [114], Huawei GaussDB early stages of development. Trusted computing serves as the
[115], and more. platform foundation for confidential computing, and TEE
is the primary supporting technology. For confidential com-
puting, both of these standards are undergoing continuous
5.3.3 Confidential blockchain improvement and refinement.
The main standards in the field of trusted computing are the
Ekiden [116] combines confidential computing TEE with smart TPM series standards developed by the international trusted
contracts to design a novel architecture that separates ‘con- computing organization and the TCM series standards in China
sensus’ and ‘execution’ to achieve efficient TEE-supported [123]. TPM 1.2 [124] is the earliest trusted computing standard
confidentiality protection for smart contracts. Microsoft pro- to be proposed and established as an international standard, and
poses a open-source framework Confidential Consortium the current TPM standards have evolved from TPM 1.2 to TPM
Framework (CCF) [117], which can build permissioned confi- 2.0 [125].
dential blockchains based on SGX-enabled Azure Confidential In the field of TEE, many organizations both domestically
Compute. Other confidential blockchain applications include and internationally have gradually developed relevant standards
552 FENG ET AL.

TABLE 5 Confidential computing open-source projects.

Open-source
community/research
Project name institutes Technology ideas Introduction

Intel SGX SDK Intel Development interface for SGX Provide developers with interfaces to develop and
support SGX enclave programs provide developers
with interfaces to develop and support SGX enclave
programs
MultiZone HEX-five TEE for RISC-V or Cortex-M Suitable for IoT scenarios, and building secure IoT
firmware and secure IoT applications based on TEE
technology
Keystone Intel Open-source TEE based on existing The TEE based on the open-source hardware
hardware functions of RISC-V architecture RISC-V, which can be customized for
specific scenarios
Gramine Invisible Things Lab/ SGX-based LibOS project For running unmodified applications in SGX enclave
Intel
Occlum Ant Financial SGX-based LibOS project Intel SGX-based memory-safe, multi-process library
operating system for running unmodified
applications in SGX enclave
Open Enclave Microsoft Providing a unified programming interface Hardware-independent open-source library of
(OE) SDK hardware TEEs, supporting C/C++ to write
enclaves
Asylo Google Providing a unified programming interface A flexible programming framework for developing
enclaves applications
OP-TEE Linaro An open-source implementation of trusted The main operating environment in the secure world
OS of Arm TrustZone, compatible with GP TEE
specifications

and evaluation specifications in recent years. Internationally, the Engineering Task Force (IETF) has introduced the RATS
Open Mobile Terminal Platform (OMTP) organization pro- specifications [130], which is an important component of con-
posed the TEE standard for mobile intelligent terminals in 2009 fidential computing. The National Institute of Standards and
[126]. This standard delineates TEE as a combination of both Technology (NIST) has also contributed by releasing spec-
hardware and software, aiming to provide necessary support ifications addressing security protection in cloud and edge
for applications. Currently, most of the TEE standard work computing scenarios [131]. Meanwhile, ‘Information Security
is driven by GlobalPlatform, which has developed a series of Technology - General Framework for Confidential Computing’
standards for TEE in terms of network, system architecture, [132], a national standard in China delineating the applica-
and application internal API [127]. Within China, ‘Informa- tion framework of confidential computing, has reached the
tion Security Technology - Basic Security Norms for TEE’ solicitation of opinions stage.
[128] establishes the overall technical framework for TEE sys- Although the current standards and assessments related to
tems. It outlines fundamental requirements for TEE, trusted confidential computing are not mature enough, we believe
virtualization systems, trusted operating systems, trusted appli- that they will evolve rapidly and cover at least the following
cation and service management, cross-platform middleware, aspects: (1) Standards of confidential computing architectures,
and other main content along with testing and evaluation meth- specifying the hardware and software components and security
ods. ‘Information Security Technology - Technical Norms for capabilities of confidential computing; (2) Standards of con-
TEE Services’, [129] establishes a technical framework for TEE fidential computing interfaces, clarifying how these hardware
services, specifying technical requirements along with testing and software components interact and a unified program-
and evaluation methods. As the application of TEE in multiple ming interface for CC security capabilities; (3) Standards
industry fields deepens, the development of relevant standards of attestation architectures, defining roles, protocols, proce-
and evaluation specifications is receiving attention from indus- dures and interactions for remote attestation and covering a
try management departments such as communication industry wide range of TEE hardware; (4) Security functions testing,
field, mobile device fields and financial industry field. Addi- clarifying how to test and use TEE hardware and security
tionally, some related groups and enterprises have developed capabilities of confidential computing; and (5) TEE interface
standards and evaluation specifications. compliance testing, specifying how to test the confidential com-
Advancements in confidential computing standards and puting platform and its interfaces for compliance with standard
evaluation specifications have been notable. The Internet specifications.
FENG ET AL. 553

7 SUMMARY AND OUTLOOK AUTHOR CONTRIBUTIONS

Dengguo Feng was responsible for the overall paper structure,
In the realm of data security, there has been significant progress research direction, and chapter design. He was also tasked with
in safeguarding data in transit and at rest. However, ensuring revising the abstract and the content of Section 7. Yu Qin was
the security of data in use remains a shortcoming, particu- in charge of writing and revising the abstract, Section 1, and
larly concerning data security for data owners, system security various sections involving figures and tables. Additionally, he
for platform operators, and algorithm security for data proces- was responsible for proofreading the entire paper. Wei Feng was
sors. Confidential computing addresses this gap by prioritizing responsible for writing and revising Sections 2, 3, and 7. He was
the security of data in use, guaranteeing data confidential- also involved in proofreading certain sections of the paper. Wei
ity and integrity through hardware-level system isolation and Li, Ketong Shang, and Hongzhan Ma were responsible for writ-
collaborative protection involving both hardware and soft- ing and revising Sections 4, 5, 6, and handling the compilation
ware. Confidential computing has emerged as a hotspot in the and modification of the reference list.
advancement of new secure computing technologies and is
poised to offer crucial security support for the healthy growth of ACKNOWLEDGEMENTS
the digital economy. Currently, confidential computing exhibits This work was supported by National Key R&D Program
the following developmental trends. of China (2022YFB4501500 and 2022YFB4501501). Thank
you to Dr. Fengwei Zhang of Southern University of Science
1. The application is more in-depth. Confidential comput- and Technology for providing relevant materials on CCA and
ing realizes the security of data, algorithms, and platforms, heterogeneous TEE.
which can protect the benefits and privacy of different
data owners, application developers, and platform service CONFLICT OF INTEREST STATEMENT
providers. Confidential computing attempts to solve the con- The authors declare no conflicts of interest.
tradiction between data protection and utilization. It can be
widely applied to cross-departmental data security sharing DATA AVAILABILITY STATEMENT
in e-government, joint risk control in the financial industry, All relavant data are within the paper.
medical, drug, and gene research in the medical field, joint
marketing in the commercial field, and so on. ORCID
2. The need for standardization is more urgent. While Dengguo Feng https://orcid.org/0000-0002-8515-7124
the conceptual framework of confidential computing hard-
ware and software collaboration is relatively well defined, REFERENCES
the absence of standardized implementation methods has 1. Alliance, P.C.: Privacy computing white paper (2022). http://221.179.172.
resulted in diverse approaches by different vendors. This 81/images/20221229/48421672279200206.pdf
2. Alliance, P.C.: Research report on privacy computing applications (2023).
variance introduces technical complexity, bringing challenges
https://dsj.guizhou.gov.cn/xwzx/gnyw/202308/t20230819_81912599.
even for professional security staff. Recognizing this, the html
Confidential Computing Alliance has initiated the develop- 3. Consortium, C.C.: A technical analysis of confidential computing
ment of a unified open-source SDK to actively promote the (2021). https://confidentialcomputing.io/wp-content/uploads/sites/
research of confidential computing standards. 10/2023/03/CCC-A-Technical-Analysis-of-Confidential-Computing-
v1.3_unlocked.pdf
3. Collaborative security is more closely integrated. Confi-
4. Lie, D., Thekkath, C., Mitchell, M., et al.: Architectural support for copy
dential computing is still in its early stages of development, and tamper resistant software. ACM Sigplan Notices 35(11), 168–177
and requires collaborative support from confidential com- (2000)
puting hardware (CPU and FPGA), cryptographic and 5. Suh, G.E., Clarke, D., Gassend, B., Van Dijk, M., Devadas, S.: AEGIS:
privacy protection mechanisms (secure transport layer pro- Architecture for tamper-evident and tamper-resistant processing. In:
ACM SIGSAC, pp. 357–368 (2003)
tocols and homomorphic encryption, etc.), trusted com-
6. Champagne, D., Lee, R.B.: Scalable architectural support for trusted
puting mechanisms (hardware root of trust and remote software. In: IEEE, pp. 1–12 (2010)
attestation), virtual machine and operating system security, 7. Moreira, L.C.J.: FutureTPM reference architecture (2018). https://cordis.
container and microservice security, and more. This collab- europa.eu/project/id/779391/results
orative effort is essential for the continued advancement of 8. Costan, V., Lebedev, I., Devadas, S.: Sanctum: Minimal hardware exten-
sions for strong software isolation. In: USENIX Association, pp.
confidential computing.
857–874 (2016)
4. Practical research is more important. Practical and inno- 9. Lee, D., Kohlbrenner, D., Shinde, S., Asanović, K., Song, D.: Keystone:
vative research on fundamental methods of confidential An open framework for architecting trusted execution environments. In:
computing is very important, as it can provide theoreti- ACM, pp. 1–16 (2020)
cal foundations and scientific methods for the development 10. Checkoway, S., Shacham, H.: Iago attacks: Why the system call API is
a bad untrusted RPC interface. ACM SIGARCH Comp. Archit. News
of confidential computing. These basic methods include
41(1), 253–264 (2013)
homomorphic encryption, secure multi-party computation, 11. Xu, Y., Cui, W., Peinado, M.: Controlled-channel attacks: Deterministic
inadvertent transmission/obfuscated circuits/secret sharing, side channels for untrusted operating systems. In: IEEE, pp. 640–656
differential privacy, federated learning, TEE, and so on. (2015)
554 FENG ET AL.

12. Van Bulck, J., Weichbrodt, N., Kapitza, R., Piessens, F., Strackx, R.: 38. Chen, S., Zhang, X., Reiter, M.K., Zhang, Y.: Detecting privileged side-
Telling your secrets without page faults: Stealthy page table-based channel attacks in shielded execution with Déjá Vu. In: IEEE, pp. 7–18
attacks on enclaved execution. In: USENIX Association, pp. 1041–1056 (2017)
(2017) 39. Cerdeira, D., Santos, N., Fonseca, P., Pinto, S.: Sok: Understanding the
13. Van Bulck, J., Piessens, F., Strackx, R.: SGX-Step: A practical attack prevailing security vulnerabilities in trustzone-assisted TEE systems. In:
framework for precise enclave execution control. In: Workshop on IEEE, pp. 1416–1432 (2020)
System Software for Trusted Execution, pp. 1–6 (2017) 40. Ferraiuolo, A., Baumann, A., Hawblitzel, C., Parno, B.: Komodo: Using
14. Pessl, P., Gruss, D., Maurice, C., Schwarz, M., Mangard, S.: DRAMA: verification to disentangle secure-enclave hardware from software. In:
Exploiting DRAM addressing for cross-CPU attacks. In: USENIX ACM SIGSAC, pp. 287–305 (2017)
Association, pp. 565–581 (2016) 41. Baumann, A., Peinado, M., Hunt, G.: Shielding applications from an
15. Evtyushkin, D., Riley, R., Abu-Ghazaleh, N., Ponomarev, D.: Branch- untrusted cloud with haven. ACM Trans. Comput. Syst. 33(3), 1–26
scope: A new side-channel attack on directional branch predictor. ACM (2015)
SIGPLAN Notices 53(2), 693–707 (2018) 42. Tsai, C.C., Porter, D.E., Vij, M.: Graphene-SGX: A practical library OS
16. Kocher, P., Horn, J., Fogh, A., et al.: Spectre attacks: Exploiting for unmodified applications on SGX. In: USENIX Association, pp. 645–
speculative execution. Commun. ACM 63(7), 93–101 (2020) 658 (2017)
17. Chen, G., Chen, S., Xiao, Y., Zhang, Y., Lin, Z., Lai, T.H.: Sgxpectre: Steal- 43. Shen, Y., Tian, H., Chen, Y., et al.: Occlum: Secure and efficient mul-
ing intel secrets from SGX enclaves via speculative execution. In: IEEE, titasking inside a single enclave of intel SGX. In: ACM, pp. 955–970
pp. 142–157 (2019) (2020)
18. Lipp, M., Schwarz, M., Gruss, D., et al.: Meltdown: Reading kernel 44. Zhao, S., Li, M., Zhangyz, Y., Lin, Z.: vSGX: Virtualizing SGX enclaves
memory from user space. Commun. ACM 63(6), 46–56 (2020) on AMD SEV. In: IEEE, pp. 321–336 (2022)
19. Van Bulck, J., Minkin, M., Weisse, O., et al.: Foreshadow: Extracting the 45. Zhao, W., Lu, K., Qi, Y., Qi, S.: MPTEE: Bringing flexible and efficient
keys to the intel SGX kingdom with transient out-of-order execution. In: memory protection to intel SGX. In: ACM, pp. 1–15 (2020)
USENIX Association, pp. 991–1008 (2018) 46. Zhang, Z., Zhang, X., Li, Q., et al.: See through walls: Detecting malware
20. Van Schaik, S., Milburn, A., Österlund, S., et al.: RIDL: Rogue in-flight in SGX enclaves with SGX-bouncer. In: ACM, pp. 931–943 (2021)
data load. In: IEEE, pp. 88–105 (2019) 47. Aga, S., Narayanasamy, S.: InvisiPage: Oblivious demand paging for
21. Schwarz, M., Lipp, M., Moghimi, D., et al.: ZombieLoad: Cross-privilege- secure enclaves. In: ACM/IEEE, pp. 372–384 (2019)
boundary data sampling. In: ACM SIGSAC, pp. 753–768 (2019) 48. Park, J., Kang, N., Kim, T., Kwon, Y., Huh, J.: Nested enclave: Support-
22. Canella, C., Genkin, D., Giner, L., et al.: Fallout: Leaking data ing fine-grained hierarchical isolation with SGX. In: IEEE, pp. 776–789
on meltdown-resistant cpus. In: ACM SIGSAC, pp. 769–784 (2020)
(2019) 49. Intel.: Intelő trust domain extensions. Intel Developer Zone (2023)
23. Van Schaik, S., Minkin, M., Kwong, A., Genkin, D., Yarom, Y.: CacheOut: 50. AMD.: AMD Secure Encrypted Virtualization (SEV). AMD Developer
Leaking data on Intel CPUs via cache evictions. In: IEEE, pp. 339–354 Portal (2021)
(2021) 51. ARM.: Learn the architecture - TrustZone for AArch64. ARM Documen-
24. Van Schaik, S., Kwong, A., Genkin, D., Yarom, Y.: SGAxe: How SGX tation (2021)
fails in practice (2020). https://sgaxe.com/files/SGAxe.pdf 52. ARM.: Arm Confidential Compute Architecture. (2021).
25. Tang, A., Sethumadhavan, S., Stolfo, S.: CLKSCREW: Exposing the perils https://www.arm.com/architecture/security-features/arm-
of security-oblivious energy management. In: USENIX Association, pp. confidential-compute-architecture
1057–1074 (2017) 53. Zhao, S., Zhang, Q., Qin, Y., Feng, W., Feng, D.: Sectee: A software-based
26. Murdock, K., Oswald, D., Garcia, F.D., Van Bulck, J., Gruss, D., Piessens, approach to secure enclave architecture using TEE. In: ACM SIGSAC,
F.: Plundervolt: Software-based fault injection attacks against Intel SGX. pp. 1723–1740 (2019)
In: IEEE, pp. 1466–1482 (2020) 54. Li, W., Xia, Y., Lu, L., Chen, H., Zang, B.: TEEv: Virtualizing trusted
27. Kenjar, Z., Frassetto, T., Gens, D., Franz, M., Sadeghi, A.R.: V0LTpwn: execution environments on mobile platforms. In: ACM, pp. 2–16 (2019)
Attacking x86 processor integrity from software. In: USENIX Associa- 55. Brasser, F., Gens, D., Jauernig, P., Sadeghi, A.R., Stapf, E.: SANCTUARY:
tion, pp. 1445–1461 (2020) ARMing TrustZone with User-space Enclaves. In: NDSS (2019)
28. Nilsson, A., Bideh, P.N., Brorsson, J.: A survey of published attacks on 56. Zhang, Y., Hu, Y., Ning, Z., et al.: SHELTER: Extending arm CCA with
Intel SGX. arXiv:2006.13598 (2020) isolation in user space. In: USENIX Association (2023)
29. Sasy, S., Gorbunov, S., Fletcher, C.W.: ZeroTrace: Oblivious memory 57. Samsung.: ISLET (2023). https://github.com/islet-project/islet
primitives from Intel SGX. Cryptology ePrint Archive (2017) 58. Li, X., Li, X., Dall, C., et al.: Design and verification of the arm con-
30. Seo, J., Lee, B., Kim, S.M., et al.: SGX-Shield: Enabling address space fidential compute architecture. In: USENIX Association, pp. 465–484
layout randomization for SGX programs. In: NDSS (2017) (2022)
31. Matetic, S., Ahmed, M., Kostiainen, K., et al.: ROTE: Rollback protection 59. Bourgeat, T., Lebedev, I., Wright, A., Zhang, S., Arvind, Devadas, S.: Mi6:
for trusted execution. In: USENIX Association, pp. 1289–1306 (2017) Secure enclaves in a speculative out-of-order processor. In: ACM, pp. 42–
32. Jangid, M.K., Chen, G., Zhang, Y., Lin, Z.: Towards formal verification 56 (2019)
of state continuity for enclave programs. In: USENIX Association, pp. 60. Drean, J., Gomez-Garcia, M., Bourgeat, T., Devadas, S.: Citadel: Side-
573–590 (2021) channel-resistant enclaves with secure shared memory on a speculative
33. Wang, W., Deng, S., Niu, J., Reiter, M.K., Zhang, Y.: Engraft: Enclave- out-of-order processor. arXiv:2306.14882 (2023)
guarded raft on Byzantine faulty nodes. In: ACM SIGSAC, pp. 2841–2855 61. Bahmani, R., Brasser, F., Dessouky, G., et al.: CURE: A security architec-
(2022) ture with Customizable and resilient enclaves. In: USENIX Association,
34. Niu, J., Peng, W., Zhang, X., Zhang, Y.: Narrator: Secure and practical pp. 1073–1090 (2021)
state continuity for trusted execution in the cloud. In: ACM SIGSAC, pp. 62. Feng, E., Lu, X., Du, D., et al.: Scalable memory protection in the
2385–2399 (2022) PENGLAI enclave. In: USENIX Association, pp. 275–294 (2021)
35. Chen, S., Lin, Z., Zhang, Y.: Controlled data races in enclaves: Attacks 63. Group, T.C., others.: TPM main specification level 2 version 1.2, revision
and detection. In: USENIX Association, pp. 4069–4086 (2023) 116 (2011). https://trustedcomputinggroup.org/wp-content/uploads/
36. Chen, G., Zhang, Y.: Securing TEEs with verifiable execution contracts. TPM-Main-Part-1-Design-Principles_v1.2_rev116_01032011.pdf
IEEE Trans. Dependable Secure Comput, pp. 3222–3237 (2022) 64. Information security technology–Functionality and interface specifica-
37. Chen, G., Wang, W., Chen, T., et al.: Racing in hyperspace: Closing hyper- tion of cryptographic support platform for trusted computing. standard,
threading side channels on SGX with contrived data races. In: IEEE, pp. Standardization Administration of the People’s Republic of China, China
178–194 (2018) (2013), GB/T 29829–2013
FENG ET AL. 555

65. Apple FIPS Secure Enclave Processor Secure Key Store Cryptographic 91. Chen, K., Deng, Q., Hou, Y., Jin, Y., Guo, X.: Hardware and software
Module v1.0. (2023) https://support.apple.com/zh-cn/HT208678 co-verification from security perspective. In: IEEE, pp. 50–55 (2019)
66. NVIDIA.: NVIDIA Confidential Computing (2023). https://www. 92. Knauth, T., Steiner, M., Chakrabarti, S., Lei, L., Xing, C., Vij, M.: Inte-
nvidia.com/en-us/data-center/solutions/confidential-computing/ grating remote attestation with transport layer security. arXiv:1801.05863
67. Volos, S., Vaswani, K., Bruno, R.: Graviton: Trusted execution environ- (2018)
ments on GPUs. In: USENIX Association, pp. 681–696 (2018) 93. Containers, I.: RATS-TLS (2021). https://github.com/inclavare-
68. Oh, H., Nam, K., Jeon, S., Cho, Y., Paek, Y.: MeetGo: A trusted execution containers/rats-tls
environment for remote applications on FPGA. IEEE Access 9, 51313– 94. Confidential 6G Consortium.: Confidential 6G (2023). https://
51324 (2021) confidential6g.eu/
69. Pereira, S., Cerdeira, D., Rodrigues, C., Pinto, S.: Towards a trusted 95. TrustedFirmware.org. OP-TEE Documentation (2023). https://optee.
execution environment via reconfigurable FPGA. arXiv:2107.03781 readthedocs.io/_/downloads/en/latest/pdf/
(2021) 96. Ahn, J., Lee, J., Ko, Y., et al.: DiskShield: A data tamper-resistant storage
70. Zeitouni, S., Vliegen, J., Frassetto, T., Koch, D., Sadeghi, A.R., Mentens, for Intel SGX. In: ACM, pp. 799–812 (2020)
N.: Trusted configuration in cloud FPGAs. In: IEEE, pp. 233–241 (2021) 97. Kumar, S., Sarangi, S.R.: SecureFS: A secure file system for Intel SGX. In:
71. Jang, I., Tang, A., Kim, T., Sethumadhavan, S., Huh, J.: Heteroge- ACM, pp. 91–102 (2021)
neous isolated execution for commodity gpus. In: ACM, pp. 455–468 98. Mofrad, S., Zhang, F., Lu, S., Shi, W.: A comparison study of intel SGX
(2019) and AMD memory encryption technology. In: ACM SIGSAC, pp. 1–8
72. Xia, K., Luo, Y., Xu, X., Wei, S.: SGX-FPGA: Trusted execution environ- (2018)
ment for CPU-FPGA heterogeneous architecture. In: IEEE, pp. 301–306 99. Hoang, T.T., Duran, C., Nguyen-Hoang, D.T., et al.: Quick boot of
(2021) trusted execution environment with hardware accelerators. IEEE Access
73. Deng, Y., Wang, C., Yu, S., et al.: Strongbox: A GPU TEE on arm 8, 74015–74023 (2020)
endpoints. In: ACM SIGSAC, pp. 769–783 (2022) 100. Shoufan, A.: An FPGA accelerator for hash tree generation in the Merkle
74. Jiang, J., Qi, J., Shen, T., et al.: CRONUS: Fault-isolated, secure signature scheme. In: Springer, pp. 145–156 (2010)
and high-performance heterogeneous computing for trusted execution 101. Mo, F., Shamsabadi, A.S., Katevas, K., et al.: Darknetz: towards model
environment. In: IEEE, pp. 124–143 (2022) privacy at the edge using trusted execution environments. In: ACM, pp.
75. Wang, C., Zhang, F., Deng, Y., et al.: CAGE: Complementing Arm CCA 161–174 (2020)
with GPU extensions. In: ISOC (2024) 102. Mo, F., Haddadi, H., Katevas, K., Marin, E., Perino, D., Kourtellis,
76. Zhu, J., Hou, R., Wang, X., et al.: Enabling rack-scale confidential com- N.: PPFL: Privacy-preserving federated learning with trusted execution
puting using heterogeneous trusted execution environment. In: IEEE, environments. In: ACM, pp. 94–108 (2021)
pp. 1450–1465 (2020) 103. Messaoud, A.A., Mokhtar, S.B., Nitu, V., Schiavoni, V.: GradSec: A TEE-
77. Intel.: Intel Trusted Execution Technology (TXT). (2023). based scheme against federated learning inference attacks. In: ACM, pp.
https://www.intel.cn/content/www/cn/zh/developer/articles/tool/ 10–12 (2021)
intel-trusted-execution-technology.html 104. Tramer, F., Boneh, D.: Slalom: Fast, verifiable and private execution of
78. AMD.: AMD and Microsoft secured-core PC. (2019). https:// neural networks in trusted hardware. arXiv:1806.03287 (2018)
community.amd.com/t5/business/amd-and-microsoft-secured-core- 105. Hashemi, H., Wang, Y., Annavaram, M.: DarKnight: An accelerated
pc/ba-p/418204 framework for privacy and integrity preserving deep learning using
79. Costan, V., Devadas, S.: Intel SGX explained. Cryptology ePrint Archive trusted hardware. In: IEEE/ACM, pp. 212–224 (2021)
(2016) 106. Ng, L.K., Chow, S.S., Woo, A.P., Wong, D.P., Zhao, Y.: Goten: GPU-
80. Wang, Z., Wu, C., Xie, M., et al.: SEIMI: Efficient and secure outsourcing trusted execution of neural network training. In: AAAI, pp.
smap-enabled intra-process memory isolation. In: IEEE, pp. 592–607 14876–14883 (2021)
(2020) 107. Chen, Y., Luo, F., Li, T., Xiang, T., Liu, Z., Li, J.: A training-integrity
81. Wu, C., Xie, M., Wang, Z., et al.: Dancing with wolves: An intra-process privacy-preserving federated learning scheme with trusted execution
isolation technique with privileged hardware. IEEE Trans. Dependable environment. Inform. Sci. 522, 69–79 (2020)
Secure Comput. pp. 1959–1978. (2022) 108. Brenna, L., Singh, I.S., Johansen, H.D., Johansen, D.: TFHE-rs: A
82. Birkholz, H., Thaler, D., Richardson, M., Smith, N., Pan, W.: library for safe and secure remote computing using fully homomorphic
Remote attestation procedures architecture. Internet-Draft draft-ietf-rats- encryption and trusted execution environments. Array 13, 100118 (2022)
architecture-18 (2022) 109. Fischer, A.: Computing on encrypted data using trusted execution
83. Intel.: Intel Enhanced Privacy ID (EPID) Security Technology environments. PhD thesis. University of Paderborn, Germany (2021)
(n.d.). https://www.intel.cn/content/www/cn/zh/developer/articles/ 110. Wu, P., Ning, J., Shen, J., Wang, H., Chang, E.C.: Hybrid trust multi-party
technical/intel-enhanced-privacy-id-epid-security-technology.html computation with trusted execution environment. In: ISOC (2022)
84. Scarlata, V., Johnson, S., Beaney, J., Zmijewski, P.: Supporting third party 111. Fisch, B., Vinayagamurthy, D., Boneh, D., Gorbunov, S.: Iron: Functional
attestation for Intel® SGX with Intel® data center attestation primitives. encryption using Intel SGX. In: ACM SIGSAC, pp. 765–782 (2017)
White Pap. 12 (2018) 112. Priebe, C., Vaswani, K., Costa, M.: EnclaveDB: A secure database using
85. Chen, G., Zhang, Y., Lai, T.H.: Opera: Open remote attestation for intel’s SGX. In: IEEE, pp. 264–278 (2018)
secure enclaves. In: ACM SIGSAC, pp. 2317–2331 (2019) 113. Antonopoulos, P., Arasu, A., Singh, K.D., et al.: Azure SQL database
86. Chen, G., Zhang, Y.: MAGE: Mutual attestation for a group of enclaves always encrypted. In: ACM SIGSAC, pp. 1511–1525 (2020)
without trusted third parties. In: USENIX Association, pp. 4095–4110 114. Wang, S., Li, Y., Li, H., et al.: Operon: An encrypted database
(2022) for ownership-preserving data management. Proc. VLDB Endowment
87. Azab, A.M., Ning, P., Shah, J., et al.: Hypervision across worlds: Real- 15(12), 3332–3345 (2022)
time kernel protection from the arm trustzone secure world. In: ACM 115. Zhu, J., Cheng, K., Liu, J., Guo, L.: Full Encryption: An end to end
SIGSAC, pp. 90–102 (2014) encryption mechanism in GaussDB. Proc. VLDB Endowment 14(12),
88. Schear, N., Cable, P.T., Moyer, T.M., Richard, B., Rudd, R.: Bootstrapping 2811–2814 (2021)
and maintaining trust in the cloud. In: ACM SIGSAC, pp. 65–77 (2016) 116. Cheng, R., Zhang, F., Kos, J., et al.: Ekiden: A platform for confidentiality-
89. Morbitzer, M., Kopf, B., Zieris, P.: GuaranTEE: Introducing control-flow preserving, trustworthy, and performant smart contracts. In: IEEE, pp.
attestation for trusted execution environments. In: IEEE, pp. 547–553 185–200 (2019)
(2023) 117. Russinovich, M., Ashton, E., Avanessians, C., et al.: CCF: A framework
90. Toffalini, F., Payer, M., Zhou, J., Cavallaro, L.: Designing a provenance for building confidential verifiable replicated services. Technical Report
analysis for SGX enclaves. In: IEEE, pp. 102–116 (2022) MSR-TR-201916 (2019)
556 FENG ET AL.

118. Lind, J., Eyal, I., Pietzuch, P., Sirer, E.G.: Teechan: Payment channels 129. Information Security Techniques - Trusted Execution Environment
using trusted execution environments. arXiv:1612.07766 (2016) Service Specification. standard, State Administration for Market Regu-
119. Yan, Y., Wei, C., Guo, X., et al.: Confidentiality support over financial lation and Standardization Administration of China, China (2023). GB/T
grade consortium blockchain. In: ACM SIGSAC, pp. 2227–2240 (2020) 42572-2023
120. Yuan, R., Xia, Y.B., Chen, H.B., Zang, B.Y., Xie, J.: Shadoweth: Private 130. Birkholz, H., Thaler, D., Richardson, M., Smith, N., Pan, W.: Remote
smart contract on public blockchain. J. Comput. Sci. Tech. 33, 542–556 attestation procedures architecture (2022). https://datatracker.ietf.org/
(2018) doc/pdf/draft-ietf-rats-architecture-22
121. Schuster, F., Costa, M., Fournet, C., et al.: VC3: Trustworthy data analytics 131. National Institute of Standards and Technology. Hardware-
in the cloud using SGX. In: IEEE, pp. 38–54 (2015) Enabled Security for Server Platforms: Enabling a Layered
122. Zheng, W., Dave, A., Beekman, J.G., Popa, R.A., Gonzalez, J.E., Stoica, I.: Approach to Platform Security for Cloud and Edge Comput-
Opaque: An oblivious and encrypted distributed analytics platform. In: ing Use Cases. NISTIR 8320, National Institute of Standards and
USENIX Association, pp. 283–298 (2017) Technology, U.S.
123. Information security technology–Functionality and interface specifica- 132. State Administration for Market Regulation and Standardization Admin-
tion of cryptographic support platform for trusted computing. standard, istration of China. Information Security Techniques - General Frame-
Standardization Administration of the People’s Republic of China, China work for Confidential Computing. Draft
(2022). GB/T 29829–2022 133. Halderman, J.A., Schoen, S.D., Heninger, N., et al.: Lest we remem-
124. ISO/IEC 11889-1:2009 Information technology-Trusted Platform ber: Cold-boot attacks on encryption keys. Commun. ACM 52(5), 91–98
Module-Part 1: Overview. standard, Trusted Computing Group (2009) (2009)
125. ISO/IEC 11889-1:2015 Information technology-Trusted Platform
Module-Part 1: Architecture. standard, Trusted Computing Group (2014)
126. OMTP Limited. Advanced Trusted Environment: OMTP TR1. tech. rep.,
Open Mobile Terminal Platform (2009)
How to cite this article: Feng, D., Qin, Y., Feng, W.,
127. GlobalPlatform. GlobalPlatform Technology TEE System Architec-
ture Version 1.3 (2022). https://globalplatform.org/specs-library/tee- Li, W., Shang, K., Ma, H.: Survey of research on
system-architecture/ confidential computing. IET Commun. 18, 535–556
128. Standard, State Administration for Market Regulation and Standardiza- (2024). https://doi.org/10.1049/cmu2.12759
tion Administration of China, China (2022). GB/T 41388-2022