Skip to content

Static tramp v5#624

Merged
atgreen merged 5 commits intolibffi:masterfrom
madvenka786:static_tramp_v5
Mar 5, 2021
Merged

Static tramp v5#624
atgreen merged 5 commits intolibffi:masterfrom
madvenka786:static_tramp_v5

Conversation

@madvenka786
Copy link
Contributor

Fixed a couple of build problems. Also excluded cygwin from static trampolines. This is ready for review as soon as the CI tests pass.

Closure Trampoline Security Issue
=================================

Currently, the trampoline code used in libffi is not statically defined in
a source file (except for MACH). The trampoline is either pre-defined
machine code in a data buffer. Or, it is generated at runtime. In order to
execute a trampoline, it needs to be placed in a page with executable
permissions.

Executable data pages are attack surfaces for attackers who may try to
inject their own code into the page and contrive to have it executed. The
security settings in a system may prevent various tricks used in user land
to write code into a page and to have it executed somehow. On such systems,
libffi trampolines would not be able to run.

Static Trampoline
=================

To solve this problem, the trampoline code needs to be defined statically
in a source file, compiled and placed in the text segment so it can be
mapped and executed naturally without any tricks. However, the trampoline
needs to be able to access the closure pointer at runtime.

PC-relative data referencing
============================

The solution implemented in this patch set uses PC-relative data references.
The trampoline is mapped in a code page. Adjacent to the code page, a data
page is mapped that contains the parameters of the trampoline:

	- the closure pointer
	- pointer to the ABI handler to jump to

The trampoline code uses an offset relative to its current PC to access its
data.

Some architectures support PC-relative data references in the ISA itself.
E.g., X64 supports RIP-relative references. For others, the PC has to
somehow be loaded into a general purpose register to do PC-relative data
referencing. To do this, we need to define a get_pc() kind of function and
call it to load the PC in a desired register.

There are two cases:

1. The call instruction pushes the return address on the stack.

   In this case, get_pc() will extract the return address from the stack
   and load it in the desired register and return.

2. The call instruction stores the return address in a designated register.

   In this case, get_pc() will copy the return address to the desired
   register and return.

Either way, the PC next to the call instruction is obtained.

Scratch register
================

In order to do its job, the trampoline code would need to use a scratch
register. Depending on the ABI, there may not be a register available for
scratch. This problem needs to be solved so that all ABIs will work.

The trampoline will save two values on the stack:

	- the closure pointer
	- the original value of the scratch register

This is what the stack will look like:

	sp before trampoline ------>	--------------------
					| closure pointer  |
					--------------------
					| scratch register |
	sp after trampoline ------->	--------------------

The ABI handler can do the following as needed by the ABI:

	- the closure pointer can be loaded in a desired register

	- the scratch register can be restored to its original value

	- the stack pointer can be restored to its original value
	  (the value when the trampoline was invoked)

To do this, I have defined prolog code for each ABI handler. The legacy
trampoline jumps to the ABI handler directly. But the static trampoline
defined in this patch jumps tp the prolog code which performs the above
actions before jumping to the ABI handler.

Trampoline Table
================

In order to reduce the trampoline memory footprint, the trampoline code
would be defined as a code array in the text segment. This array would be
mapped into the address space of the caller. The mapping would, therefore,
contain a trampoline table.

Adjacent to the trampoline table mapping, there will be a data mapping that
contains a parameter table, one parameter block for each trampoline. The
parameter block will contain:

	- a pointer to the closure
	- a pointer to the ABI handler

The static trampoline code would finally look like this:

	- Make space on the stack for the closure and the scratch register
	  by moving the stack pointer down
	- Store the original value of the scratch register on the stack
	- Using PC-relative reference, get the closure pointer
	- Store the closure pointer on the stack
	- Using PC-relative reference, get the ABI handler pointer
	- Jump to the ABI handler

Mapping size
============

The size of the code mapping that contains the trampoline table needs to be
determined on a per architecture basis. If a particular architecture
supports multiple base page sizes, then the largest supported base page size
needs to be chosen. E.g., we choose 16K for ARM64.

Trampoline allocation and free
==============================

Static trampolines are allocated in ffi_closure_alloc() and freed in
ffi_closure_free().

Normally, applications use these functions. But there are some cases out
there where the user of libffi allocates and manages its own closure
memory. In such cases, static trampolines cannot be used. These will
fall back to using legacy trampolines. The user has to make sure that
the memory is executable.

ffi_closure structure
=====================

I did not want to make any changes to the size of the closure structure for
this feature to guarantee compatibility. But the opaque static trampoline
handle needs to be stored in the closure. I have defined it as follows:

-  char tramp[FFI_TRAMPOLINE_SIZE];
+  union {
+    char tramp[FFI_TRAMPOLINE_SIZE];
+    void *ftramp;
+  };

If static trampolines are used, then tramp[] is not needed to store a
dynamic trampoline. That space can be reused to store the handle. Hence,
the union.

Architecture Support
====================

Support has been added for x64, i386, aarch64 and arm. Support for other
architectures can be added very easily in the future.

OS Support
==========

Support has been added for Linux. Support for other OSes can be added very
easily.

Signed-off-by: Madhavan T. Venkataraman <[email protected]>
	- Define the arch-specific initialization function ffi_tramp_arch ()
	  that returns trampoline size information to common code.

	- Define the trampoline code mapping and data mapping sizes.

	- Define the trampoline code table statically. Define two tables,
	  actually, one with CET and one without.

	- Introduce a tiny prolog for each ABI handling function. The ABI
	  handlers addressed are:

	  	- ffi_closure_unix64
		- ffi_closure_unix64_sse
		- ffi_closure_win64

	  The prolog functions are called:

		- ffi_closure_unix64_alt
		- ffi_closure_unix64_sse_alt
		- ffi_closure_win64_alt

	  The legacy trampoline jumps to the ABI handler. The static
	  trampoline jumps to the prolog function. The prolog function uses
	  the information provided by the static trampoline, sets things up
	  for the ABI handler and then jumps to the ABI handler.

	- Call ffi_tramp_set_parms () in ffi_prep_closure_loc () to
	  initialize static trampoline parameters.

Signed-off-by: Madhavan T. Venkataraman <[email protected]>
	- Define the arch-specific initialization function ffi_tramp_arch ()
	  that returns trampoline size information to common code.

	- Define the trampoline code table statically. Define two tables,
	  actually, one with CET and one without.

	- Define the trampoline code table statically.

	- Introduce a tiny prolog for each ABI handling function. The ABI
	  handlers addressed are:

	  	- ffi_closure_i386
		- ffi_closure_STDCALL
		- ffi_closure_REGISTER

	  The prolog functions are called:

	  	- ffi_closure_i386_alt
		- ffi_closure_STDCALL_alt
		- ffi_closure_REGISTER_alt

	  The legacy trampoline jumps to the ABI handler. The static
	  trampoline jumps to the prolog function. The prolog function uses
	  the information provided by the static trampoline, sets things up
	  for the ABI handler and then jumps to the ABI handler.

	- Call ffi_tramp_set_parms () in ffi_prep_closure_loc () to
	  initialize static trampoline parameters.

Signed-off-by: Madhavan T. Venkataraman <[email protected]>
	- Define the arch-specific initialization function ffi_tramp_arch ()
	  that returns trampoline size information to common code.

	- Define the trampoline code mapping and data mapping sizes.

	- Define the trampoline code table statically.

	- Introduce a tiny prolog for each ABI handling function. The ABI
	  handlers addressed are:

	  	- ffi_closure_SYSV
		- ffi_closure_SYSV_V

	  The prolog functions are called:

	  	- ffi_closure_SYSV_alt
		- ffi_closure_SYSV_V_alt

	  The legacy trampoline jumps to the ABI handler. The static
	  trampoline jumps to the prolog function. The prolog function uses
	  the information provided by the static trampoline, sets things up
	  for the ABI handler and then jumps to the ABI handler.

	- Call ffi_tramp_set_parms () in ffi_prep_closure_loc () to
	  initialize static trampoline parameters.

Signed-off-by: Madhavan T. Venkataraman <[email protected]>
	- Define the arch-specific initialization function ffi_tramp_arch ()
	  that returns trampoline size information to common code.

	- Define the trampoline code mapping and data mapping sizes.

	- Define the trampoline code table statically.

	- Introduce a tiny prolog for each ABI handling function. The ABI
	  handlers addressed are:

	  	- ffi_closure_SYSV
		- ffi_closure_VFP

	  The prolog functions are called:

	  	- ffi_closure_SYSV_alt
		- ffi_closure_VFP_alt

	  The legacy trampoline jumps to the ABI handler. The static
	  trampoline jumps to the prolog function. The prolog function uses
	  the information provided by the static trampoline, sets things up
	  for the ABI handler and then jumps to the ABI handler.

	- Call ffi_tramp_set_parms () in ffi_prep_closure_loc () to
	  initialize static trampoline parameters.

Signed-off-by: Madhavan T. Venkataraman <[email protected]>
@atgreen
Copy link
Member

atgreen commented Feb 14, 2021

Can you comment on the thread-safety of your patch? ffi_tramp_init in particular. We punt responsibility for thread-safety to the calling program, but we should at least add a note to the thread-safety section of the documentation.

@atgreen
Copy link
Member

atgreen commented Feb 14, 2021

If I'm reading this patch right, it will try to use the static trampolines and fall back to the dynamic method if that fails. Why would static trampolines every not be available on a supported platform. I'm wondering if we can simply not include the old dynamic method.

@madvenka786
Copy link
Contributor Author

ffi_tramp_init() is a (static) internal function that is only called by ffi_tramp_is_supported() and ffi_tramp_alloc(). In both cases, the global static trampoline lock is acquired via ffi_tramp_lock(). In fact, all static trampoline operations acquire the lock. So, they are thread-safe.

@madvenka786
Copy link
Contributor Author

On the fallback question - the issue is in ffi_tramp_init_os (). This is the OS-specific initialization function. It needs to call OS-specific functions which can potentially fail. libffi must be resilient to that. E.g., let us consider loading the trampoline code in a temporary file and mapping it. Today, this is allowed. Tomorrow, an OS may require a valid file signature to allow mmap(file) with EXEC permissions. That will not work on temporary files.

@madvenka786
Copy link
Contributor Author

I have addressed all of the comments so far. Is there anything else I should address? If not, can these changes be merged into libffi?
Please let me know. Thanks!

@atgreen atgreen merged commit 9ba5592 into libffi:master Mar 5, 2021
@atgreen
Copy link
Member

atgreen commented Mar 5, 2021

Thank you. I have merged this great improvement to libffi. The testing all looks good, so I hope there will be no problems.

@madvenka786
Copy link
Contributor Author

madvenka786 commented Mar 5, 2021 via email

uintptr_t addr = (uintptr_t) tramp_globals.text;
int nfields, found;

snprintf (file, PATH_MAX, "/proc/%d/maps", getpid());
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi. I'm curious if there is some case where /proc/self/maps does not work here. If there is, a comment would have been useful.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maps has been around for a while and has remained stable AFAICT. However, I have assumed that it can fail. If it does, the code falls back to using a temporary file to map the trampoline table.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was curious enough to actually trace /proc/pid/maps to first appear in Linux 0.99.13k, but at that time it did not have the file name column. The file name column was at least in 2.4.12 if not even earlier. But I was thinking about use of snprintf and getpid being useless because can't you just open /proc/self/maps. Unless I miss something. /proc/self symlinks to getpid() so it should work the same. As it works the same, this is nothing major, just avoids any chance of mistakes with snprintf to stack. I checked /proc/self too, and it appeared in Linux-0.99 back in 1992.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. I could/should have used /proc/self/maps. That would have been more elegant.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. Thanks. My major concern was if there is something I did not know and you could tell me why it indeed makes sense to use getpid and snprintf.

@kohtala
Copy link

kohtala commented Mar 19, 2021

E.g., let us consider loading the trampoline code in a temporary file and mapping it. Today, this is allowed. Tomorrow, an OS may require a valid file signature to allow mmap(file) with EXEC permissions. That will not work on temporary files.

To add, it is not just about signatures. Today, some security conscious have all their executable files on read only filesystems. If any mount is writable, it can be noexec mounted and there is no executable mmap from it. Some mount only /tmp noexec.

@sempervictus
Copy link

Neat, thank you for the effort.
Far as /proc restrictions go - the Grsecurity patches have offered those for a long time, though up until pretty recently they could cause some serious confusion in userspace. LSMs and namespaces (systemd comes to mind, ditto container engines) may apply similar restrictions, so might be worthwhile to create a super-restricted systemd service using the new code to make sure it doesn't trip over itself when using foreign interfaces. If systemd's involved, humans will be likely just blame those hapless people since its basically fait accompli that they're at fault for everything around init nowadays.
For the LSM bit, i'd be most concerned with the Android SELinux implementation - they're starting to get up there with GrapheneOS as time goes on. Probaifbly merits testing as well - orgs like No Such Animal are heavy users, would not be great to break their stuff in a major foundational library.
On the subject to writing out mmap-able files, this is (IMO) a great idea since today FFI trips on RWX defenses (or did before this), and as more systems implement W^X, this may become more and more of a viable use-case. The noexec mount thing is becoming more common too, but that's actually a good thing as it should eventually lead to some new mountpoint for this specific use-case (some tmpfs style mounts per-user/per-pid to have semi-scary code run in a way other UIDs/PIDs cant reach). The discussion on signed ELF is quite old, and may bear fruit at some point, but probably not until were in next-gen OS territory at this rate.

@madvenka786
Copy link
Contributor Author

madvenka786 commented Mar 2, 2022 via email

@atgreen
Copy link
Member

atgreen commented Mar 7, 2022 via email

@madvenka786
Copy link
Contributor Author

madvenka786 commented Mar 7, 2022 via email

xry111 added a commit to xry111/libffi that referenced this pull request Jul 10, 2022
For the benefit and technical details of static trampoline, see
libffi#624.  As a new architecture, let's
be "safer" from the start.

The change survived libffi testsuite on loongarch64-linux-gnu.
atgreen pushed a commit that referenced this pull request Jul 21, 2022
For the benefit and technical details of static trampoline, see
#624.  As a new architecture, let's
be "safer" from the start.

The change survived libffi testsuite on loongarch64-linux-gnu.
madvenka786 added a commit to madvenka786/libffi that referenced this pull request Aug 8, 2022
* Static Trampolines

Closure Trampoline Security Issue
=================================

Currently, the trampoline code used in libffi is not statically defined in
a source file (except for MACH). The trampoline is either pre-defined
machine code in a data buffer. Or, it is generated at runtime. In order to
execute a trampoline, it needs to be placed in a page with executable
permissions.

Executable data pages are attack surfaces for attackers who may try to
inject their own code into the page and contrive to have it executed. The
security settings in a system may prevent various tricks used in user land
to write code into a page and to have it executed somehow. On such systems,
libffi trampolines would not be able to run.

Static Trampoline
=================

To solve this problem, the trampoline code needs to be defined statically
in a source file, compiled and placed in the text segment so it can be
mapped and executed naturally without any tricks. However, the trampoline
needs to be able to access the closure pointer at runtime.

PC-relative data referencing
============================

The solution implemented in this patch set uses PC-relative data references.
The trampoline is mapped in a code page. Adjacent to the code page, a data
page is mapped that contains the parameters of the trampoline:

	- the closure pointer
	- pointer to the ABI handler to jump to

The trampoline code uses an offset relative to its current PC to access its
data.

Some architectures support PC-relative data references in the ISA itself.
E.g., X64 supports RIP-relative references. For others, the PC has to
somehow be loaded into a general purpose register to do PC-relative data
referencing. To do this, we need to define a get_pc() kind of function and
call it to load the PC in a desired register.

There are two cases:

1. The call instruction pushes the return address on the stack.

   In this case, get_pc() will extract the return address from the stack
   and load it in the desired register and return.

2. The call instruction stores the return address in a designated register.

   In this case, get_pc() will copy the return address to the desired
   register and return.

Either way, the PC next to the call instruction is obtained.

Scratch register
================

In order to do its job, the trampoline code would need to use a scratch
register. Depending on the ABI, there may not be a register available for
scratch. This problem needs to be solved so that all ABIs will work.

The trampoline will save two values on the stack:

	- the closure pointer
	- the original value of the scratch register

This is what the stack will look like:

	sp before trampoline ------>	--------------------
					| closure pointer  |
					--------------------
					| scratch register |
	sp after trampoline ------->	--------------------

The ABI handler can do the following as needed by the ABI:

	- the closure pointer can be loaded in a desired register

	- the scratch register can be restored to its original value

	- the stack pointer can be restored to its original value
	  (the value when the trampoline was invoked)

To do this, I have defined prolog code for each ABI handler. The legacy
trampoline jumps to the ABI handler directly. But the static trampoline
defined in this patch jumps tp the prolog code which performs the above
actions before jumping to the ABI handler.

Trampoline Table
================

In order to reduce the trampoline memory footprint, the trampoline code
would be defined as a code array in the text segment. This array would be
mapped into the address space of the caller. The mapping would, therefore,
contain a trampoline table.

Adjacent to the trampoline table mapping, there will be a data mapping that
contains a parameter table, one parameter block for each trampoline. The
parameter block will contain:

	- a pointer to the closure
	- a pointer to the ABI handler

The static trampoline code would finally look like this:

	- Make space on the stack for the closure and the scratch register
	  by moving the stack pointer down
	- Store the original value of the scratch register on the stack
	- Using PC-relative reference, get the closure pointer
	- Store the closure pointer on the stack
	- Using PC-relative reference, get the ABI handler pointer
	- Jump to the ABI handler

Mapping size
============

The size of the code mapping that contains the trampoline table needs to be
determined on a per architecture basis. If a particular architecture
supports multiple base page sizes, then the largest supported base page size
needs to be chosen. E.g., we choose 16K for ARM64.

Trampoline allocation and free
==============================

Static trampolines are allocated in ffi_closure_alloc() and freed in
ffi_closure_free().

Normally, applications use these functions. But there are some cases out
there where the user of libffi allocates and manages its own closure
memory. In such cases, static trampolines cannot be used. These will
fall back to using legacy trampolines. The user has to make sure that
the memory is executable.

ffi_closure structure
=====================

I did not want to make any changes to the size of the closure structure for
this feature to guarantee compatibility. But the opaque static trampoline
handle needs to be stored in the closure. I have defined it as follows:

-  char tramp[FFI_TRAMPOLINE_SIZE];
+  union {
+    char tramp[FFI_TRAMPOLINE_SIZE];
+    void *ftramp;
+  };

If static trampolines are used, then tramp[] is not needed to store a
dynamic trampoline. That space can be reused to store the handle. Hence,
the union.

Architecture Support
====================

Support has been added for aarch64. Support for other architectures can be
added very easily in the future.

OS Support
==========

Support has been added for Linux. Support for other OSes can be added very
easily.

* arm64: Support for Static Trampolines

	- Define the arch-specific initialization function ffi_tramp_arch ()
	  that returns trampoline size information to common code.

	- Define the trampoline code mapping and data mapping sizes.

	- Define the trampoline code table statically.

	- Introduce a tiny prolog for each ABI handling function. The ABI
	  handlers addressed are:

	  	- ffi_closure_SYSV
		- ffi_closure_SYSV_V

	  The prolog functions are called:

	  	- ffi_closure_SYSV_alt
		- ffi_closure_SYSV_V_alt

	  The legacy trampoline jumps to the ABI handler. The static
	  trampoline jumps to the prolog function. The prolog function uses
	  the information provided by the static trampoline, sets things up
	  for the ABI handler and then jumps to the ABI handler.

	- Call ffi_tramp_set_parms () in ffi_prep_closure_loc () to
	  initialize static trampoline parameters.

Signed-off-by: Madhavan T. Venkataraman <[email protected]>
Assioftware added a commit to Assioftware/libffi that referenced this pull request Feb 5, 2023
For the benefit and technical details of static trampoline, see
libffi/libffi#624.  As a new architecture, let's
be "safer" from the start.

The change survived libffi testsuite on loongarch64-linux-gnu.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants