Introduce the transient files engine #41055

Konamiman · 2023-10-27T07:00:45Z

Changes proposed in this Pull Request:

This pull request introduces a transient files engine intended for registering files that have an expiration date and can be optionally served via a public, unauthenticated HTTP URL. Initially, the only mechanism provided to create these files is rendering them from a template file (available template files are part of the WooCommerce codebase, additional templates can be used via hook) consisting of a mix of literal text and PHP code; but additional mechanisms could be added in the future, such as directly providing binary content.

See the README file and the documentation comments in the TransientFilesEngine class for all the nitty-gritty details.

How to test the changes in this Pull Request:

Initial setup

Either perform a clean install of WooCommerce with this code applied, or perform the following manual steps in an already existing site:

Create the database tables, you have the schema in TemplatingEngine::get_database_schema.
Create the directory for the transient files and trigger the scheduled action to periodically delete expired files: wp eval "include WP_PLUGIN_DIR . '/woocommerce/includes/wc-update-functions.php'; wc_update_850_create_transient_files_directory();"
Flush the rewrite rules for the new rendering endpoint to be recognized: wp eval flush_rewrite_rules());'
Recreate the WooCommerce user roles, this is needed so that administrators and shop managers get the capabilities needed for the JSON REST API endpoints: wp eval 'WC_Install::create_roles();'

Testing the code API

Run the following:

wp eval "echo wc_get_container()->get(\Automattic\WooCommerce\TransientFiles\TransientFilesEngine::class)->create_file_by_rendering_template('examples/show_variables', ['foo'=>'bar', 'fizz'=>'buzz']);"

You will see HTML content in your console. If you want to see the content in a browser, append > example.html to the command and open the file in the browser. The "source" (the template file with code and markup) is at src/TransientFiles/Templates/examples/show_variables.template.

Now run the following:

wp eval "echo wc_get_container()->get(\Automattic\WooCommerce\TransientFiles\TransientFilesEngine::class)->create_file_by_rendering_template('examples/show_variables', ['foo'=>'bar', 'fizz'=>'buzz'], ['expiration_date'=>'2034-01-01 00:00:00']);"

This time a file and a database entry have been created, and what you get is the random name assigned to the generated file. The file itself will be at wp-content/uploads/woocommerce_transient_files/2034-01-01 (yes, that's the expiration date) and a row will have been added to the wp_wc_transient_files table.

Run the following to see information about the generated file:

wp eval "var_export(wc_get_container()->get(\Automattic\WooCommerce\TransientFiles\TransientFilesEngine::class)->get_file_by_name('xxxxx...'));"

where xxxxx... is the name of the file that you got in step 2. There's also a get_file_by_id method taking the database id.

Run the following to delete the generated file:

wp eval "var_export(wc_get_container()->get(\Automattic\WooCommerce\TransientFiles\TransientFilesEngine::class)->delete_file_by_name('xxxxx...'));"

Now if you repeat step 4 with the same file name you'll get null. Again, there's a delete_rendered_file_by_id method too.

Testing the expired files deletion

To easily test the expired files deletion mechanism we need to be able to create transient files that have already expired. This is not possible out of the box, but can be achieved by using the expiration seconds filter with a negative value:

add_filter('woocommerce_transient_files_minimum_expiration_seconds', fn($seconds) => -100000);

After adding that code to your WooCommerce install, go ahead and create 10 files that expired 24h ago by repeating the following command 10 times:

wp eval "echo wc_get_container()->get(\Automattic\WooCommerce\TransientFiles\TransientFilesEngine::class)->create_file_by_rendering_template('examples/show_variables', ['foo'=>'bar', 'fizz'=>'buzz'], ['expiration_seconds'=>-86400]);"

The wp_wc_transient_files table should have 10 rows now (additionally to any rows created by your previous tests), and the /wordpress/wp-content/uploads/woocommerce-transient-files directory should have a subdirectory that represents the date of yesterday and contains 10 files.

Now run the following:

wp eval "var_export(wc_get_container()->get(\Automattic\WooCommerce\TransientFiles\TransientFilesEngine::class)->delete_expired_files(7));"

The result will be:

array (
  'files_deleted' => 7,
  'rows_deleted' => 0,
)

and there will be only 3 files remaining, but the database table will be unchanged.

Running the same command again will yield the following results:

'files_deleted' => 3, 'rows_deleted' => 7: The directory representing the date of yesterday is gone, 3 rows remain in the database table.
'files_deleted' => 0, 'rows_deleted' => 3: The database table is now empty.
'files_deleted' => 0, 'rows_deleted' => 0: Nothing left to delete.

You can also try the scheduled action by going to WooCommerce - Status - Tools and re-scheduling the cleanup:

By rescheduling the cleanup the action will be scheduled to keep running immediately as long as there's still data to delete, and then every 24h. You can change that interval with a filter, e.g. every 10 seconds:

add_filter('woocommerce_delete_expired_transient_files_interval', fn($interval) => 10);

Note however that the scheduled action deletes 1000 files every time, so for testing you'll need to either create as many transient files or tweak the code to use a smaller batch (pass the batch size to the delete_expired_files call in TransientFilesEngine::handle_expired_files_cleanup_action).

Testing the rendering endpoint

Generate a public rendered file by repeating step 1 but making the file public:

wp eval "echo wc_get_container()->get(\Automattic\WooCommerce\Templating\TemplatingEngine::class)->render_template('examples/show_variables', ['foo'=>'bar', 'fizz'=>'buzz'], ['expiration_date'=>'2034-01-01 00:00:00', 'is_public'=>true]);"

Verify that you can access the rendered file in your browser or using any tool able to perform HTTP requests (replace <filename> with the file name that you get in the previous step): http://localhost/wc/file/transient/<filename>.
Repeat 8 and 9 but add an extra 'content-type' => 'text/plain' to the metadata array, next to 'is_public'. This time the file will be served as plain text.

Testing the JSON REST API endpoints

Assuming that your server allows basic authentication (use Postman or a similar tool with OAuth authentication instead otherwise):

Run the following in your terminal, setting the appropriate user and password (user must be an administrator or a shop manager):

curl -X POST http://localhost/wp-json/wc/v3/files/transient/render \
   -u user:password \
   -H 'Content-Type: application/json' \
   -d '{ "template_name": "examples/show_variables", "expiration_seconds": 10000, "is_public": true, "variables": { "hello": "...from REST!" }, "metadata": { "foo": "bar" } }'

Take note of the file_name in the response and verify that you can render it in your browser: http://localhost/wc/file/transient/<filename>.
Take note of the id in the response and run the following, you'll get the same response as when you hit the wc/file/transient endpoint:

curl http://localhost/wp-json/wc/v3/files/transient/<id> -u user:password

Run the following, you'll get the same information that you got when the file was created:

curl http://localhost/wp-json/wc/v3/files/transient/<id>/info -u user:password

Run the following, you'll get the same information but this time including the file metadata:

curl http://localhost/wp-json/wc/v3/files/transient/<id>/info?include_metadata=true -u user:password

Run the following to delete the rendered file:

curl -X DELETE http://localhost/wp-json/wc/v3/files/transient/<id> -u user:password

Running it again will yield {"deleted":false} instead of {"deleted":true}.

Repeat step 11 but this time setting is_public to false (or removing the is_public key). Now repeating step 13 will still return the file contents, but step 12 will return a "not found" error.

Changelog entry

Automatically create a changelog entry from the details below.

Details

Significance

Patch
Minor
Major

Type

Fix - Fixes an existing bug
Add - Adds functionality
Update - Update existing functionality
Dev - Development related task
Tweak - A minor adjustment to the codebase
Performance - Address performance issues
Enhancement - Improvement to existing functionality

Message

Comment

github-actions · 2023-10-27T07:43:59Z

Test Results Summary

Commit SHA: 3c76aa9

Test 🧪	Passed ✅	Failed 🚨	Broken 🚧	Skipped ⏭️	Unknown ❔	Total 📊	Duration ⏱️
API Tests	259	0	0	2	0	261	0m 35s
E2E Tests	240	1	2	4	0	247	13m 39s

⚠️ Warning

Please address the following issues prior to merging this pull request:

FAILED/BROKEN TESTS. There were failed and/or broken API and E2E tests.
INCOMPLETE E2E TEST RUN. We have a total of 255 E2E tests, but only 247 were executed. Note that in CI, E2E tests automatically end when they encounter too many failures.

To view the full API test report, click here.
To view the full E2E test report, click here.
To view all test reports, visit the WooCommerce Test Reports Dashboard.

coreymckrill · 2023-11-06T23:05:35Z

plugins/woocommerce/src/Templating/TemplatingEngine.php

+		$render_to_file = ! is_null( $metadata );
+		$render_ok      = false;
+
+		// phpcs:disable WordPress.WP.AlternativeFunctions


I think you might be able to do some, if not all, of the file operations in this method using WP_Filesystem, which is really just a wrapper around a lot of the same functions. The advantage is that is does some error handling for you, and handles some gotchas related to the quirks of varying file systems on different servers.

You just need to start by making sure the file system global is initialized, and then you can call its methods. Here are the ones that I think would pertain to render_template:

mkdir

delete

put_contents

I see that WP_Filesystem::mkdir doesn't have the $recursive argument, which is a problem since I need to create a directory like base/expiration. Also I see that WP_Filesystem::mkdir is just resorting to the PHP built-in mkdir after all.

Edit: If found this, though, which seems useful since it creates the new directory with the permissions of the parent directory: https://developer.wordpress.org/reference/functions/wp_mkdir_p/

As for put_contents, I can't use it. I need to open the file and use the file handle in the callback that is passed to ob_start. And as for delete, for an empty directory it just does an rmdir; but for files I could use wp_delete_file (https://developer.wordpress.org/reference/functions/wp_delete_file/).

Well, instead of giving the output buffer a callback, you could just do something like this:

ob_start(); // output stuff $content = ob_get_clean(); $wp_filesystem->put_contents( $file_path, $content );

We don't know how big the rendered file will be (or put another way: we can figure it out by looking the file size of the template, but it could be arbitrarily large) so it's better to process the output in chunks (letting the ob_* engine handle buffering by itself). That's also why in TemplatingRestController::handle_parse_request I'm sending the rendered file in chunks of 1KByte.

coreymckrill · 2023-11-06T23:27:50Z

plugins/woocommerce/src/Templating/TemplatingEngine.php

+	 * @return array|null Rendered template information array, or null if no entry exists with the supplied id.
+	 * @throws Exception The base directory for rendered files (possibly changed via filter) doesn't exist, or error deleting the file.
+	 */
+	private function get_rendered_file_core( string $sql_query, bool $include_metadata, bool $delete_if_expired ): ?array {


It doesn't look like this method checks to see if the file actually exists in the filesystem. What happens if there is a database entry for a file, but the file is missing (because of some external file operation or something)?

I think the method has a bad name, it should rather be get_rendered_file_information: it just returns the (processed) database entry for the file, including the full path where the physical file is supposed to be located. It's the responsibility of the caller to ensure that the file actually exists, which is exactly what the handle_parse_request method does when the file is requested via the dedicated REST API endpoint.

Three endpoints are provided: POST /templates/render GET /templates/<id> (retrieves rendered file metadata) DELETE /templates/<id> Three new capabilities are created for these, and assigned to administrators and shop managers in WC_Install::create_roles: create_rendered_template read_rendered_template_info delete_rendered_template Also, the endpoint to retrieve rendered file contents has been moved from TemplatingEngine to this new class.

lkraav · 2023-11-08T10:51:49Z

Hi all. What business initiative is this PR looking to support? Not able to find any links or anything into a "bigger picture" artifact.

Blocks weren't very useful anyway since code inside blocks was being rendered at block definition time, not at block rendering time; reusable blocks can be created anyway by putting the relevant code+markup inside a variable and 'eval'-ing it.

coreymckrill · 2023-11-08T23:54:04Z

plugins/woocommerce/src/Templating/TemplatingRestController.php

+		register_rest_route(
+			$this->route_namespace,
+			'/templates/(?P<id>[\d]+)',
+			array(
+				'methods'             => WP_REST_Server::READABLE,
+				'callback'            => fn( $request ) => $this->run( 'get_template', $request ),
+				'permission_callback' => fn( $request ) => $this->check_permission( $request, 'read_rendered_template_info' ),
+				'args'                => $this->get_args_for_get_or_delete_template(),
+			)
+		);
+
+		register_rest_route(
+			$this->route_namespace,
+			'/templates/(?P<id>[\d]+)',
+			array(
+				'methods'             => WP_REST_Server::DELETABLE,
+				'callback'            => fn( $request ) => $this->run( 'delete_template', $request ),
+				'permission_callback' => fn( $request ) => $this->check_permission( $request, 'delete_rendered_template' ),
+				'args'                => $this->get_args_for_get_or_delete_template(),
+			)
+		);


Since these have the same namespace and route, they can be combined into one register_rest_route call, like this:

Suggested change

register_rest_route(

$this->route_namespace,

'/templates/(?P<id>[\d]+)',

array(

'methods' => WP_REST_Server::READABLE,

'callback' => fn( $request ) => $this->run( 'get_template', $request ),

'permission_callback' => fn( $request ) => $this->check_permission( $request, 'read_rendered_template_info' ),

'args' => $this->get_args_for_get_or_delete_template(),

)

);

register_rest_route(

$this->route_namespace,

'/templates/(?P<id>[\d]+)',

array(

'methods' => WP_REST_Server::DELETABLE,

'callback' => fn( $request ) => $this->run( 'delete_template', $request ),

'permission_callback' => fn( $request ) => $this->check_permission( $request, 'delete_rendered_template' ),

'args' => $this->get_args_for_get_or_delete_template(),

)

);

register_rest_route(

$this->route_namespace,

'/templates/(?P<id>[\d]+)',

array(

array(

'methods' => WP_REST_Server::READABLE,

'callback' => fn( $request ) => $this->run( 'get_template', $request ),

'permission_callback' => fn( $request ) => $this->check_permission( $request, 'read_rendered_template_info' ),

'args' => $this->get_args_for_get_or_delete_template(),

),

array(

'methods' => WP_REST_Server::DELETABLE,

'callback' => fn( $request ) => $this->run( 'delete_template', $request ),

'permission_callback' => fn( $request ) => $this->check_permission( $request, 'delete_rendered_template' ),

'args' => $this->get_args_for_get_or_delete_template(),

),

)

);

It would also be good to have a schema property for the route, like we do for other endpoints, e.g. Orders.

Konamiman · 2023-11-09T07:28:33Z

Hi @lkraav, the trigger for this project was the need to render receipts to be sent via public URL to a printing service from one of our mobile apps. We considered that implementing a generic solution would need about the same effort than implementing an ad-hoc solution for that specific use case.

The action is scheduled to run immediately. After the cleanup happens, if there are more files pending deletion a new action is scheduled to run immediately, otherwise a new action is scheduled to run in 24h (but this can be adjusted with a new filter, woocommerce_delete_expired_rendered_template_files_interval). Also add two new debug tools, one to: (re)schedule the action and another one to unschedule it.

This endpoint serves the raw rendered file contents as the existing /wc/file/name endpoint does, except that it works even when the rendered file isn't marked as public. Also the 'read_rendered_template' capability is renamed to just 'read_rendered_template'.

Also some minor adjustments for the TemplatingEngine class.

Now the REST API endpoint always requires authentication but doesn't look at 'is_public' of the file, while the unauthenticated endpoint only looks at 'is_public' and not to authentication info.

REST endpoints have been renamed too.

Also add a mechanism for mocking the "exit" construct in LegacyProxy.

Konamiman · 2023-11-28T09:25:32Z

@coreymckrill Please ignore the failing unit tests for now (unless you know how to fix them of course!), these are due to the migration targetting WooCommerce 8.5 (they disappear if the migration targets 8.4 instead, but we're past the code freeze date for that version).

github-actions · 2023-11-28T09:25:56Z

Hi @coreymckrill,

Apart from reviewing the code changes, please make sure to review the testing instructions as well.

You can follow this guide to find out what good testing instructions should look like:
https://github.com/woocommerce/woocommerce/wiki/Writing-high-quality-testing-instructions

Konamiman · 2023-12-15T16:16:53Z

Closed in favor of #42877 which is a simplified version.

github-actions bot added the plugin: woocommerce Issues related to the WooCommerce Core plugin. label Oct 27, 2023

Konamiman force-pushed the add_template_engine branch 9 times, most recently from 70f1841 to f3980df Compare November 3, 2023 11:33

Konamiman force-pushed the add_template_engine branch 2 times, most recently from 0ecdce4 to bc5d554 Compare November 3, 2023 12:07

Konamiman added 2 commits November 6, 2023 12:54

Add the TemplatingEngine and TemplateRenderingContext classes.

d1f1944

Add the "wc/file/name" endpoint to serve public rendered files.

a41c138

Konamiman force-pushed the add_template_engine branch from f848ccc to a41c138 Compare November 6, 2023 11:54

coreymckrill reviewed Nov 6, 2023

View reviewed changes

Konamiman added 2 commits November 7, 2023 16:57

Use wp_mkdir_p instead of mkdir, and wp_delete_file instead of unlink

673526e

Remove support for reusable blocks.

cd83473

Blocks weren't very useful anyway since code inside blocks was being rendered at block definition time, not at block rendering time; reusable blocks can be created anyway by putting the relevant code+markup inside a variable and 'eval'-ing it.

coreymckrill reviewed Nov 8, 2023

View reviewed changes

Konamiman added 3 commits November 9, 2023 12:29

Join two 'register_rest_route's into a single one.

053a756

Konamiman force-pushed the add_template_engine branch 2 times, most recently from ba7e51a to 2d43b65 Compare November 15, 2023 12:00

Konamiman added 2 commits November 15, 2023 13:38

Add the TemplatingEngineTest class.

71765b5

Also some minor adjustments for the TemplatingEngine class.

DI container registrations moved to a dedicated service provider class.

01684a1

Konamiman added 6 commits November 16, 2023 12:58

Refactor on the template serving endpoints.

9951575

Now the REST API endpoint always requires authentication but doesn't look at 'is_public' of the file, while the unauthenticated endpoint only looks at 'is_public' and not to authentication info.

Rename the entire thing to "Transient Files Engine".

dfe2944

REST endpoints have been renamed too.

Add a migration to create the transient files directory.

3aaa8e5

A couple of hooks have been refactored.

01edb4a

Add schema information to the REST API endpoints definition.

2ea079f

Change the 'since 8.4.0's to 8.5.0

e3090c1

Konamiman force-pushed the add_template_engine branch 2 times, most recently from 1b557f2 to 90abddc Compare November 20, 2023 12:03

Konamiman changed the title ~~Introduce the templating engine~~ Introduce the transient files engine Nov 20, 2023

Add the README file for the transient files engine.

95660a1

Konamiman force-pushed the add_template_engine branch from 90abddc to 95660a1 Compare November 21, 2023 09:36

Konamiman added 2 commits November 21, 2023 12:12

Trigger the expired files deletion action as part of WC install

ebbc7f6

Add unit tests for the TransientFilesRestController class.

ef218ab

Also add a mechanism for mocking the "exit" construct in LegacyProxy.

Konamiman force-pushed the add_template_engine branch from 0ab6319 to ef218ab Compare November 24, 2023 12:36

Konamiman added 8 commits November 24, 2023 16:32

Merge branch 'trunk' into add_template_engine

02d815d

Change '-' to '*' in lists in README (because linters)

e6c168a

Yet another change in the README for the linter

44f4691

Add changelog file

fe911fc

Fix base url of the site in unit tests

1471bdf

Use LegacyProxy for 'get_site_url'

e5c63c8

Merge branch 'trunk' into add_template_engine

760063d

Fix db schema definition (tinyint --> tinyint(1))

3c76aa9

Konamiman marked this pull request as ready for review November 28, 2023 09:23

Konamiman requested a review from coreymckrill November 28, 2023 09:25

Konamiman mentioned this pull request Dec 15, 2023

Introduce the transient files engine #42877

Merged

11 tasks

Konamiman closed this Dec 15, 2023

kalessil deleted the add_template_engine branch February 14, 2025 07:56

Introduce the transient files engine #41055

Introduce the transient files engine #41055

Uh oh!

Conversation

Konamiman commented Oct 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes proposed in this Pull Request:

How to test the changes in this Pull Request:

Initial setup

Testing the code API

Testing the expired files deletion

Testing the rendering endpoint

Testing the JSON REST API endpoints

Changelog entry

Significance

Type

Message

Comment

Uh oh!

github-actions bot commented Oct 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results Summary

⚠️ Warning

Uh oh!

coreymckrill Nov 6, 2023

Choose a reason for hiding this comment

Uh oh!

Konamiman Nov 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Konamiman Nov 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coreymckrill Nov 8, 2023

Choose a reason for hiding this comment

Uh oh!

Konamiman Nov 9, 2023

Choose a reason for hiding this comment

Uh oh!

coreymckrill Nov 6, 2023

Choose a reason for hiding this comment

Uh oh!

Konamiman Nov 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lkraav commented Nov 8, 2023

Uh oh!

coreymckrill Nov 8, 2023

Choose a reason for hiding this comment

Uh oh!

Konamiman commented Nov 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Konamiman commented Nov 28, 2023

Uh oh!

github-actions bot commented Nov 28, 2023

Uh oh!

Konamiman commented Dec 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Konamiman commented Oct 27, 2023 •

edited

Loading

github-actions bot commented Oct 27, 2023 •

edited

Loading

Konamiman Nov 8, 2023 •

edited

Loading

Konamiman Nov 8, 2023 •

edited

Loading

Konamiman Nov 7, 2023 •

edited

Loading

Konamiman commented Nov 9, 2023 •

edited

Loading

Konamiman commented Dec 15, 2023 •

edited

Loading