Fix unintended manual flushing mode due to DataNucleus `ExecutionContext` pooling by nscuro · Pull Request #4221 · DependencyTrack/dependency-track

nscuro · 2024-10-04T17:54:06Z

Description

Fixes unintended manual flushing mode due to DataNucleus ExecutionContext pooling.

For BOM upload processing, we customize the PersistenceManager for better performance:

dependency-track/src/main/java/org/dependencytrack/tasks/BomUploadProcessingTask.java

Lines 231 to 259 in 6f7c49c

    
           // Disable reachability checks on commit. 
        
           // See https://www.datanucleus.org/products/accessplatform_4_1/jdo/performance_tuning.html 
        
           // 
        
           // Persistence-by-reachability is an expensive operation that involves traversing the entire 
        
           // object graph, and potentially issuing multiple database operations in doing so. 
        
           // 
        
           // It also enables cascading operations (both for persisting and deleting), but we don't need them here. 
        
           // If this circumstance ever changes, this property may be flicked to "true" again, at the cost of 
        
           // a noticeable performance hit. 
        
           // See: 
        
           //   https://www.datanucleus.org/products/accessplatform_6_0/jdo/persistence.html#cascading 
        
           //   https://www.datanucleus.org/products/accessplatform_6_0/jdo/persistence.html#_managing_relationships 
        
           qm.getPersistenceManager().setProperty(PROPERTY_PERSISTENCE_BY_REACHABILITY_AT_COMMIT, "false"); 
        
           // Save some database round-trips by flushing changes manually. 
        
           // See https://www.datanucleus.org/products/accessplatform_4_1/jdo/performance_tuning.html 
        
           // 
        
           // Note: Queries (SELECT) will always directly hit the database. Using manual flushing means 
        
           // changes made before flush are not visible to queries. If "read-your-writes" is critical, 
        
           // either flush immediately after making changes, or change the FlushMode back to AUTO (the default). 
        
           // AUTO will flush all changes to the database immediately, on every single setter invocation. 
        
           // 
        
           // Another option would be to set FlushMode to QUERY, where flushes will be performed before *any* 
        
           // query. Hibernate has a smart(er) behavior, where it checks if the query "touches" non-flushed 
        
           // data, and only flushes if that's the case. DataNucleus is not as smart, and will always flush. 
        
           // Still, QUERY may be a nice middle-ground between AUTO and MANUAL. 
        
           // 
        
           // BomUploadProcessingTaskTest#informWithBloatedBomTest can be used to profile the impact on large BOMs. 
        
           qm.getPersistenceManager().setProperty(PROPERTY_FLUSH_MODE, FlushMode.MANUAL.name());

It turns out that the custom options are internally delegated to a DataNucleus ExecutionContext. Other than PersistenceManagers, ExecutionContexts are never really closed, but instead reused (https://github.com/datanucleus/datanucleus-core/blob/master/src/main/java/org/datanucleus/ExecutionContextPool.java). Per default, DN pools up to 20 ECs.

The issue with this behavior is that customizing a PM in code location A, can unintentionally impact code location B. An example of this is project cloning, where an EC can get picked that has FlushMode=MANUAL, whereas the cloning logic assumes the default FlushMode=AUTO (#4220).

Disable EC pooling to prevent such unintended behavior changes.

Addressed Issue

Fixes #4220

Additional Details

N/A

Checklist

I have read and understand the contributing guidelines
This PR fixes a defect, and I have provided tests to verify that the fix is effective
~~This PR implements an enhancement, and I have provided tests to verify that it works as intended~~
~~This PR introduces changes to the database model, and I have added corresponding update logic~~
~~This PR introduces new or alters existing behavior, and I have updated the documentation accordingly~~

…ext` pooling For BOM upload processing, we customize the `PersistenceManager` for better performance: https://github.com/DependencyTrack/dependency-track/blob/6f7c49c0f2f163895b95e5b80554a49a3ec500f8/src/main/java/org/dependencytrack/tasks/BomUploadProcessingTask.java#L231-L259 It turns out that the custom options are internally delegated to a DataNucleus `ExecutionContext`. Other than `PersistenceManager`s, `ExecutionContext`s are never really closed, but instead reused (https://github.com/datanucleus/datanucleus-core/blob/master/src/main/java/org/datanucleus/ExecutionContextPool.java). Per default, DN pools up to 20 ECs. The issue with this behavior is that customizing a PM in code location A, can unintentionally impact code location B. An example of this is project cloning, where an EC can get picked that has `FlushMode=MANUAL`, whereas the cloning logic assumes the default `FlushMode=AUTO` (DependencyTrack#4220). Disable EC pooling to prevent such unintended behavior changes. Fixes DependencyTrack#4220 Signed-off-by: nscuro <[email protected]>

codacy-production · 2024-10-04T18:14:07Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.00% (target: -1.00%)	✅ ∅ (target: 70.00%)

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`6f7c49c`)	22460	17770	79.12%
Head commit (`5e56fda`)	22460 (+0)	17770 (+0)	79.12% (+0.00%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#4221)	0	0	∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

nscuro added defect Something isn't working backport/4.12.1 labels Oct 4, 2024

nscuro added this to the 4.13 milestone Oct 4, 2024

nscuro merged commit c98fc03 into DependencyTrack:master Oct 9, 2024

nscuro deleted the issue-4220 branch October 9, 2024 09:12

nscuro mentioned this pull request Oct 9, 2024

Backport: Fix unintended manual flushing mode due to DataNucleus ExecutionContext pooling #4233

Merged

2 tasks

github-actions bot locked as resolved and limited conversation to collaborators Nov 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix unintended manual flushing mode due to DataNucleus `ExecutionContext` pooling#4221

Fix unintended manual flushing mode due to DataNucleus `ExecutionContext` pooling#4221
nscuro merged 1 commit intoDependencyTrack:masterfrom
nscuro:issue-4220

nscuro commented Oct 4, 2024

Uh oh!

codacy-production bot commented Oct 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	// Disable reachability checks on commit.
	// See https://www.datanucleus.org/products/accessplatform_4_1/jdo/performance_tuning.html
	//
	// Persistence-by-reachability is an expensive operation that involves traversing the entire
	// object graph, and potentially issuing multiple database operations in doing so.
	//
	// It also enables cascading operations (both for persisting and deleting), but we don't need them here.
	// If this circumstance ever changes, this property may be flicked to "true" again, at the cost of
	// a noticeable performance hit.
	// See:
	// https://www.datanucleus.org/products/accessplatform_6_0/jdo/persistence.html#cascading
	// https://www.datanucleus.org/products/accessplatform_6_0/jdo/persistence.html#_managing_relationships
	qm.getPersistenceManager().setProperty(PROPERTY_PERSISTENCE_BY_REACHABILITY_AT_COMMIT, "false");

	// Save some database round-trips by flushing changes manually.
	// See https://www.datanucleus.org/products/accessplatform_4_1/jdo/performance_tuning.html
	//
	// Note: Queries (SELECT) will always directly hit the database. Using manual flushing means
	// changes made before flush are not visible to queries. If "read-your-writes" is critical,
	// either flush immediately after making changes, or change the FlushMode back to AUTO (the default).
	// AUTO will flush all changes to the database immediately, on every single setter invocation.
	//
	// Another option would be to set FlushMode to QUERY, where flushes will be performed before any
	// query. Hibernate has a smart(er) behavior, where it checks if the query "touches" non-flushed
	// data, and only flushes if that's the case. DataNucleus is not as smart, and will always flush.
	// Still, QUERY may be a nice middle-ground between AUTO and MANUAL.
	//
	// BomUploadProcessingTaskTest#informWithBloatedBomTest can be used to profile the impact on large BOMs.
	qm.getPersistenceManager().setProperty(PROPERTY_FLUSH_MODE, FlushMode.MANUAL.name());

Uh oh!

Conversation

nscuro commented Oct 4, 2024

Description

Addressed Issue

Additional Details

Checklist

Uh oh!

codacy-production bot commented Oct 4, 2024

Coverage summary from Codacy

See diff coverage on Codacy

See your quality gate settings Change summary preferences

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant