Optimize vulnerability synchronization logic to not perform redundant writes by LaVibeX · Pull Request #4359 · DependencyTrack/dependency-track

LaVibeX · 2024-11-07T14:29:47Z

Description

This modification is intended to avoid multiple updates/writes in the database when there are no changes. In general we want to reduce unnecessary load on the DB.

Part of the logic in updateVulnerability was optimized. The differ.applyIfChanged method was added, similar to what NIST is doing

dependency-track/src/main/java/org/dependencytrack/tasks/AbstractNistMirrorTask.java

Lines 222 to 257 in 8087a94

    
           private static Map<String, PersistenceUtil.Diff> updateVulnerability(final Vulnerability existingVuln, final Vulnerability reportedVuln) { 
        
               assertPersistent(existingVuln, "existingVuln must be persistent in order for changes to be effective"); 
        
               final var differ = new PersistenceUtil.Differ<>(existingVuln, reportedVuln); 
        
               differ.applyIfChanged("title", Vulnerability::getTitle, existingVuln::setTitle); 
        
               differ.applyIfChanged("subTitle", Vulnerability::getSubTitle, existingVuln::setSubTitle); 
        
               differ.applyIfChanged("description", Vulnerability::getDescription, existingVuln::setDescription); 
        
               differ.applyIfChanged("detail", Vulnerability::getDetail, existingVuln::setDetail); 
        
               differ.applyIfChanged("recommendation", Vulnerability::getRecommendation, existingVuln::setRecommendation); 
        
               differ.applyIfChanged("references", Vulnerability::getReferences, existingVuln::setReferences); 
        
               differ.applyIfChanged("credits", Vulnerability::getCredits, existingVuln::setCredits); 
        
               differ.applyIfChanged("created", Vulnerability::getCreated, existingVuln::setCreated); 
        
               differ.applyIfChanged("published", Vulnerability::getPublished, existingVuln::setPublished); 
        
               differ.applyIfChanged("updated", Vulnerability::getUpdated, existingVuln::setUpdated); 
        
               differ.applyIfNonEmptyAndChanged("cwes", Vulnerability::getCwes, existingVuln::setCwes); 
        
               differ.applyIfChanged("severity", Vulnerability::getSeverity, existingVuln::setSeverity); 
        
               differ.applyIfChanged("cvssV2BaseScore", Vulnerability::getCvssV2BaseScore, existingVuln::setCvssV2BaseScore); 
        
               differ.applyIfChanged("cvssV2ImpactSubScore", Vulnerability::getCvssV2ImpactSubScore, existingVuln::setCvssV2ImpactSubScore); 
        
               differ.applyIfChanged("cvssV2ExploitabilitySubScore", Vulnerability::getCvssV2ExploitabilitySubScore, existingVuln::setCvssV2ExploitabilitySubScore); 
        
               differ.applyIfChanged("cvssV2Vector", Vulnerability::getCvssV2Vector, existingVuln::setCvssV2Vector); 
        
               differ.applyIfChanged("cvssV3BaseScore", Vulnerability::getCvssV3BaseScore, existingVuln::setCvssV3BaseScore); 
        
               differ.applyIfChanged("cvssV3ImpactSubScore", Vulnerability::getCvssV3ImpactSubScore, existingVuln::setCvssV3ImpactSubScore); 
        
               differ.applyIfChanged("cvssV3ExploitabilitySubScore", Vulnerability::getCvssV3ExploitabilitySubScore, existingVuln::setCvssV3ExploitabilitySubScore); 
        
               differ.applyIfChanged("cvssV3Vector", Vulnerability::getCvssV3Vector, existingVuln::setCvssV3Vector); 
        
               differ.applyIfChanged("owaspRRLikelihoodScore", Vulnerability::getOwaspRRLikelihoodScore, existingVuln::setOwaspRRLikelihoodScore); 
        
               differ.applyIfChanged("owaspRRTechnicalImpactScore", Vulnerability::getOwaspRRTechnicalImpactScore, existingVuln::setOwaspRRTechnicalImpactScore); 
        
               differ.applyIfChanged("owaspRRBusinessImpactScore", Vulnerability::getOwaspRRBusinessImpactScore, existingVuln::setOwaspRRBusinessImpactScore); 
        
               differ.applyIfChanged("owaspRRVector", Vulnerability::getOwaspRRVector, existingVuln::setOwaspRRVector); 
        
               differ.applyIfChanged("vulnerableVersions", Vulnerability::getVulnerableVersions, existingVuln::setVulnerableVersions); 
        
               differ.applyIfChanged("patchedVersions", Vulnerability::getPatchedVersions, existingVuln::setPatchedVersions); 
        
               // EPSS is an additional enrichment that no source currently provides natively. We don't want EPSS scores of CVEs to be purged. 
        
               differ.applyIfNonNullAndChanged("epssScore", Vulnerability::getEpssScore, existingVuln::setEpssScore); 
        
               differ.applyIfNonNullAndChanged("epssPercentile", Vulnerability::getEpssPercentile, existingVuln::setEpssPercentile); 
        
               return differ.getDiffs(); 
        
           }

, and callInTransaction was added as seen

dependency-track/src/main/java/org/dependencytrack/persistence/NotificationQueryManager.java

Lines 162 to 171 in 7ce8e04

    
           return callInTransaction(() -> { 
        
               final NotificationPublisher publisher = new NotificationPublisher(); 
        
               publisher.setName(name); 
        
               publisher.setDescription(description); 
        
               publisher.setPublisherClass(publisherClass.getName()); 
        
               publisher.setTemplate(templateContent); 
        
               publisher.setTemplateMimeType(templateMimeType); 
        
               publisher.setDefaultPublisher(defaultPublisher); 
        
               return pm.makePersistent(publisher); 
        
           });

.

synchronizeVulnerability now returns null if there is nothing to synchronize.

Some modifications were needed to align with the new logic. For example:

if (synchronizeVulnerability == null) continue;

Tests are provided.

Addressed Issue

Closes #3922

Additional Details

Checklist

I have read and understand the contributing guidelines
This PR fixes a defect, and I have provided tests to verify that the fix is effective
This PR implements an enhancement, and I have provided tests to verify that it works as intended
This PR introduces changes to the database model, and I have added corresponding update logic
This PR introduces new or alters existing behavior, and I have updated the documentation accordingly

This modification is in order to avoid multiple writes and/or updates when there is no changes. Part of the logic in updateVulnerability was optimized. Add differ.applyIfChanged and callInTransaction. SynchronizeVulnerability returns null if there is nothing to synchronize. Some modifications were needed to couple with new logic. e.g. if (SynchronizeVulnerability = null) continue. Test are provided. Signed-off-by: Andres Tito <[email protected]>

codacy-production · 2024-11-07T14:50:36Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.00% (target: -1.00%)	✅ 94.92% (target: 70.00%)

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`8547447`)	22551	17839	79.11%
Head commit (`f08804f`)	22565 (+14)	17849 (+10)	79.10% (+0.00%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#4359)	59	56	94.92%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

nscuro

Thanks @LaVibeX! Looks good, just some minor suggestions.

src/main/java/org/dependencytrack/persistence/VulnerabilityQueryManager.java

-Remove unnecesary persistance in updateVulnerability function. -Return updated exisitingVulnerability in updateVulnerability -Remove return of empty vulnerability -Wrap synchronizedVUlnerability in callInTransaction to share same transaction. Signed-off-by: Andres Tito <[email protected]>

nscuro

Thanks!

nscuro requested changes Nov 10, 2024

View reviewed changes

nscuro added this to the 4.13 milestone Nov 10, 2024

nscuro added the enhancement New feature or request label Nov 10, 2024

nscuro approved these changes Nov 12, 2024

View reviewed changes

nscuro merged commit f7e2fa4 into DependencyTrack:master Nov 12, 2024

github-actions bot locked as resolved and limited conversation to collaborators Dec 13, 2024

nscuro changed the title ~~Optimized synchronizeVulnerability~~ Optimize vulnerability synchronization logic to not perform redundant writes Apr 1, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Optimize vulnerability synchronization logic to not perform redundant writes#4359

Optimize vulnerability synchronization logic to not perform redundant writes#4359
nscuro merged 2 commits intoDependencyTrack:masterfrom
LaVibeX:lastModifiedIndicator

LaVibeX commented Nov 7, 2024 •

edited by nscuro

Loading

Uh oh!

codacy-production bot commented Nov 7, 2024 •

edited

Loading

Uh oh!

nscuro left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nscuro left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	private static Map<String, PersistenceUtil.Diff> updateVulnerability(final Vulnerability existingVuln, final Vulnerability reportedVuln) {
	assertPersistent(existingVuln, "existingVuln must be persistent in order for changes to be effective");

	final var differ = new PersistenceUtil.Differ<>(existingVuln, reportedVuln);
	differ.applyIfChanged("title", Vulnerability::getTitle, existingVuln::setTitle);
	differ.applyIfChanged("subTitle", Vulnerability::getSubTitle, existingVuln::setSubTitle);
	differ.applyIfChanged("description", Vulnerability::getDescription, existingVuln::setDescription);
	differ.applyIfChanged("detail", Vulnerability::getDetail, existingVuln::setDetail);
	differ.applyIfChanged("recommendation", Vulnerability::getRecommendation, existingVuln::setRecommendation);
	differ.applyIfChanged("references", Vulnerability::getReferences, existingVuln::setReferences);
	differ.applyIfChanged("credits", Vulnerability::getCredits, existingVuln::setCredits);
	differ.applyIfChanged("created", Vulnerability::getCreated, existingVuln::setCreated);
	differ.applyIfChanged("published", Vulnerability::getPublished, existingVuln::setPublished);
	differ.applyIfChanged("updated", Vulnerability::getUpdated, existingVuln::setUpdated);
	differ.applyIfNonEmptyAndChanged("cwes", Vulnerability::getCwes, existingVuln::setCwes);
	differ.applyIfChanged("severity", Vulnerability::getSeverity, existingVuln::setSeverity);
	differ.applyIfChanged("cvssV2BaseScore", Vulnerability::getCvssV2BaseScore, existingVuln::setCvssV2BaseScore);
	differ.applyIfChanged("cvssV2ImpactSubScore", Vulnerability::getCvssV2ImpactSubScore, existingVuln::setCvssV2ImpactSubScore);
	differ.applyIfChanged("cvssV2ExploitabilitySubScore", Vulnerability::getCvssV2ExploitabilitySubScore, existingVuln::setCvssV2ExploitabilitySubScore);
	differ.applyIfChanged("cvssV2Vector", Vulnerability::getCvssV2Vector, existingVuln::setCvssV2Vector);
	differ.applyIfChanged("cvssV3BaseScore", Vulnerability::getCvssV3BaseScore, existingVuln::setCvssV3BaseScore);
	differ.applyIfChanged("cvssV3ImpactSubScore", Vulnerability::getCvssV3ImpactSubScore, existingVuln::setCvssV3ImpactSubScore);
	differ.applyIfChanged("cvssV3ExploitabilitySubScore", Vulnerability::getCvssV3ExploitabilitySubScore, existingVuln::setCvssV3ExploitabilitySubScore);
	differ.applyIfChanged("cvssV3Vector", Vulnerability::getCvssV3Vector, existingVuln::setCvssV3Vector);
	differ.applyIfChanged("owaspRRLikelihoodScore", Vulnerability::getOwaspRRLikelihoodScore, existingVuln::setOwaspRRLikelihoodScore);
	differ.applyIfChanged("owaspRRTechnicalImpactScore", Vulnerability::getOwaspRRTechnicalImpactScore, existingVuln::setOwaspRRTechnicalImpactScore);
	differ.applyIfChanged("owaspRRBusinessImpactScore", Vulnerability::getOwaspRRBusinessImpactScore, existingVuln::setOwaspRRBusinessImpactScore);
	differ.applyIfChanged("owaspRRVector", Vulnerability::getOwaspRRVector, existingVuln::setOwaspRRVector);
	differ.applyIfChanged("vulnerableVersions", Vulnerability::getVulnerableVersions, existingVuln::setVulnerableVersions);
	differ.applyIfChanged("patchedVersions", Vulnerability::getPatchedVersions, existingVuln::setPatchedVersions);
	// EPSS is an additional enrichment that no source currently provides natively. We don't want EPSS scores of CVEs to be purged.
	differ.applyIfNonNullAndChanged("epssScore", Vulnerability::getEpssScore, existingVuln::setEpssScore);
	differ.applyIfNonNullAndChanged("epssPercentile", Vulnerability::getEpssPercentile, existingVuln::setEpssPercentile);

	return differ.getDiffs();
	}

	return callInTransaction(() -> {
	final NotificationPublisher publisher = new NotificationPublisher();
	publisher.setName(name);
	publisher.setDescription(description);
	publisher.setPublisherClass(publisherClass.getName());
	publisher.setTemplate(templateContent);
	publisher.setTemplateMimeType(templateMimeType);
	publisher.setDefaultPublisher(defaultPublisher);
	return pm.makePersistent(publisher);
	});

Uh oh!

Conversation

LaVibeX commented Nov 7, 2024 • edited by nscuro Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Addressed Issue

Additional Details

Checklist

Uh oh!

codacy-production bot commented Nov 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage summary from Codacy

See diff coverage on Codacy

See your quality gate settings Change summary preferences

Uh oh!

nscuro left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nscuro left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

LaVibeX commented Nov 7, 2024 •

edited by nscuro

Loading

codacy-production bot commented Nov 7, 2024 •

edited

Loading