Sensitive info is removed from source files with media type "text"#1052
Conversation
| final String value = filterSensitiveInfo(new String(source.getContents()), variables, isAdmin); | ||
| source.setContents(value.getBytes()); | ||
| @Value("${sensitiveinfo.analysis.extensions}") | ||
| private String sensitiveAnalysisExtensions; |
There was a problem hiding this comment.
Why not @Value("#{'${sensitiveinfo.analysis.extensions}'.split(',')}") or @Value("${sensitiveinfo.analysis.extensions}") private String[] sensitiveAnalysisExtensions;
There was a problem hiding this comment.
fixed, changed to array version
| return source; | ||
| } | ||
|
|
||
| private boolean isNeedFiltering(AnalysisResultFile source) { |
There was a problem hiding this comment.
Could you rename it to isFilteringRequired?
|
|
||
| private boolean checkExtension(AnalysisResultFile source) { | ||
| String extension = FilenameUtils.getExtension(source.getFileName()); | ||
| for (String value : sensitiveExtensions) { |
There was a problem hiding this comment.
Wouldn't it be simpler if you store sensitiveExtensions in a HashSet and use contains?
|
The result files, which you iterate over, are:
while the mentioned How is it going to be filtered? |
|
Also, a back-merge will be required |
|
added filtering of archive content |
| return source; | ||
| } | ||
|
|
||
| private byte[] doFilterSensitiveInfo(Path path, byte[] bytes, Map<String, Object> variables, boolean isAdmin) { |
There was a problem hiding this comment.
I would stick to single-source of truth: bytes represents info which is derived from path. So why cannot we just pass path to avoid potential contradictions and anomalies?
| return EXTENSION_ZIP.equalsIgnoreCase(extension); | ||
| } | ||
|
|
||
| private AnalysisResultFile checkZipArchive(AnalysisResultFile resultFile, Map<String, Object> variables, boolean isAdmin) { |
There was a problem hiding this comment.
Seems like a compute intense task. I suggest doing this processing one-time when results are received from EE. Not on each download
…nsitive_connection_info
…rom service which returns execution result to user(#1046)
|
The migration could break as a result of future changes to the repository definition. Instead of using the |
|
@chrisknoll |
|
Ok, I think it may be possible to work around this by making copies or 'legacy' impelmentations of the Service interfaces that we can inject into the migration. But, that sounds like a lot of work. We should be able to see errors in the migration script if something gets updated in the future, so we can address making the migrations work with the new changes when the need arises. |
|
@chrisknoll |
|
Excellent work, @ssuvorov-fls . This will guard against future changes that may break migration. Thanks! |
|
Why wasn't this branch deleted? |
* increase version to 2.7.1 (#995) * Added permissions to watch IR report (#991) * added permissions to watch IR report * added migrations to cover newly created sources * simplify migration queries * Set Priority of first daimon "1" (#998) * Fixes performance issues with PLE & PLP generations retrieval (#1010) * Fixes performance issues with PLE & PLP generations retrieval * Adds filter to do cache invalidation in case of threads re-usage * Forgotten change * Fixed PLE & PLP jobs invalidation (2.7.1) (#1006) * Fixed PLE & PLP jobs invalidation * Fixes transaction * BQ support for PLE / PLP (#1012) * Update circe version (#1013) * Fixes SQL execution for BQ when keyfile is uploaded (#1016) * Fixes creation of new Datasource with keyfile (#1017) * Customizable PLE / PLP skeletons (#1021) * Split EE result files storage into 2 tables and avoid contents retrieval on each request (#1024) * Split EE result files storage into 2 tables and avoid contents retrieval on each request * Testing and review fixes * Fixes results persistence for PLE R script (2.7.1) (#1026) * Fixes results persistence for PLE R script * Apply consistent approach for creating output folder for PLE & PLP * Fixes issue with import new estimation/prediction (#1035) * Utilizes Request compressed results from Execution Engine (#1044) * Implements RunAs functionality (#1075) * Improvements for PLE & PLP starter scripts (#1092) - Install packages into local lib - Install packages using `R CMD build` to avoid issues on Windows and forced packages `upgrade` - Do not build vignettes * Sensitive info is removed from analyses result files (#1052) * Fix Estimation R script
3 participants
fixes #1046