[Feature:InstructorUI] Add bulk upload redactions#11539
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #11539 +/- ##
============================================
- Coverage 21.73% 21.60% -0.13%
- Complexity 9335 9363 +28
============================================
Files 268 268
Lines 35847 35914 +67
Branches 474 474
============================================
- Hits 7793 7761 -32
- Misses 27584 27683 +99
Partials 470 470
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
| img = Image.open(image_filename) | ||
| draw = ImageDraw.Draw(img) | ||
| draw.rectangle(redaction.coordinates, fill="black") | ||
| img.save(image_filename, |
There was a problem hiding this comment.
might be better to save the image after all the redactions are completed on it in case something goes wrong with redacting
There was a problem hiding this comment.
Also it'd be good if you could show an image example on the description for your PR
…, including JSON upload and removal options
williamschen23
left a comment
There was a problem hiding this comment.
comments from quickyl looking at code
jeffrey-cordero
left a comment
There was a problem hiding this comment.
The bulk upload redactions functionality still looks good to me. The code looks slightly more readable and maintainable since my last review. One potential note I'd add is making an issue for converting this into a Vue component when it gets merged.
bmcutler
left a comment
There was a problem hiding this comment.
-
please update PR description to follow new format.
-
In addition to the high level 'why' - please add explaination of what & why for each of the significant changes in the processing of bulk upload submissions (why lines are deleted & relocated from qr_split, etc)
Hmm, install failed so I can't test right away. Might be user error on my part.
### Why is this Change Important & Necessary? <!-- Include any GitHub issue that is fixed/closed using "Fixes #<number>" or "Closes #<number>" syntax. Alternately write "Partially addresses #<number>" or "Related to #<number>" as appropriate. --> #11539 added this line. This line does not work for checkpoint gradeables as `gradeable.js` does not get included. This would cause all the dates to fail as the event listeners below it are in control of the dates ### What is the New Behavior? <!-- Include before & after screenshots/videos if the user interface has changed. --> We should only load bulk uploads if we have an electronic gradeable ### What steps should a reviewer take to reproduce or test the bug or new feature? Visit the checkpoint gradeable edit page. Notice error in console. Go to dates, notice that you cannot change any as a refresh will clear it ### Automated Testing & Documentation <!-- Is this feature sufficiently tested by unit tests and end-to-end tests? If this PR does not add/update the necessary automated tests, write a new GitHub issue and link it below. Is this feature sufficiently documented on submitty.org? Link related PRs or new GitHub issue to update documentation. --> #11982 ### Other information <!-- Is this a breaking change? Does this PR include migrations to update existing installations? Are there security concerns with this PR? -->
- Replace pdf2image with PyMuPDF for faster PDF processing - Add pattern caching for instant redaction rendering - Implement parallel processing for bulk operations (1.7x speedup) - Add file caching to skip unchanged submissions - Optimize memory usage (70% reduction) - Remove external poppler dependency Resolves performance concerns from PR Submitty#11539
**Why is this Change Important & Necessary?** Fixes #11867. When testing PR #11539 (PDF Redactions), performance issues were found in PDF image generation with redactions enabled. This PR improves performance by switching the rendering backend and adds safety fixes for concurrent bulk regeneration and caching. **What is the New Behavior?** 1)PyMuPDF Integration: Replaced the previous PDF rendering approach with PyMuPDF (fitz) for faster page rendering. 2) Caching with Redaction Awareness: Adds a cache keyed on the PDF contents and redaction data, so changing redactions correctly forces regeneration. 3) Daemon support updates. **What steps should a reviewer take to reproduce or test the new feature?** 1. Test it on a gradeable with pdf redactions enabled. I tested on this gradeable Bulk Upload Scanned Exam. 2. Upload the json redactions file You can use this- [ { "page": 1, "x1": 0.25, "y1": 0.25, "x2": 0.75, "y2": 0.75 }, { "page": 2, "x1": 0.1, "y1": 0.1, "x2": 0.9, "y2": 0.9 } ] 3. I have tested on a 28 page pdf file. 4. Watch the daemon logs (sudo journalctl -u submitty_daemon_jobs_handler.service -f) 5. Performance on my system- On main it took around 10 seconds On the PR's branch it took around 5.27 second <img width="958" height="170" alt="Screenshot 2026-06-03 143240" src="https://github.com/user-attachments/assets/6c0c1a95-6759-431c-bf9a-2bb6f3e2528b" /> 6. Modify redactions and regenerate again, images should update (cache invalidated). **Automated Testing & Documentation** No new tests were added. **Other information**



Why is this Change Important & Necessary?
Images of test PDFs frequently contain a student's information, so now users have the option to redact certain parts of certain pages of tests.
What is the New Behavior?
An instructor can upload redactions json files to specify areas of the bulk split pdf that should not be visible to graders.
What steps should a reviewer take to reproduce or test the bug or new feature?
To add redactions to a gradeable visit the update page on the rubric tab. The JSON can only be uploaded when " And do you expect each specific problem/item/component to appear on a specific page number of the PDF document? " is turned on
Automated Testing & Documentation
Other information
See #11538
What is the current behavior?
Other information?
Documentation in Submitty/submitty.github.io#682
This pull request introduces a new feature to handle redactions for gradeable submissions, along with updates to related database migrations, job processing, and API endpoints. The key changes include adding a
gradeable_redactiontable, implementing a redaction processing workflow, and modifying the PDF image generation process to apply redactions.Database Changes:
gradeable_redactiontable to store redaction data, including constraints for valid coordinate ranges (migration/migrator/data/course_tables.sql).gradeable_redactiontable and constraints (migration/migrator/migrations/course/20250312145730_add_redactions.py).Job Processing Updates:
generate_pdf_imagesjob to accept redactions and output redacted images with a checkered pattern (sbin/submitty_daemon_jobs/submitty_jobs/generate_pdf_images.py).RegenerateBulkImagesjob to regenerate images for all submissions in a bulk upload, applying redactions (sbin/submitty_daemon_jobs/submitty_jobs/regenerate_bulk_images.py).RegenerateBulkImagesjob into the job processing pipeline (sbin/submitty_daemon_jobs/submitty_jobs/jobs.py).API and Controller Enhancements:
AdminGradeableController, including validation and triggering the regeneration job (site/app/controllers/admin/AdminGradeableController.php).SubmissionControllerto include theRedactionmodel (site/app/controllers/student/SubmissionController.php).Test and Example Updates:
sbin/submitty_daemon_jobs/tests/test_bulk_pdf_split.py).redactions.jsonfile to demonstrate redaction data format (more_autograding_examples/bulk_upload_pdfs/submissions/redactions.json).These changes collectively enable the system to manage redactions effectively, ensuring sensitive information in gradeable submissions can be obscured as needed.