Switch from Minio to Versitygw#2656
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## release/v7.0.0 #2656 +/- ##
==================================================
- Coverage 79.25% 79.22% -0.03%
==================================================
Files 670 670
Lines 54135 54164 +29
Branches 734 734
==================================================
+ Hits 42906 42913 +7
- Misses 11149 11171 +22
Partials 80 80
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| tar xzf /tmp/versitygw_v${OPENC3_VERSITYGW_VERSION}_Linux_${ARCH}.tar.gz -C /tmp versitygw_v${OPENC3_VERSITYGW_VERSION}_Linux_${ARCH}/versitygw && \ | ||
| mv /tmp/versitygw_v${OPENC3_VERSITYGW_VERSION}_Linux_${ARCH}/versitygw /usr/local/bin/ && \ | ||
| chmod +x /usr/local/bin/versitygw && \ | ||
| rm -rf /tmp/versitygw_* |
There was a problem hiding this comment.
This install uses the released binaries from the release assets page: https://github.com/versity/versitygw/releases/tag/v1.0.10. We'll have to manually grab updates ... probably needs to be in the package_audit script.
| # Batch delete in chunks of 1000 (S3 limit) | ||
| keys.each_slice(1000) do |key_batch| | ||
| @bucket.delete_objects(bucket: ENV['OPENC3_CONFIG_BUCKET'], keys: key_batch) | ||
| end |
There was a problem hiding this comment.
I think this is a great optimization claude found
scripts/linux/openc3_migrate_s3.sh
Outdated
| echo "" | ||
| echo "Fixing permissions on ${NEW_VOLUME} for host user ${HOST_UID}:${HOST_GID}..." | ||
| docker run --rm -v "${NEW_VOLUME}:/data" alpine chown -R "${HOST_UID}:${HOST_GID}" /data | ||
| echo -e "${GREEN}✓ Permissions fixed${NC}" |
There was a problem hiding this comment.
This section was key to getting it to work
ryanmelt
left a comment
ryanmelt
left a comment
There was a problem hiding this comment.
I would like to keep the reduced ScriptRunner permissions if possible at all. From the docs I think it is possible... They seem to support policies and users. Please see if that can be restored.
- Rename openc3-s3 to openc3-bucket for clarity - Implement IAM user creation in docker-entrypoint.sh for ScriptRunner with "user" role (limited permissions vs admin) - Add bucket policies via ensure_scriptrunner_policy() to grant ScriptRunner access to config and logs buckets - Update all minio references to bucket/versitygw throughout codebase - Update traefik configs: service-minio -> service-bucket - Update credentials: openc3minio -> openc3bucket - Update package_audit scripts to check versitygw releases - Remove obsolete openc3-minio directory Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Add auto-detection of running MINIO/versitygw containers - Support pre-migration while COSMOS 6 is running - Use separate credentials for source (MINIO) and destination (versitygw) - Match user IDs with openc3.sh behavior (rootless detection) - Use minio/mc from openc3-cosmos-init:6.10.4 container - Add proper health checks and status reporting with file counts - Fix compose.yaml volume name to prevent prefix issues - Update README with migration documentation and credential info Co-Authored-By: Claude Opus 4.5 <[email protected]>
|
Summary is that we've completely removed the minio container and renamed it |
# Conflicts: # .env # .github/workflows/clamav.yml # .github/workflows/cli.yml # .github/workflows/post_release_clamav.yml # .github/workflows/post_release_trivy.yml # .github/workflows/trivy.yml # compose-build.yaml # compose.yaml # openc3-cosmos-init/init.sh # openc3-traefik/traefik-allow-http.yaml # openc3-traefik/traefik-letsencrypt.yaml # openc3-traefik/traefik-ssl.yaml # openc3-traefik/traefik.yaml # openc3.sh # openc3/lib/openc3/script/storage.rb # openc3/lib/openc3/utilities/aws_bucket.rb # openc3/lib/openc3/utilities/s3_autoload.rb # openc3/python/openc3/script/storage.py # openc3/python/openc3/utilities/aws_bucket.py # scripts/linux/openc3_build_ubi.sh # scripts/linux/openc3_migrate_s3.sh # scripts/linux/openc3_setup.sh # scripts/linux/openc3_util.sh # scripts/release/build_multi_arch.sh # scripts/windows/openc3_setup.bat # scripts/windows/openc3_util.bat
- Rename openc3-bucket container/service to openc3-buckets (plural) - Rename openc3-block-v volume to openc3-buckets-v - Update all references across compose files, scripts, code, and docs - Add openc3-cosmos-migration service definition to compose.yaml (commented out) - Add migration configuration variables to .env - Consolidate openc3-s3 and openc3-bucket directories into openc3-buckets - Keep IAM support from openc3-bucket's docker-entrypoint.sh - Add tar/gzip installation to Dockerfile-ubi from openc3-s3 - Delete redundant openc3-s3 directory Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Update openc3-redis UBI build to use Dockerfile-ubi which builds Valkey from source, fixing the 'invalid user valkey' error when using Iron Bank Redis image - Add -fno-lto flags to Dockerfile-ubi to fix parallel build failures caused by LTO jobserver issues - Add architecture detection to openc3_build_ubi.sh to skip --platform linux/amd64 on ARM Macs, allowing native builds - Simplify openc3-redis/Dockerfile by removing conditional package updates Co-Authored-By: Claude Opus 4.5 <[email protected]>
2 participants
I ran through the migration script and successfully migrated my data. This will obviously need extensive testing in Enterprise (haven't done that yet) but it's a good start and all works in Core.