RELEASE.2022-10-29T06-21-33Z migration from older release #15967

martadinata666 · 2022-10-29T11:39:52Z

martadinata666
Oct 29, 2022

RELEASE.2022-10-29T06-21-33Z got this note.
Removes remaining gateway implementations of NAS/S3 and legacy FS mode completely - refer #15929. Existing users should continue to use their current releases if they wish to continue using gateway mode.

After upgrade results
ERROR Unable to use the drive /data: Drive /data: found backend type fs, expected xl or xl-single: Invalid arguments specified

Is there migrating guide i can follow? thanks

Answered by harshavardhana

Oct 29, 2022

Don't upgrade as mentioned in the release notes. There is nothing new that you will get by upgrading.

View full answer

harshavardhana · 2022-10-29T15:51:15Z

harshavardhana
Oct 29, 2022
Maintainer

Don't upgrade as mentioned in the release notes. There is nothing new that you will get by upgrading.

6 replies

martadinata666 Oct 29, 2022
Author

as my minio installation is pretty small, I deploy a new one, then mc mirror from from old deployment to new one. that's the only way I can think of and the path I'm using. maybe another way? im not really sure.

jtackaberry Oct 29, 2022

Don't upgrade as mentioned in the release notes. There is nothing new that you will get by upgrading.

Does this imply there will be a migration path introduced in the future, once there is something new that we will get by upgrading to some future version?

I assumed not, given that the fs backend code has been entirely ripped out, so like the OP I was confused and ultimately surprised that I could find no documentation on how to migrate, or even what the recommended strategy is.

@martadinata666 's suggestion to just stand up a parallel deployment and mirror the files is at least a strategy, albeit problematic when you don't have sufficient storage to float a duplicate deployment. But if that's the only viable path in order to continue to receive updates (and more importantly security fixes) for users currently on the fs backend, then IMO it should be included in the documentation.

harshavardhana Oct 29, 2022
Maintainer

While there is no reason to update this time, it is not viable to stay on older version for too long. Is there a way to migrate setup using FS backend into a new one? Maybe we should open an issue and ask for documentation update regarding such migration. I could see that the question was asked in #14331 (comment), but couldn't find an answer to it, unfortunately.

Feel free to open an issue at minio/docs

jtackaberry Oct 29, 2022

Thanks. Filed minio/docs#624

markovendelin Oct 29, 2022

As far as I understand, mc mirror will just sync the data. I presume we would have to go bucket-by-bucket in this case. On a top of that, somehow users have to be migrated as well as all the policies. @jtackaberry - thank you for opening an issue!

Link: minio/docs#624

chrisjenx · 2022-11-01T19:11:29Z

chrisjenx
Nov 1, 2022

Your answer is don't upgrade... you're joking right?

2 replies

harshavardhana Nov 1, 2022
Maintainer

Your answer is don't upgrade... you're joking right?

No need to be sarcastic @chrisjenx - you shouldn't upgrade if you don't need to. There is nothing you will get by upgrading - nothing has changed on the legacy front in 2+yrs.

If you want to build server with latest go in some manner for the release you use just install via go install -v github.com/minio/minio@<release-tag>

chrisjenx Nov 2, 2022

Mmmm, thankfully picking a downgraded docker image fixed it.

The annoyance came from the unintuitive error message thrown. Better DevX would be to link to the upgrade doc or change log with breaking changes in it so people have a good stepping off point.

LucasFrecia · 2022-11-07T09:08:34Z

LucasFrecia
Nov 7, 2022

When running locally on a mac, updated minio through brew and had the error:

2022/11/07 08:32:28 ERROR Unable to use the drive /usr/local/var/minio: Drive /usr/local/var/minio: found backend type fs, expected xl or xl-single: Invalid arguments specified

WARNING: This "fix", is with minio data loss and was done to have my local env running faster. Read @harshavardhana comment for proper migration steps.

The error was solved by removing old minio files and reinstalling:
rm -rf /usr/local/etc/minio /usr/local/var/minio

1 reply

harshavardhana Nov 7, 2022
Maintainer

Please don't do things like rm -rf with your backend - please follow the migration guide https://min.io/docs/minio/linux/operations/install-deploy-manage/migrate-fs-gateway.html

goodale · 2022-11-10T16:03:48Z

goodale
Nov 10, 2022

what do we do if the mc admin update already ran and we're now on 20221108 and minio won't start? The migration steps presume the prior version is still available and running in order to do the migration. So the migration documentation isn't helpful if the binary has already been updated...

1 reply

goodale Nov 10, 2022

I was able to downgrade so I can get back to work and revisit the lengthy migration procedure at some point in the future. Here are the steps I took in case it is helpful to anyone else:

_cd to the directory where minio is currently installed_
wget https://dl.min.io/server/minio/release/linux-amd64/archive/minio_20221024183507.0.0_amd64.deb -O minio.deb
sudo dpkg -i minio.deb
_start minio as normal_

While the move away from the gateway was announced awhile ago there is no mention of it in the actual mc admin update process that broke things as a result to the update after the 10/24 version. It would have been nice had the update script warned users of the breaking changes given it temporarily bricks things for anyone using minio gateway or fs. (The statement [Upgrades Are Non-Disruptive](https://min.io/docs/minio/linux/operations/install-deploy-manage/upgrade-minio-deployment.html?ref=docs-redirect#id2) clearly wasn't the case for me ;)

mirekphd · 2022-11-25T08:21:46Z

mirekphd
Nov 25, 2022

I don't think it's the only breaking change that happened since the 2022-10-29 release (or at least since our last working version container (mirekphd/ml-minio22:20221021, which is based on the upstream minio/minio container).

There is apparently one more breaking change there: write-only-once protection is put on everything, including MinIO's own config folder.

More info:
If you start the server in a "fresh" folder / PVC (i.e. one without ~/.minio.sys - you can replicate by for instance renaming it), minio fails to list any existing files (managing to list only folders as buckets), and the newly created ~/.minio.sys is apparently put in the write-once-read-many (WORM) mode... this is how minio/minio is now configured.

Expand detailed error log

WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
WARNING: Detected Linux kernel version older than 4.0.0 release, there are some known potential performance problems with this kernel version. MinIO recommends a minimum of 4.x.x linux kernel version for best performance
Formatting 1st pool, 1 set(s), 1 drives per set.
WARNING: Host local has more than 0 drives of set. A host failure will result in data becoming unavailable.
Warning: Default parity set to 0. This can lead to data loss.

API: SYSTEM()
Time: 08:02:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/config/config.json) returned an error (file access denied) (*fmt.wrapError)
      17: internal/logger/logger.go:258:logger.LogIf()
      16: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
      15: internal/logger/logonce.go:135:logger.LogOnceIf()
      14: cmd/erasure-object.go:555:cmd.readAllXL()
      13: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
      12: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
      11: cmd/erasure-sets.go:934:cmd.(*erasureSets).GetObjectNInfo()
      10: cmd/erasure-server-pool.go:791:cmd.(*erasureServerPools).GetObjectNInfo()
       9: cmd/config-common.go:38:cmd.readConfigWithMetadata()
       8: cmd/config-common.go:59:cmd.readConfig()
       7: cmd/config-migrate.go:2506:cmd.checkConfigVersion()
       6: cmd/config-migrate.go:2479:cmd.migrateMinioSysConfig()
       5: cmd/config.go:225:cmd.initConfig()
       4: cmd/config.go:195:cmd.(*ConfigSys).Init()
       3: cmd/server-main.go:444:cmd.initConfigSubsystem()
       2: cmd/server-main.go:411:cmd.initServer()
       1: cmd/server-main.go:597:cmd.serverMain()

API: SYSTEM()
Time: 08:02:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Unable to initialize config, some features may be missing migrateMinioSysConfig: Prefix access is denied: .minio.sys/config/config.json (*fmt.wrapError)
       4: internal/logger/logger.go:258:logger.LogIf()
       3: cmd/server-main.go:450:cmd.initConfigSubsystem()
       2: cmd/server-main.go:411:cmd.initServer()
       1: cmd/server-main.go:597:cmd.serverMain()

API: SYSTEM()
Time: 08:02:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: open /data/.minio.sys/config/iam/format.json/xl.meta: operation not permitted (*fs.PathError)
       5: internal/logger/logger.go:258:logger.LogIf()
       4: cmd/metacache-walk.go:327:cmd.(*xlStorage).WalkDir.func2()
       3: cmd/metacache-walk.go:351:cmd.(*xlStorage).WalkDir()
       2: cmd/metacache-walk.go:361:cmd.(*xlStorageDiskIDCheck).WalkDir()
       1: cmd/metacache-set.go:905:cmd.listPathRaw.func2()
MinIO Object Storage Server
Copyright: 2015-2022 MinIO, Inc.
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Version: RELEASE.2022-11-17T23-20-09Z (go1.19.3 linux/amd64)

Status:         1 Online, 0 Offline. 
API: https://ml-minio-rs-node1-master.app.warta.pl 
Console: http://10.131.3.129:9001 http://127.0.0.1:9001   

Documentation: https://min.io/docs/minio/linux/index.html

API: SYSTEM()
Time: 08:04:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:04:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:04:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:05:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:05:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:05:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:06:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:06:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:06:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:07:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage.json) returned an error (file access denied) (*fmt.wrapError)
      12: internal/logger/logger.go:258:logger.LogIf()
      11: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
      10: internal/logger/logonce.go:135:logger.LogOnceIf()
       9: cmd/erasure-object.go:555:cmd.readAllXL()
       8: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       7: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       6: cmd/erasure-sets.go:934:cmd.(*erasureSets).GetObjectNInfo()
       5: cmd/erasure-server-pool.go:791:cmd.(*erasureServerPools).GetObjectNInfo()
       4: cmd/config-common.go:38:cmd.readConfigWithMetadata()
       3: cmd/config-common.go:59:cmd.readConfig()
       2: cmd/data-usage.go:95:cmd.loadDataUsageFromBackend()
       1: cmd/bucket-replication.go:3070:cmd.(*ReplicationPool).saveStatsToDisk()

API: SYSTEM()
Time: 08:07:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:07:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:07:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:08:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:08:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:08:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:09:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:09:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:09:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:10:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:10:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:10:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:11:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:11:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:11:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:12:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: open /data/.minio.sys/config/iam/format.json/xl.meta: operation not permitted (*fs.PathError)
       5: internal/logger/logger.go:258:logger.LogIf()
       4: cmd/metacache-walk.go:327:cmd.(*xlStorage).WalkDir.func2()
       3: cmd/metacache-walk.go:351:cmd.(*xlStorage).WalkDir()
       2: cmd/metacache-walk.go:361:cmd.(*xlStorageDiskIDCheck).WalkDir()
       1: cmd/metacache-set.go:905:cmd.listPathRaw.func2()

API: SYSTEM()
Time: 08:12:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage.json) returned an error (file access denied) (*fmt.wrapError)
      12: internal/logger/logger.go:258:logger.LogIf()
      11: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
      10: internal/logger/logonce.go:135:logger.LogOnceIf()
       9: cmd/erasure-object.go:555:cmd.readAllXL()
       8: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       7: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       6: cmd/erasure-sets.go:934:cmd.(*erasureSets).GetObjectNInfo()
       5: cmd/erasure-server-pool.go:791:cmd.(*erasureServerPools).GetObjectNInfo()
       4: cmd/config-common.go:38:cmd.readConfigWithMetadata()
       3: cmd/config-common.go:59:cmd.readConfig()
       2: cmd/data-usage.go:95:cmd.loadDataUsageFromBackend()
       1: cmd/bucket-replication.go:3070:cmd.(*ReplicationPool).saveStatsToDisk()

API: SYSTEM()
Time: 08:12:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:12:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:12:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:13:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:13:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:13:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:14:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:14:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:14:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:15:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:15:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:15:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:16:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:16:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:16:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:17:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage.json) returned an error (file access denied) (*fmt.wrapError)
      12: internal/logger/logger.go:258:logger.LogIf()
      11: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
      10: internal/logger/logonce.go:135:logger.LogOnceIf()
       9: cmd/erasure-object.go:555:cmd.readAllXL()
       8: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       7: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       6: cmd/erasure-sets.go:934:cmd.(*erasureSets).GetObjectNInfo()
       5: cmd/erasure-server-pool.go:791:cmd.(*erasureServerPools).GetObjectNInfo()
       4: cmd/config-common.go:38:cmd.readConfigWithMetadata()
       3: cmd/config-common.go:59:cmd.readConfig()
       2: cmd/data-usage.go:95:cmd.loadDataUsageFromBackend()
       1: cmd/bucket-replication.go:3070:cmd.(*ReplicationPool).saveStatsToDisk()

API: SYSTEM()
Time: 08:17:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:17:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:17:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

2 replies

harshavardhana Nov 25, 2022
Maintainer

You simply re-used the previous drive that is not how migration works @mirekphd - you need to migrate to a fresh instance of MinIO. Please read https://min.io/docs/minio/linux/operations/install-deploy-manage/migrate-fs-gateway.html

mirekphd Nov 25, 2022

Note that I have not necessarily re-used the "drive" [1]. Some of the "drives" I have this issue with, are PVCs in the "ab" namespace which were completely untouched before (unless they were re-cycled due to incorrect PV recycling policy, which is rare), should be pristine and both of them were indeed totally empty, created a few days ago for a new user.

[1] (not a drive as such - it's a PVC on a NFS server run on localhost in a k8s clone Openshift. I suppose the only non-standard feature distinguishing our (and other OKD / Openshift users) setup from the typical hard drive is a difference between the container user ID and owner of the files / folders - user nobody (as usual for NFS root-squash setups). The container ID which the MinIO server uses at runtime is a high ID randomly imposed at container startup - but we can pin the ID to fix it forever, however not to a value we choose, like the typical 1000 or 1001, it is necessarily a value that must be chosen from a different range. However, MinIO has no problem creating its ~/.minio.sys config folder in all these scenarios, only it later becomes write-protected, as seen from the logs. As for the reason why the files are not listed, only buckets, I can only speculate that the listing permission may be separate for buckets and for files.

As a side note, the last working version of MinIO is one day newer, mirekphd/ml-minio22:20221022, not the one tagged 20221021 (both of these working containers should correspond to the official minio/minio container daily tags). For an example of a non-working container affected by this config access issue see: mirekphd/ml-minio22:20221124.

RELEASE.2022-10-29T06-21-33Z migration from older release #15967

Uh oh!

Uh oh!

Replies: 5 comments · 12 replies

Uh oh!

harshavardhana Oct 29, 2022 Maintainer

Uh oh!

Uh oh!

martadinata666 Oct 29, 2022 Author

Uh oh!

Uh oh!

harshavardhana Oct 29, 2022 Maintainer

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

harshavardhana Nov 1, 2022 Maintainer

Uh oh!

Uh oh!

Uh oh!

Uh oh!

harshavardhana Nov 7, 2022 Maintainer

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

harshavardhana Nov 25, 2022 Maintainer

Uh oh!

Uh oh!

Replies: 5 comments 12 replies

harshavardhana
Oct 29, 2022
Maintainer

martadinata666 Oct 29, 2022
Author

harshavardhana Oct 29, 2022
Maintainer

harshavardhana Nov 1, 2022
Maintainer

harshavardhana Nov 7, 2022
Maintainer

harshavardhana Nov 25, 2022
Maintainer