Is it possible to get passwordless IMAP access from a specific host?

I would like some feedback concerning a use case

Description

I'm considering using an IMAP backup solution, rather than my current shutdown mailserver, backup dirs, start mailserver solution. I would have a job for each user, which would connect over IMAP, do a backup and then disconnect. I don't want to have to store every users password in order to do this. Is it possible to get IMAP access from a given host without a password for all users?