Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Question: Docker on Windows archiving to an SMB path that doesn't support FSYNC #722

Closed
aluhrs13 opened this issue Apr 24, 2021 · 8 comments
Closed

Question: Docker on Windows archiving to an SMB path that doesn't support FSYNC #722

aluhrs13 opened this issue Apr 24, 2021 · 8 comments
Labels
touches: data/schema/architecture

Comments

@aluhrs13
Copy link

I've been fighting with this long enough to hope that someone with more ArchiveBox and/or Docker knowledge can spot something I'm doing wrong. I'm attempting to run ArchiveBox on Windows following the docker-compose directions, with my Volume on an SMB path (my NAS, not my server).

First I make the volume:
docker volume create --driver local --opt type=cifs --opt device=//<ipaddress>/ArchiveBox --opt o=username=<username>,password=<pass> ArchiveBox

Then I run the first command in the directions:
docker-compose run archivebox init --setup

And hit errors:

Creating network "archivebox3_default" with the default driver
Creating archivebox3_archivebox_run ... done
[i] [2021-04-24 19:07:10] ArchiveBox v0.6.2: archivebox init --setup
    > /data

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/logging/config.py", line 564, in configure
    handler = self.configure_handler(handlers[name])
  File "/usr/local/lib/python3.9/logging/config.py", line 745, in configure_handler
    result = factory(**kwargs)
  File "/usr/local/lib/python3.9/logging/handlers.py", line 153, in __init__
    BaseRotatingHandler.__init__(self, filename, mode, encoding=encoding,
  File "/usr/local/lib/python3.9/logging/handlers.py", line 58, in __init__
    logging.FileHandler.__init__(self, filename, mode=mode,
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1146, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1175, in _open
    return open(self.baseFilename, self.mode, encoding=self.encoding,
PermissionError: [Errno 13] Permission denied: '/data/logs/errors.log'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/archivebox", line 33, in <module>
    sys.exit(load_entry_point('archivebox', 'console_scripts', 'archivebox')())
  File "/app/archivebox/cli/__init__.py", line 140, in main
    run_subcommand(
  File "/app/archivebox/cli/__init__.py", line 74, in run_subcommand
    setup_django(in_memory_db=subcommand in fake_db, check_db=cmd_requires_db and not init_pending)
  File "/app/archivebox/config.py", line 1136, in setup_django
    django.setup()
  File "/usr/local/lib/python3.9/site-packages/django/__init__.py", line 19, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
  File "/usr/local/lib/python3.9/site-packages/django/utils/log.py", line 75, in configure_logging
    logging_config_func(logging_settings)
  File "/usr/local/lib/python3.9/logging/config.py", line 809, in dictConfig
    dictConfigClass(config).configure()
  File "/usr/local/lib/python3.9/logging/config.py", line 571, in configure
    raise ValueError('Unable to configure handler '
ValueError: Unable to configure handler 'logfile'
ERROR: 1

A folder named "logs" is created, but nothing else. Obviously seems to be a permissions issue of some sort, but I don't know enough to figure out what it is. Any suggestions or alternative approaches?
@pirate
Copy link
Member

pirate commented Apr 24, 2021 •
edited
Loading

Try adding a :z flag to your volume:

services:
    archivebox:
        volumes:
            - ArchiveBox:/data:z

Network drives throw red-herring permissions errors sometimes because we use atomic_write to FSYNC all writes, and some filesystems don't support it. The :z flag seems to often fix that quirk by having docker middleman the writes (which seems to hide the lack of FSYNC support and fix the error as a side effect).

@aluhrs13
Copy link
Author

No dice 😢, here's the contents of my docker-compose.yml in case it's something there.

version: '2.4'

volumes:
  vol:
    external: true
    name: ArchiveBox

services:
    archivebox:
        image: ${DOCKER_IMAGE:-archivebox/archivebox:latest}     
        command: server --quick-init 0.0.0.0:8000
        ports:
            - 3081:8000
        environment:
            - ALLOWED_HOSTS=*
            - MEDIA_MAX_SIZE=750m
            - SUBMIT_ARCHIVE_DOT_ORG=False
        volumes:
            - vol:/data:z

@pirate
Copy link
Member

pirate commented Apr 25, 2021 •
edited
Loading 

docker-compose run --rm archivebox /bin/ls -lah /data
docker-compose run --rm archivebox /bin/ls -lah /data/logs

Can you post the output of these ^

@aluhrs13
Copy link
Author

Here you go, thanks for the help!

C:\Users\aluhr\Docker\ArchiveBox3>docker-compose run --rm archivebox /bin/ls -lah /data
Creating archivebox3_archivebox_run ... done
total 4.0K
drwxr-xr-x 2 root root    0 Apr 14 08:57 .
drwxr-xr-x 1 root root 4.0K Apr 25 01:05 ..
drwxr-xr-x 2 root root    0 Apr 14 08:57 logs

C:\Users\aluhr\Docker\ArchiveBox3>docker-compose run --rm archivebox /bin/ls -lah /data/logs
Creating archivebox3_archivebox_run ... done
total 0
drwxr-xr-x 2 root root 0 Apr 14 08:57 .
drwxr-xr-x 2 root root 0 Apr 14 08:57 ..

@pirate
Copy link
Member

pirate commented Apr 25, 2021 •
edited
Loading

Can you try giving all permissions on the dir? chmod -R 777 data (or whatever the windows equivalent of chmod is).

You may also have to change the settings on the cifs cluster to allow read/write for all users, as often permissions are enforced on the server side of the fileshare.

@aluhrs13
Copy link
Author

The permissions definitely changed according to SSH and Synology's dashboard, but re-running the the last commands still show the same permissions from Docker's perspective and I get the same error.

I wonder if I should re-think my approach entirely. Is there any way to change what folder archived data is saved to after initial setup? I didn't see an environment variable to configure that. Or maybe I finally go nuclear and swap this server to Linux like I should've done from the start 😅.

@aluhrs13
Copy link
Author

Alright I've bit a partial bullet and opted to just run ArchiveBox on my Synology instead of trying to fight this more. Feel free to close this out, unless there's something here you want to investigate further, I'm happy to continue trying troubleshooting steps.

@pirate pirate mentioned this issue May 13, 2021
Closed
@pirate pirate closed this as completed May 13, 2021
@pirate pirate changed the title Question: Docker on Windows archiving to an SMB path Question: Docker on Windows archiving to an SMB path that doesn't support FSYNC May 13, 2021
@pirate
Copy link
Member

pirate commented Apr 12, 2022

Note I've added a new DB/filesystem troubleshooting area to the wiki that may help people arriving here from Google: https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives#database-troubleshooting

Contributions/suggestions welcome there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
touches: data/schema/architecture
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pirate @aluhrs13