chore(scripts): Removing flock so NFS works by NorseGaud · Pull Request #1980 · docker-mailserver/docker-mailserver

NorseGaud · 2021-05-18T02:25:48Z

BLOCKERS TO MERGING THIS PR (respectively ordered)

MacOS linting & testing support + docs #2001

Description

#1979

Using a temp file instead of flock to support NFS mounts
cmp checks could return a failure for any reason so we should check specifically for exit code 1 as that indicates that it's actually a failure due to differences

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Improvement (non-breaking change that does improve existing functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Checklist:

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation (README.md or the documentation under docs/)
If necessary I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

polarathene · 2021-05-18T22:34:05Z

I'm not super proficient in shell scripts, but is it possible to be more DRY here by moving the if conditional logic that I'm seeing repeated in diffs into the helper util as well?

Does it make more sense to wrap the logic in a function that can be passed into the helper util call along with the lock file name?

NorseGaud · 2021-05-19T00:53:48Z

I'm not super proficient in shell scripts, but is it possible to be more DRY here by moving the if conditional logic that I'm seeing repeated in diffs into the helper util as well?

Does it make more sense to wrap the logic in a function that can be passed into the helper util call along with the lock file name?

This would require eval, but it's possible:

example -

Helpers

function runWithLock() {
	LOCK_FILE="/tmp/docker-mailserver/$1.lock"
  if [[ ! -e "\${LOCK_FILE}" ]]; then
    trap rmlock EXIT
    touch $LOCK_FILE
	  eval "$2"
  fi
}

function rmlock() {
  rm -f "${LOCK_FILE}"
}

addmailuser (and others)

# Protect config file with lock to avoid race conditions
touch "${DATABASE}"
runWithLock "$(basename "$0")" "$(cat <<BLOCK
  if grep -qi "^$(escape "${USER}")|" "${DATABASE}" 2>/dev/null
  then
    echo "User '${USER}' already exists."
    exit 1
  fi

  if [[ -z ${PASSWD} ]]
  then
    read -r -s -p "Enter Password: " PASSWD
    echo
    [[ -z ${PASSWD} ]] && errex "Password must not be empty"
  fi

  HASH="$(doveadm pw -s SHA512-CRYPT -u "${USER}" -p "${PASSWD}")"
  echo "${USER}|${HASH}" >> "${DATABASE}"
BLOCK
)"

See any issues @polarathene ?

polarathene · 2021-05-19T05:08:37Z

See any issues @polarathene ?

I'm not really able to give a proper review regarding this unfortunately. @aendeavor or @casperklein are often working on the shell scripts for the project and can likely weigh in on what is preferred.

@aendeavor especially makes an effort to have shell scripts meet certain guidelines, but is a bit busy with other commitments at present, so they may take a little longer to chime in with a review.

casperklein · 2021-05-19T10:29:10Z

Proposal, make two function:

_removeLock(){}
_createLock() {
  if lock successfull; return 0
  if lock unsuccessfull; return 1
}

Why do you think eval() is needed?

addmailuser:

_createLock || { echo "could not create lock file.... Exiting"; exit 1}
command1
command2
_removeLock

Also check, why the tests are failing.

georglauterbach · 2021-05-19T11:10:39Z

As @polarathene mentioned, I'm low on time, so I'll keep this short.

This:

# Protect config file with lock to avoid race conditions
touch "${DATABASE}"
runWithLock "$(basename "$0")" "$(cat <<BLOCK
  if grep -qi "^$(escape "${USER}")|" "${DATABASE}" 2>/dev/null
  then
    echo "User '${USER}' already exists."
    exit 1
  fi

  if [[ -z ${PASSWD} ]]
  then
    read -r -s -p "Enter Password: " PASSWD
    echo
    [[ -z ${PASSWD} ]] && errex "Password must not be empty"
  fi

  HASH="$(doveadm pw -s SHA512-CRYPT -u "${USER}" -p "${PASSWD}")"
  echo "${USER}|${HASH}" >> "${DATABASE}"
BLOCK
)"

is not an option at all. One would use the Bash-idiomatic way for "function pointers":

touch "${DATABASE}"

function __admailuser
{
  if grep -qi "^$(escape "${USER}")|" "${DATABASE}" 2>/dev/null
  then
    echo "User '${USER}' already exists."
    exit 1
  fi

  if [[ -z ${PASSWD} ]]
  then
    read -r -s -p "Enter Password: " PASSWD
    echo
    [[ -z ${PASSWD} ]] && errex "Password must not be empty"
  fi

  HASH="$(doveadm pw -s SHA512-CRYPT -u "${USER}" -p "${PASSWD}")"
  echo "${USER}|${HASH}" >> "${DATABASE}"

}

# Protect config file with lock to avoid race conditions
runWithLock "$(basename "$0")" "$(declare -f __addmailuser)"

As @casperklein mentioned, is eval really needed?
This PR is bigger than you think @NorseGaud, since there are many scripts that use flock. One of them is check-for-changes.sh (which IIRC integrates tightly with start-mailserver.sh or one of its stacks).
Please adhere to the styling guidelines set for Bash. These can be found in CONTRIBUTING.md
In your example, there is a line stating if [[ ! -e "\${LOCK_FILE}" ]]; then. Where does the backslash come from?

georglauterbach · 2021-05-19T11:11:44Z

Proposal, make two function:
_removeLock(){}
_createLock() {
  if lock successfull; return 0
  if lock unsuccessfull; return 1
}
Why do you think eval() is needed?

addmailuser:
_createLock || { echo "could not create lock file.... Exiting"; exit 1}
command1
command2
_removeLock
Also check, why the tests are failing.

Wouldn't this exit early if the lock cannot be obtained? We should make this more like a spin-lock with exists after trying too often.

NorseGaud · 2021-05-19T11:46:14Z

Thanks, Georg (@aendeavor). I appreciate you taking the time! I won't answer some of your points as they are not relevant for the hacked together example I provided. However, I do like passing in of the function with declare. Ultimately, the function supports either your recommended function-based idiomatic approach or the one I provided, so it remains flexible (using HEREDOC is something I don't really care about so I'll avoid it).

However, eval is very important. I personally have never and have even tried to exploit eval to no avail, but know that some people are VERY sensitive when using it in code as it supposedly comes with security risks. Here is why eval is needed (a very common issue when doing things like this):

    ~/docker-mailserver  on   issues/1979 !2  
function test() {                                         
        LOCK_FILE="$1"
        echo "$2"
        $2
}


function __addmailuser
{
        echo 123
        echo 456
}

test "/tmp/lock" "$(declare -f __addmailuser); __addmailuser"
__addmailuser () {
        echo 123
        echo 456
}; __addmailuser
test:3: command not found: __addmailuser () {\n\techo 123\n\techo 456\n}; __addmailuser

    ~/docker-mailserver  on   issues/1979 !2  
function test() {                                             
        LOCK_FILE="$1"
        echo "$2"
        "$2"
}


function __addmailuser
{
        echo 123
        echo 456
}

test "/tmp/lock" "$(declare -f __addmailuser) && __addmailuser"
__addmailuser () {
        echo 123
        echo 456
} && __addmailuser
test:3: command not found: __addmailuser () {\n\techo 123\n\techo 456\n} && __addmailuser

    ~/docker-mailserver  on   issues/1979 !2  
function test() { 
        LOCK_FILE="$1"
        echo "$2"
        "$2"
}


function __addmailuser
{
        echo 123
        echo 456
}

test "/tmp/lock" "$(declare -f __addmailuser); __addmailuser"
__addmailuser () {
        echo 123
        echo 456
}; __addmailuser
test:3: command not found: __addmailuser () {\n\techo 123\n\techo 456\n}; __addmailuser

    ~/docker-mailserver  on   issues/1979 !2  
function test() {                                         
        LOCK_FILE="$1"
        echo "$2"
        eval "$2"
}


function __addmailuser
{
        echo 123
        echo 456
}

test "/tmp/lock" "$(declare -f __addmailuser); __addmailuser"
__addmailuser () {
        echo 123
        echo 456
}; __addmailuser
123
456

Let me know what you think and I can rework the code to support this and test it thoroughly.

Lastly, I don't think the test failure is due to changes I made, but I could be wrong. I'll check again after the final code.

georglauterbach · 2021-05-19T11:59:29Z

I know what eval does and why you have chosen to use it :) But I currently do not understand why one could not do as @casperklein has shown above. Moreover, you could use bash -c "$(declare -f __addmailuser); __addmailuser" too. So why the complicated way? Is there a race-condition I do not see, or maybe some other problem? Why must we pass a function to another function here?

NorseGaud · 2021-05-19T12:14:08Z

I know what eval does and why you have chosen to use it :) But I currently do not understand why one could not do as @casperklein has shown above. Moreover, you could use bash -c "$(declare -f __addmailuser); __addmailuser" too. So why the complicated way? Is there a race-condition I do not see, or maybe some other problem? Why must we pass a function to another function here?

Oh, there must be a misunderstanding on my part. There isn't a need to do it. I also like a spin-lock approach. I'll update the code with the recommended approach.

NorseGaud · 2021-05-19T12:42:50Z

Is it weird that ec linting wanted...

target/scripts/helper-functions.sh:
	20: Wrong amount of left-padding spaces(want multiple of 2)
	21: Wrong amount of left-padding spaces(want multiple of 2)
	22: Wrong amount of left-padding spaces(want multiple of 2)
	23: Wrong amount of left-padding spaces(want multiple of 2)
	24: Wrong amount of left-padding spaces(want multiple of 2)
	25: Wrong amount of left-padding spaces(want multiple of 2)
	26: Wrong amount of left-padding spaces(want multiple of 2)
	27: Wrong amount of left-padding spaces(want multiple of 2)
	28: Wrong amount of left-padding spaces(want multiple of 2)
	29: Wrong amount of left-padding spaces(want multiple of 2)

... when those lines use the same indentation that all other functions use (a single tab indent)?

Now it just looks weird:

casperklein · 2021-05-19T12:43:35Z

Wouldn't this exit early if the lock cannot be obtained? We should make this more like a spin-lock with exists after trying too often.

Yes, but that is the intention of a lock? If it cannot be obtained, then abort. If it would continue never the less, then why using locks at all?
If there is a lock, I don't think retrying will help. If a lock is there then a) another script is running at the same time or b) there is a stale lock left.
If someone uses setup.sh it's unlikely that at the same time another setup.sh call is running.

NorseGaud · 2021-05-19T12:45:06Z

Wouldn't this exit early if the lock cannot be obtained? We should make this more like a spin-lock with exists after trying too often.

Yes, but that is the intention of a lock? If it cannot be obtained, then abort. If it would continue never the less, then why using locks at all?
If there is a lock, I don't think retrying will help. If a lock is there then a) another script is running at the same time or b) there is a stale lock left.
If someone uses setup.sh it's unlikely that at the same time another setup.sh call is running.

@casperklein @aendeavor , should I ditch the spin-lock? It seems as if failing immediately is far less logic and the user could just retry.

georglauterbach · 2021-05-19T12:52:10Z

@casperklein @aendeavor , should I ditch the spin-lock? It seems as if failing immediately is far less logic and the user could just retry.

I will leave this up to you guys. I like the spinlock, but it may be overkill. Contact me if there is something very urgent :D

PS: @NorseGaud Styling is now on point, good job!

casperklein · 2021-05-19T17:44:21Z

should I ditch the spin-lock? It seems as if failing immediately is far less logic and the user could just retry.

ack 👍

NorseGaud · 2021-05-20T11:27:09Z

Hey @casperklein, any recommendations for #1980 (comment) ?

NorseGaud · 2021-06-09T12:44:27Z

Ok, ran some tests and this seems to be working great.

EFS mounted using NFS4.1:

XXXXX:/  8.0E   41M  8.0E   1% /mnt/efs-us-west-2

XXXXX:/ /mnt/efs-us-west-2 nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport 0 0

Default state:

Added test user:

Deleting test user:

Then I added /tmp/docker-mailserver/addmailuser.lock and ran it again:

[ec2-user@ip-172-31-3-247 ~]$ ./setup.sh email add [email protected] test123
Error :: Lock file /tmp/docker-mailserver/addmailuser.lock exists. Another addmailuser execution is happening. Try again later.
Aborting.

--- UNCHECKED ERROR
  - script    = setup.sh
  - function  = _docker_image / ${CRI} run --rm -v "${CONFIG_PATH}:/tmp/docker-mailserver${USE_SELINUX}" "${USE_TTY}" "${IMAGE_NAME}" "${@:+$@}"
  - line      = 243
  - exit code = 1

Make sure you use a version of this script that matches
the version / tag of docker-mailserver. Please read the
'Get the tools' section in the README on GitHub careful-
ly and use ./setup.sh help and read the VERSION section.

I also see the check-for-changes.lock being created when the checksum file exists:

I'm continuing to check the various places that locks are used and even setting up multiple machines to which I'll issue concurrent commands and make sure the locking works.

NorseGaud · 2021-06-12T03:41:53Z

Tests with two servers and an EFS NFS4.1 mount looks great:

Add + Del User

    ~/blah.com  on   master !1 ?1  time ssh -i ./blah.com-us-west-2.pem [email protected] -p 2244 "sleep 3; ./setup.sh email add [email protected] test123" &

time ssh -i ./blah.com-ap-southeast-1.pem [email protected] -p 2244 "./setup.sh email add [email protected] test123"

[1] 39800
Error :: Lock file /tmp/docker-mailserver/addmailuser.lock exists. Another addmailuser execution is happening. Try again later.
Aborting.

--- UNCHECKED ERROR
  - script    = setup.sh
  - function  = _docker_image / ${CRI} run --rm -v "${CONFIG_PATH}:/tmp/docker-mailserver${USE_SELINUX}" "${USE_TTY}" "${IMAGE_NAME}" "${@:+$@}"
  - line      = 243
  - exit code = 1

Make sure you use a version of this script that matches
the version / tag of docker-mailserver. Please read the
'Get the tools' section in the README on GitHub careful-
ly and use ./setup.sh help and read the VERSION section.
ssh -i ./blah.com-us-west-2.pem [email protected] -p 2244   0.02s user 0.01s system 0% cpu 4.998 total
[1]  + 39800 exit 1     time ssh -i ./blah.com-us-west-2.pem [email protected] -p 2244 
ssh -i ./blah.com-ap-southeast-1.pem [email protected] -p 2244   0.02s user 0.00s system 0% cpu 6.719 total



    ~/blah.com  on   master !1 ?1  time ssh -i ./blah.com-us-west-2.pem [email protected] -p 2244 "sleep 4; ./setup.sh email del -y [email protected]" &

ssh -i ./blah.com-ap-southeast-1.pem [email protected] -p 2244 "./setup.sh email del -y [email protected]"
[1] 39792
Error :: Lock file /tmp/docker-mailserver/delmailuser.lock exists. Another delmailuser execution is happening. Try again later.
Aborting.

--- UNCHECKED ERROR
  - script    = setup.sh
  - function  = _docker_container / ${CRI} exec "${USE_TTY}" "${CONTAINER_NAME}" "${@:+$@}"
  - line      = 253
  - exit code = 1

Make sure you use a version of this script that matches
the version / tag of docker-mailserver. Please read the
'Get the tools' section in the README on GitHub careful-
ly and use ./setup.sh help and read the VERSION section.
ssh -i ./blah.com-us-west-2.pem [email protected] -p 2244   0.02s user 0.01s system 0% cpu 5.682 total
[1]  + 39792 exit 1     time ssh -i ./blah.com-us-west-2.pem [email protected] -p 2244 
[email protected] and potential aliases deleted.
Mailbox deleted.

Set quota


    ~/blah.com  on   master !1 ?1  time ssh -i ./blah.com-us-west-2.pem [email protected] -p 2244 "sleep 3; ./setup.sh quota set [email protected] 100G" &

time ssh -i ./blah.com-ap-southeast-1.pem [email protected] -p 2244 "./setup.sh quota set [email protected] 100G"

[1] 39839
Error :: Lock file /tmp/docker-mailserver/setquota.lock exists. Another setquota execution is happening. Try again later.
Aborting.

--- UNCHECKED ERROR
  - script    = setup.sh
  - function  = _docker_image / ${CRI} run --rm -v "${CONFIG_PATH}:/tmp/docker-mailserver${USE_SELINUX}" "${USE_TTY}" "${IMAGE_NAME}" "${@:+$@}"
  - line      = 243
  - exit code = 1

Make sure you use a version of this script that matches
the version / tag of docker-mailserver. Please read the
'Get the tools' section in the README on GitHub careful-
ly and use ./setup.sh help and read the VERSION section.
ssh -i ./blah.com-us-west-2.pem [email protected] -p 2244   0.02s user 0.01s system 0% cpu 4.961 total
[1]  + 39839 exit 1     time ssh -i ./blah.com-us-west-2.pem [email protected] -p 2244 
ssh -i ./blah.com-ap-southeast-1.pem [email protected] -p 2244   0.02s user 0.01s system 0% cpu 8.423 total

Update email

    ~/blah.com  on   master !1 ?1  time ssh -i ./blah.com-us-west-2.pem [email protected] -p 2244 "sleep 4; ./setup.sh email update [email protected] test123" &

time ssh -i ./blah.com-ap-southeast-1.pem [email protected] -p 2244 "./setup.sh email update [email protected] test123"


[1] 39862
Error :: Lock file /tmp/docker-mailserver/updatemailuser.lock exists. Another updatemailuser execution is happening. Try again later.
Aborting.

--- UNCHECKED ERROR
  - script    = setup.sh
  - function  = _docker_image / ${CRI} run --rm -v "${CONFIG_PATH}:/tmp/docker-mailserver${USE_SELINUX}" "${USE_TTY}" "${IMAGE_NAME}" "${@:+$@}"
  - line      = 243
  - exit code = 1

Make sure you use a version of this script that matches
the version / tag of docker-mailserver. Please read the
'Get the tools' section in the README on GitHub careful-
ly and use ./setup.sh help and read the VERSION section.
ssh -i ./blah.com-us-west-2.pem [email protected] -p 2244   0.02s user 0.01s system 0% cpu 5.939 total
[1]  + 39862 exit 1     time ssh -i ./blah.com-us-west-2.pem [email protected] -p 2244 
ssh -i ./blah.com-ap-southeast-1.pem [email protected] -p 2244   0.01s user 0.01s system 0% cpu 7.267 total

Ready for review!

casperklein · 2021-06-12T13:06:21Z

-  echo "${USER}|${HASH}" >> "${DATABASE}"
-) 200< "${DATABASE}"
+create_lock "$(basename "$0")"
+if grep -qi "^$(escape "${FULL_EMAIL}")|" "${DATABASE}" 2>/dev/null


2>/dev/null
Why? Is this, when $DATABASE does not exist yet or what should be catched here?

Probably. Although this should never happen, since there is always a touch $DATABASE beforehand, right?

Hey Casper, that line was actually already in the script before I modified it for the lock. Not sure I can answer your question :(

No problem. But every time something is changed, I try to take the opportunity to review the whole file. If we all agree/think it's not necessary, i would remove it.

@casperklein awesome, so if grep -qi "^$(escape "${FULL_EMAIL}")|" "${DATABASE}" 2>/dev/null now becomes if grep -qi "^$(escape "${FULL_EMAIL}")|" "${DATABASE}" ?

georglauterbach

LGTM but wee need to resolve the "issues" @casperklein pointed out with his comments first :)

Great work so far!

…into issues/1979

wernerfred

LGTM 👍🏻
Great work!

georglauterbach

👍🏼

georglauterbach · 2021-06-14T10:13:20Z

@NorseGaud not sure whether reviews will be dismissed if you do the rebase? Shall I update the branch?

NorseGaud · 2021-06-14T11:26:26Z

@NorseGaud not sure whether reviews will be dismissed if you do the rebase? Shall I update the branch?

Sure, I would appreciate it (I'm traveling) 🙂

casperklein · 2021-06-14T16:49:12Z

@NorseGaud See my last two little comments. Beside that, this PR looks great 👍

…into issues/1979

NorseGaud · 2021-06-15T16:28:08Z

Thanks y'all!

NorseGaud closed this May 18, 2021

NorseGaud reopened this May 18, 2021

NorseGaud closed this May 18, 2021

NorseGaud reopened this May 18, 2021

wernerfred linked an issue May 18, 2021 that may be closed by this pull request

[BUG] flock doesn't work with NFS #1979

Closed

wernerfred added this to the v10.0.1 milestone May 19, 2021

georglauterbach added area/scripts kind/improvement Improve an existing feature, configuration file or the documentation meta/feature freeze On hold due to upcoming release process meta/wip priority/low labels May 19, 2021

georglauterbach modified the milestones: v10.0.1, v10.1.0 May 19, 2021

pull/1980#commitcomment-51931012

8be19c7

wernerfred removed the meta/wip label Jun 12, 2021

mail_override_hostname

eca086f

casperklein reviewed Jun 12, 2021

View reviewed changes

Comment thread target/bin/delmailuser Outdated

georglauterbach reviewed Jun 13, 2021

View reviewed changes

email address

63b43e5

NorseGaud requested review from casperklein and georglauterbach June 13, 2021 20:20

Merge branch 'issues/1979' of github.com:NorseGaud/docker-mailserver …

f698ff8

…into issues/1979

wernerfred previously approved these changes Jun 14, 2021

View reviewed changes

georglauterbach previously approved these changes Jun 14, 2021

View reviewed changes

Merge branch 'master' into issues/1979

0cf4868

casperklein reviewed Jun 14, 2021

View reviewed changes

Comment thread target/scripts/check-for-changes.sh

NorseGaud added 2 commits June 14, 2021 19:31

pull/1980#discussion_r650390778 & pull/1980#discussion_r651231737

8f4285f

Merge branch 'issues/1979' of github.com:NorseGaud/docker-mailserver …

d93c262

…into issues/1979

NorseGaud dismissed stale reviews from georglauterbach and wernerfred via d93c262 June 14, 2021 23:32

wernerfred approved these changes Jun 15, 2021

View reviewed changes

casperklein approved these changes Jun 15, 2021

View reviewed changes

georglauterbach approved these changes Jun 15, 2021

View reviewed changes

georglauterbach merged commit 5becce8 into docker-mailserver:master Jun 15, 2021

NorseGaud deleted the issues/1979 branch June 15, 2021 16:28

polarathene mentioned this pull request Aug 20, 2022

chore: addmailuser - Remove delaying completion until /var/mail is ready #2729

Merged

5 tasks

Uh oh!

Conversation

NorseGaud commented May 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Checklist:

Uh oh!

polarathene commented May 18, 2021

Uh oh!

NorseGaud commented May 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Helpers

addmailuser (and others)

Uh oh!

polarathene commented May 19, 2021

Uh oh!

casperklein commented May 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

georglauterbach commented May 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

georglauterbach commented May 19, 2021

Uh oh!

NorseGaud commented May 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

georglauterbach commented May 19, 2021

Uh oh!

NorseGaud commented May 19, 2021

Uh oh!

NorseGaud commented May 19, 2021

Uh oh!

casperklein commented May 19, 2021

Uh oh!

NorseGaud commented May 19, 2021

Uh oh!

georglauterbach commented May 19, 2021

Uh oh!

casperklein commented May 19, 2021

Uh oh!

NorseGaud commented May 20, 2021

Uh oh!

NorseGaud commented Jun 9, 2021

Uh oh!

NorseGaud commented Jun 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add + Del User

Set quota

Update email

Uh oh!

casperklein Jun 12, 2021

Choose a reason for hiding this comment

Uh oh!

georglauterbach Jun 13, 2021

Choose a reason for hiding this comment

Uh oh!

NorseGaud Jun 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

casperklein Jun 14, 2021

Choose a reason for hiding this comment

Uh oh!

NorseGaud Jun 14, 2021

Choose a reason for hiding this comment

Uh oh!

casperklein Jun 14, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

georglauterbach left a comment

Choose a reason for hiding this comment

Uh oh!

wernerfred left a comment

Choose a reason for hiding this comment

Uh oh!

georglauterbach left a comment

Choose a reason for hiding this comment

NorseGaud commented May 18, 2021 •

edited

Loading

NorseGaud commented May 19, 2021 •

edited

Loading

casperklein commented May 19, 2021 •

edited

Loading

georglauterbach commented May 19, 2021 •

edited

Loading

NorseGaud commented May 19, 2021 •

edited

Loading

NorseGaud commented Jun 12, 2021 •

edited

Loading

NorseGaud Jun 13, 2021 •

edited

Loading