Regarding the verbosity, can't stdout be redirected to /dev/null, but not stderr? Granted that may sometimes hide helpful debug info like with the postfix post-install script output from #2023 ?

We could also just make a note of it in debugging/troubleshooting contributor docs if the verbosity is an issue?

There's also apt-get -qq which is meant to be less noisy and implicitly applies -y, but again may hide info that may be of interest. Easy find & replace on Dockerfile when more verbosity is required though?

The problems we encountered over time stem from the fact that we actually needed output from stdout. The verbosity is not ridiculous though. Your idea of adding -q or -qq is good though and provides a middle way, which I like:) Shall I add -q or -qq @polarathene ?

Note: Tests are failing due to the infamous quota exceeded problem. We should monitor this.

The problems we encountered over time stem from the fact that we actually needed output from stdout.

Oh were there other cases of this happening? I think if the verbosity won't bother anyone we can just have it. But if it's to keep say CI logs more terse when the verbosity is usually noise, lets minimize it with a documented way to manually bring back verbose output when actual problems with builds are encountered? (which I assume isn't that frequent in ratio to number of builds)

-qq doesn't silence some apt-get output mind you.

My main issue with &>/dev/null was actually a result of me disabling the SHELL directive, instead of bash, it ran the commands with sh and &>/dev/null isn't valid in sh but I didn't know that, hence the confusion I had thinking errors were being hidden from me.

Consistency wise, my concern was that one case was > /dev/null instead of >/dev/null like all the others. For easy find & replace, they should ideally be consistent there. Likewise, you had one case of 2>&1 >/dev/null while others were >/dev/null 2>&1 (or rather the bash short form of &>/dev/null). Additionally, I don't know why some similar commands like apt-get install would sometimes use &>/dev/null and others >/dev/null, was that difference intentional or accidental?

Fixing that consistency would be good, they can remain to keep the output less noisy and we just document a find & replace command with sed to run on the Dockerfile should build issues occur?

Note: Tests are failing due to the infamous quota exceeded problem. We should monitor this.

I'm not familiar with this. It's specific to a dovecot test, not a CI limit being exceeded? Do we have a way to reliably reproduce it?

The problems we encountered over time stem from the fact that we actually needed output from stdout.

Oh were there other cases of this happening? I think if the verbosity won't bother anyone we can just have it. But if it's to keep say CI logs more terse when the verbosity is usually noise, lets minimize it with a documented way to manually bring back verbose output when actual problems with builds are encountered? (which I assume isn't that frequent in ratio to number of builds)

I would avoid this solution under all circumstances. I think this would be bad practice, having to look at the documentation just for building or debugging errors. I'm not a fan of very verbose output too, but I think that the current level of verbosity is fine.

-qq doesn't silence some apt-get output mind you.

Yes, I do not want to silence every last output, which is why the -q should provide adequate help.

My main issue with &>/dev/null was actually a result of me disabling the SHELL directive, instead of bash, it ran the commands with sh and &>/dev/null isn't valid in sh but I didn't know that, hence the confusion I had thinking errors were being hidden from me.

Why would one want to do that? Besides &>... (which is now gone btw), the [[ ... ]] construct is only available in Bash too.

Consistency wise, my concern was that one case was > /dev/null instead of >/dev/null like all the others. For easy find & replace, they should ideally be consistent there.

The inconsistency issues are now fixed too. Only in sed commands are there still variations of this, but we cannot change it as search-and-replace would then fail.

Likewise, you had one case of 2>&1 >/dev/null while others were >/dev/null 2>&1 (or rather the bash short form of &>/dev/null). Additionally, I don't know why some similar commands like apt-get install would sometimes use &>/dev/null and others >/dev/null, was that difference intentional or accidental?

This actually was intended, as the order makes a difference. The usage of & was intentional based on the output. But that may be misleading. Hence I removed them completely. With the -q flag, everything is clear.

Fixing that consistency would be good, they can remain to keep the output less noisy and we just document a find & replace command with sed to run on the Dockerfile should build issues occur?

Like I said, I would very much try to void this solution.

Note: Tests are failing due to the infamous quota exceeded problem. We should monitor this.

I'm not familiar with this. It's specific to a dovecot test, not a CI limit being exceeded? Do we have a way to reliably reproduce it?

It's inherent to the tests, and I have not seen it for a long time. It's not a CI thing and we sadly cannot reliably reproduce it.

Documentation preview for this PR is ready! 🎉

Built with commit: 8dfdc4b

having to look at the documentation just for building or debugging errors.

There could always be an inline comment note at the top of the Dockerfile about it, it wouldn't be hard to miss.

If there's no issue with the verbosity and wanting to keep that toned down in logs for the majority of builds where this isn't an issue, that's fine too (it was just a suggestion since there was clear efforts to reduce the output).

FWIW, there does appear to be a bunch of other errors/warnings in the output elsewhere that don't negatively affect the build from progressing, I haven't noted these down and I'm not sure how relevant they are to be concerned about.

Why would one want to do that? Besides &>... (which is now gone btw), the [[ ... ]] construct is only available in Bash too.

This was my fault. I didn't know any better 😅

I didn't know originally that these were Bash specific things, I just noticed that I got what I thought was more actionable errors... I'm not used to using a SHELL directive in my Dockerfiles and was mostly wondering if the pipefail had something to do with it (something I still haven't quite read up on yet either).

It took a while to realize that the error was coming from postfix install, and then why my newer Ubuntu VPS triggered it but not my older one. More verbosity probably wouldn't have helped identify the systemd cause, but it did get me the helpful postfix post-install output 👍

The usage of & was intentional based on the output.

Was it intentional to use it on one apt-get install but not another? That's what I was referring to as feeling inconsistent, IIRC that also made stderr hidden, which I didn't get why you'd want that, especially only on some of the same command such as apt-get install.

Like I said, I would very much try to void this solution.

👍

Do I understand 2>&1 correctly? Does it redirect stderr into stdout? In testing a basic Dockerfile I noticed the two red lines about dpkg were blended into the white noise of the rest of the output when 2>&1 was used. Why is that desirable? (I know you don't use them as the below example, just curious regarding their purpose in the Dockerfile)

Dockerfile:

FROM docker.io/debian:buster-slim
ARG DEBIAN_FRONTEND=noninteractive
SHELL ["/bin/bash", "-o", "pipefail", "-c"]

RUN \
  apt-get -qq update && \
  apt-get -qq install apt-utils 2>&1 && \
  apt-get -qq dist-upgrade && \
  apt-get -qq install postfix 2>&1

2>&1 (only this output if using >/dev/null, thus only this is from stderr?):

1. debconf: delaying package configuration, since apt-utils is not installed - Not important.
2. E: Sub-process /usr/bin/dpkg returned an error code (1) - Not particularly helpful in itself, but may still have value being highlighted.

Regarding -qq output of postfix, it still provides the error information encountered for #2023 (newaliases lines), along with other unrelated info of note:

mailname is not a fully qualified domain name.  Not changing /etc/mailname.
/etc/aliases does not exist, creating it.
WARNING: /etc/aliases exists, but does not have a root alias.
newaliases: warning: valid_hostname: misplaced delimiter: ad7f8f206270..
newaliases: fatal: file /etc/postfix/main.cf: parameter myhostname: bad parameter value: ad7f8f206270..
dpkg: error processing package postfix (--configure):
 installed postfix package post-installation script subprocess returned error exit status 75
Errors were encountered while processing:
 postfix
E: Sub-process /usr/bin/dpkg returned an error code (1)

I notice there is >&2 to echo text into stderr, this works as expected highlighting it as red. Is the usage of 2>&1 meaning that it's stderr output is not important and should instead be redirected into stdout? I believe that's the case with gpg, IIRC outputting to stderr for some unnecessary reason, redirection makes sense there.

Regarding -y, -yq and -qq differences

It would seem -qq is always preferable?

DEBIAN_FRONTEND=noninteractive apt-get -y install jq:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  libjq1 libonig5
The following NEW packages will be installed:
  jq libjq1 libonig5
0 upgraded, 3 newly installed, 0 to remove and 1 not upgraded.
Need to get 355 kB of archives.
After this operation, 1072 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian buster/main amd64 libonig5 amd64 6.9.1-1 [171 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 libjq1 amd64 1.5+dfsg-2+b1 [124 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 jq amd64 1.5+dfsg-2+b1 [59.4 kB]
Fetched 355 kB in 0s (24.1 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libonig5:amd64.
(Reading database ... 6460 files and directories currently installed.)
Preparing to unpack .../libonig5_6.9.1-1_amd64.deb ...
Unpacking libonig5:amd64 (6.9.1-1) ...
Selecting previously unselected package libjq1:amd64.
Preparing to unpack .../libjq1_1.5+dfsg-2+b1_amd64.deb ...
Unpacking libjq1:amd64 (1.5+dfsg-2+b1) ...
Selecting previously unselected package jq.
Preparing to unpack .../jq_1.5+dfsg-2+b1_amd64.deb ...
Unpacking jq (1.5+dfsg-2+b1) ...
Setting up libonig5:amd64 (6.9.1-1) ...
Setting up libjq1:amd64 (1.5+dfsg-2+b1) ...
Setting up jq (1.5+dfsg-2+b1) ...
Processing triggers for libc-bin (2.28-10) ...

-q only removes the 2 Done at the top. -qq implicitly sets -y and excludes everything prior to debconfline halfway. Is the added output verbosity of -q vs -qq likely to provide more meaningful information?

Note if apt-get update has not been run prior, here's the difference in failure output:

-y:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package jq

-yq:

Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package jq

-qq:

E: Unable to locate package jq

Potential output of interest

Setting up clamav-base (0.103.2+dfsg-0+deb10u1) ...
id: 'clamav': no such user

Setting up opendmarc (1.3.2-6+deb10u1) ...
dbconfig-common: writing config to /etc/dbconfig-common/opendmarc.conf
Creating config file /etc/dbconfig-common/opendmarc.conf with new version
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2).
unable to connect to mysql server.
error encountered creating user:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
dbconfig-common: opendmarc configure: noninteractive fail.
dbconfig-common: opendmarc configure: ignoring errors from here forwards
populating database via sql...  done.
dbconfig-common: flushing administrative password

Setting up postgrey (1.36-5.1) ...
Warning: The home dir /var/lib/postgrey you specified can't be accessed: No such file or directory
Adding system user `postgrey' (UID 110) ...
Adding new group `postgrey' (GID 112) ...
Adding new user `postgrey' (UID 110) with group `postgrey' ...
Not creating home directory `/var/lib/postgrey'.

Tue Jun  8 00:45:41 2021 -> ^Clamd was NOT notified: Can't connect to clamd through /var/run/clamav/clamd.ctl: No such file or directory

Dockerfile concerns

While looking over it, two things that caught my attention was:

1. mkcert.sh purpose, the PR and commit that introduced it don't mention it at all.
   • Although they are a follow up PR to this one which has some brief TODO comments in the Dockerfile and a reference to this issue which provides a bit more insight into the mkcert.sh introduction, and concerns about the clamav in the Dockerfile that are resolved/explained in a comment (thus the clamd error is expected).
2. The letsencrypt cert was added without any information as to why 5 years ago. Presumably this was required back then as the image may not have had the required certs for supporting LetsEncrypt... however I'm pretty sure that's a non-issue now with IdenTrust root CA being present? I could be mistaken, but we lack context and perhaps tests to validate that.

I also noticed some helpful inline comments about certain parts of the Dockerfile have since been removed for some reason :\

Oops, looks like I was a little too slow 😅

EDIT: Ah... it was my fault, I approved and it was set to auto-merge with a single approval 😬

Oops, looks like I was a little too slow sweat_smile

EDIT: Ah... it was my fault, I approved and it was set to auto-merge with a single approval grimacing

😆 I was confused because I could not remember merging this myself. But yes, I enabled auto-merge :D I will create a follow-up PR :) Your changes should be implemented!

Regarding your notes:

Do I understand 2>&1 correctly? Does it redirect stderr into stdout?

Exactly. 2>&1 is chosen when the output from stderr may be confusing for readers who may think this error is important when it isn't. This was tested beforehand and is only applied where it is known to not cause problems (yet, of course 😆 ). So we still want to see the output, but not in red :) About the debconf: delaying package configuration, since apt-utils is not installed: I will take care of this. This is known to be a Dockerfile thing where an initial package is missing that is reporting these warnings. I will just use 2>/dev/null and remove this output.

The usage of & was intentional based on the output.

Was it intentional to use it on one apt-get install but not another?

Yes :)