(Should we have a VPS-project-Phabricator -> collaboration-services bot? It seems that otherwise tasks here get ignored)

The mail config appears to be:

$mail_config = [
    {   
        'key'      => 'wikimedia-smtp',  
        'type'     => 'smtp',
        'options'  => {
            'host' => 'localhost',
            'port' => 25,
        }
    }
]

So mostly this question would be "is it even possible to deliver mail from cloud VPS instances to the outside world" or "why does mail to localhost not get delivered on a default cloud VPS instance" or something like that. Which would be more a general question to ask wmcs team first.

Then after that we could see if and which part of the config would have to be adjusted in cloud vs prod while keeping the difference as minimal as possible.

Also Phabricator (Phorge) config is split between puppet (SRE) and scap (releng) so additionally it depends what a task is about which team really should look. That being said, SRE collab and releng typically meet for Phabricator deploys anyways, so this should be doable.

Finally, no "VPS project Phabricator" exists, at least not anymore. There is only a VPS project called "devtools" nowadays with some test instances of some services.

So probably herald rule to add both collab and releng and/or retire that tag or replace it.

So mostly this question would be "is it even possible to deliver mail from cloud VPS instances to the outside world" or "why does mail to localhost not get delivered on a default cloud VPS instance" or something like that. Which would be more a general question to ask wmcs team first.

It is possible, see Help:Email in Cloud VPS. Is the from address @wmcloud.org?

@JJMC89 Thank you! The part "in most cases you can set the SMTP host to localhost and port to 25" is perfect because then our existing code should just work without changes.

We might indeed just need Sending_emails_from_a_non-Cloud_VPS_domain but have to debug some more what it is actually trying.

Random thought: if fixing the email-sending issue is proving to be a challenge, maybe we could mitigate this by (temporarily?) setting https://phabricator.wmcloud.org/config/edit/auth.require-email-verification/ to false — ie., no longer requiring email verification in order for accounts to be able to login to the test Phabricator instance?
(Of course, I guess one possible downside to this idea is the potential for an increased risk of automated spambots using the test Phab instance :/)

As another potential idea… we could set auth.require-email-verification to false, but set auth.require-approval to true, and ask people to file a task in VPS-project-Phabricator (using a prefilled form like this) for an admin on the test instance to approve their account. These sorts of approvals would hopefully require less time/effort than the de facto current process of a collaboration-services SRE SSHing to the test instance to manually activate a user's email address (xref T377236, T397280), and would hopefully make the test instance available for more people to use.

If either of those ideas sound good, we could then slightly reframe this task to just be something like "Phabricator test instance can't send email" (or resolve this one and split that issue out into a new task).

@Dzahn @Aklapper (and/or anyone else), what do you think?

a task to request account activation is a bit of a hurdle to get into the test instance [and requires SRE time

To be honest the SRE time required to manually activate some phab users on request is orders of magnitude smaller than resolving this ticket or dealing with spam bots.

Your suggestions sound good and are really thoughtful but there is also no simple way to have different configuration settings on test vs prod instance.

an admin on the test instance to approve their account. These sorts of approvals would hopefully require less time/effort than the de facto current process of a collaboration-services SRE SSHing to the test instance

To me personally SSHing to the instance and running a single command is less effort than logging in on a web UI and finding where to approve a user in there. The new suggestion would require a list of admins to exist that can be contacted or a ticket can be assigned to somehow, which wouldn't be obvious to me how.

There are two separate issues being conflated here:

• Issue 1: Should some human have to manually approve accounts on the test instance (in some way or another)
• Issue 2: If the answer to Issue 1 is yes, what should that process be.

You seem to be thinking the answer to issue 1 is yes. I would think it is no; it's illogical for the test instance to be more locked-down than the production instance here.

The production instance is connected to other systems handling the user sign-up. MediaWiki/SUL and LDAP/developer accounts with their own mechanisms to prevent abuse. The test instance doesn't have these. So I am not sure about the "more locked down" part here.

It seems very likely that just removing the approval step will result in spam users.

In T388022#11230217, @Pppery wrote:

There are two separate issues being conflated here:

• Issue 1: Should some human have to manually approve accounts on the test instance (in some way or another)
• Issue 2: If the answer to Issue 1 is yes, what should that process be.

You seem to be thinking the answer to issue 1 is yes. I would think it is no; it's illogical for the test instance to be more locked-down than the production instance here.

I feel like there might be some inadvertent crossed wires here. (I apologise if I could have worded my last comment better!)

To be clear, I was suggesting that - if it's proving difficult to get the test Phab instance to successfully send emails - the "must have a verified email address" config requirement could be disabled for the test instance. (Judging by @Dzahn's comment re. there not being a simple way for the test instance to have different config to the prod instance, though, I guess this might not be possible in the first place.)

My second idea of requiring account approval was in case folks would (e.g.) be worried about increased spam on the test instance if the email verification requirement was removed; and I was putting forward the idea of account approval to try to pre-emptively offer a solution to that potential problem. I don't necessarily per se think that a human should have to manually approve accounts on the test instance — I intended to just offer it as an idea, for if folks thought that removing the email verification requirement would be unworkable on its own :)

In T388022#11230175, @Dzahn wrote:

[…] The new suggestion would require a list of admins to exist that can be contacted or a ticket can be assigned to somehow, which wouldn't be obvious to me how.

FWIW, I watch the VPS-project-Phabricator tag & I imagine I'd've been be happy to approve accounts if/when I saw a ticket requesting account approval. However, I get that this would introduce a bus factor of 1, so I guess my suggestion would've been for a prefilled form to also tag Release-Engineering-Team (if that would've been okay with them), in a similar way to the GitLab account activation form. (To be fair though, I don't know how amenable RelEng would've been to this idea, which on its own might've made it unworkable 😅)

In T388022#11230175, @Dzahn wrote:

To be honest the SRE time required to manually activate some phab users on request is orders of magnitude smaller than resolving this ticket or dealing with spam bots.

Fair enough :) In which case, do you mind if I add a note (e.g.) under https://www.mediawiki.org/wiki/Phabricator#Get_started (& maybe also on the homepage of https://phabricator.wmcloud.org?) to inform folks that they can file a task to request that collab-services manually verifies their test instance account? That wouldn't fully mitigate the additional hurdle to using the test instance, but would hopefully at least make people aware of how they can request account activation (and that they can request it).
[genuine question]

I don't mind if you do that. That would give us an idea how common it is (or will become).

Taking a step back though: I just did a quick test and sent an email from the shell of the phab test host to myself.. and I .. got that email.

Seems like this is only in the phabricator config itself after all.. or something was fixed meanwhile since this ticket was created.

It hasn't been fixed. I asked phabricator.wmcloud.org to resend me a verification email and it didn't arrive.

In T388022#11231303, @Dzahn wrote:

I don't mind if you do that. That would give us an idea how common it is (or will become).

Thanks :) I've added some text to https://phabricator.wmcloud.org/W1 so far, and I'll probably add something to the wiki pages once I think of what to write on them. Wording improvement suggestions welcome!

Fwiw, while debugging an unrelated issue I spotted this in the Cloud VPS outbound mail server logs that looks highly relevant:

2025-10-01 13:06:08 H=phabricator-bullseye.devtools.eqiad1.wikimedia.cloud [172.16.4.197]:12115 I=[172.16.2.248]:25 F=<[email protected]> rejected RCPT <[redacted]>: Mail sent from Cloud VPS using non-supported domain phabricator.wmcloud.org

As noted in https://wikitech.wikimedia.org/wiki/Help:Email_in_Cloud_VPS#Sending_emails_from_a_Cloud_VPS_domain, subdomains of wmcloud.org cannot send mail by default (since DKIM signing mail from arbitrary subdomains is non-trivial).

How hard would it be to support sending mail from phabricator.wmcloud.org on the cloud-vps side? That seems like it would solve this problem by itself.

In T388022#11231344, @Pppery wrote:

It hasn't been fixed. I asked phabricator.wmcloud.org to resend me a verification email and it didn't arrive.

What I meant was the general sending of mail from a cloud VPS project shell to the world. I wanted to see if it's a general problem or specific to Phabricator. "fixed" in this context referred to delivering mail in general (by using the mail command on the shell).

Also I would like to add some general comments about this ticket. We do like it if people get something out of this test instance and want to use it.. we really do. But it was not planned for this. The main reason this was created was that we have something where we can test puppet and phab config changes before merging in production. So please bear with us when things like sending email or user auth wasn't a consideration before we got actual requests like this.

In T388022#11234003, @Dzahn wrote:

Also I would like to add some general comments about this ticket. We do like it if people get something out of this test instance and want to use it.. we really do. But it was not planned for this. The main reason this was created was that we have something where we can test puppet and phab config changes before merging in production. So please bear with us when things like sending email or user auth wasn't a consideration before we got actual requests like this.

The default Phabricator home page says:

• To test and experiment with Phabricator, go to phabricator.wmcloud.org instead.

Similarly both https://www.mediawiki.org/wiki/Phabricator and https://www.mediawiki.org/wiki/Phabricator/Help prominently advertise that instance. Should those be changed?

In T388022#11234012, @taavi wrote:

Should those be changed?

IMO it would be a shame if these were changed - it's just my personal opinion, but I think there's real value in having a test Phabricator instance available to people, to experiment around with things in. I (for one) have definitely found it very helpful since I've been able to access it :) I also probably wouldn't've known about the possibility of using it, if it wasn't for the advertisements of it that @taavi mentions.
(To be clear, I'm very grateful to collab-services/RelEng and everyone else for the work that's put into keeping the test instance running!)

On a slight side note: for what it's worth, it looks like the invitation on [[mw:Phabricator]] to experiment in (what I'm assuming was then called) the Phabricator Labs instance may date back to around 2014.

In T388022#11233681, Pppery wrote:

How hard would it be to support sending mail from phabricator.wmcloud.org on the cloud-vps side? That seems like it would solve this problem by itself.

I assume this question would welcome input from cloud-services-team.
Is it feasible? Or out of scope? Or something in-between?
Thanks in advance for sharing some additional setup/config knowledge.

Setting up the needed DKIM signing keys and other mail-related config is a non-trivial chunk of manual work, and I'm not sure off-hand how the additional DNS records play with the novaproxy logic that manages the proxy-related DNS records. So first I'd like to hear why Phorge can not use a differently configured sender address like every other piece of software that sends mail in Cloud VPS before deciding to support that configuration.

Judging by https://we.phorge.it/book/phorge/article/configuring_outbound_email/#outbound-quot-from-quot-and-quot, I wonder if this might be able to be resolved by setting https://phabricator.wmcloud.org/config/edit/metamta.default-address/ to something like [email protected] (or maybe [email protected])?

(disclaimer: this might not work & I have no way of testing it, but it seemed worth mentioning just in case :))

@A_smart_kitten Meanwhile I have written docs how to auth a user manually, put them on wikitech/linked on mediawiki.org and told my team. So if there are requests it won't be blocked on me personally. Cheers.

(let's see how common this actually is and only worry about it if it becomes a regular thing)

在T388022#11485293中，@Pppery写道：

What?

What? See the homepage of testing instance, we already made a form for request account

The homepage of the testing instance acknowledges the existence of this bug. That doesn't change the fact that the bug exists or make it an intended feature.

在T388022#11485297中，@Pppery写道：

The homepage of the testing instance acknowledges the existence of this bug. That doesn't change the fact that the bug exists or make it an intended feature.

This is a workaround, and this talk looks is trying to select a workaround, may need a new task for fix the original problem, but this task is not for this

This task is requesting the original problem be fixed. I have no idea what you are trying to say.

在T388022#11485299中，@Pppery写道：

This task is requesting the original problem be fixed. I have no idea what you are trying to say.

But actually, everybody ignored original problem and trying to find a workaround :)

... hence this task is, and should be, still open because it's not fixed yet