This doc describes how to test and develop changes to Zulip's mobile push notifications.
- General tips
- Testing server-side changes (iOS or Android)
- Testing client-side changes on Android
- Testing client-side changes on iOS
When testing Zulip's push notifications:
-
Try simulating a DM conversation between two users: one on the mobile device, and one from your desktop. For example, if you've logged in as "Iago" on your mobile device, log in as "Polonius" via a web browser, and send a DM from Polonius to Iago.
-
Test different types of messages that will cause different types of notifications: a 1:1 DM conversation, a group DM conversation, an @-mention in a channel, a channel with notifications turned on, even an @-mention in a 1:1 or group DM conversation.
-
Make sure "mobile push notifications" are on in the mobile user's Zulip settings!
- Try also turning on the "even while online" setting -- this is extremely helpful for testing notifications effectively, with basically the one exception of if you're specifically working on the "if online" aspect of the system.
-
Make sure notifications are enabled for Zulip in the device's system settings!
-
To get to these, find "Notifications" or "Apps & Notifications" in the system settings app, depending on OS and version; then find Zulip in the list. Or on Android, in the launcher give the app's icon a long-press -> "App Info" -> "Notifications".
-
Also check the settings there for whether and how the app's notifications will pop on the screen, make a sound, etc.
-
Particularly for the debug build on one's personal device, it's natural to disable notifications between development sessions to suppress duplicates... then forget to re-enable them.
-
-
Leave the app in the background: that is, switch to the launcher / home screen or to a different app to get it off the screen. Or keep it on screen, or force-kill it, to test different scenarios!
When making changes to the Zulip server, use these steps to test how they affect notifications.
First, three one-time setup steps:
-
Create a
zproject/custom_dev_settings.pywith the following line:ZULIP_SERVICES_URL = 'https://push.zulipchat.com'
This matches the normal setting for a production install of the Zulip server (set by
zproject/default_settings.py).The Zulip server will helpfully print a line on startup to remind you that this settings override file exists.
-
Register your development server with our production bouncer by running a command like this one, but with
[yourname]replaced by your name:EXTERNAL_HOST=[yourname].zulipdev.com:9991 python manage.py register_server --agree-to-terms-of-service(The exact value of
EXTERNAL_HOSTdoesn't matter; in particular it doesn't need to match theEXTERNAL_HOSTthat you use when actually running the server, withtools/run-dev. It does need to be distinct from any names that others have already registered.)This is a variation of our instructions for production deployments, adapted for the Zulip dev environment.
You should only have to do this step once, unless you build a new Zulip server dev environment from scratch. The credentials which this command registers with the bouncer are kept in the
zproject/dev-secrets.conffile. -
If you were already running
tools/run-dev, quit and restart it after these setup steps.
Then, each time you test:
-
Run
tools/run-devaccording to the instructions in dev-server.md. Then follow that doc's instructions to log into the dev server. Use the release build of the app -- that is, the Zulip app installed from the App Store or Play Store. -
Follow the general tips above to cause a push notification. For example, log in from a browser as a different user, and send the mobile user a DM.
You should see a push notification appear on the mobile device!
If you don't, check the general tips above. Then ask in chat and let's debug.
Happily, this is straightforward: just edit, build, and run the app the same as for any other change.
If you're editing Kotlin code (not only Dart), then
Flutter's hot reload and hot restart won't pick that up;
you'll need to stop the app and use flutter run afresh.
Apple makes this much more of a pain than it is on Android: APNs (*) does not allow our production bouncer to send notifications to a development build of the app.
(*) i.e. "Apple Push Notification service" -- Apple is very Apple about this name, styling it without an article and with the idiosyncratic capitalization shown.
This workaround means using a development server. You'll first need to set up the dev server for mobile development.
You can tell your development server to talk to Apple's APNs "sandbox" server, instead of its server meant for production, but you'll need a certificate signed by Apple authorizing you to do so. Some background on that is here.
-
First, generate a certificate signing request (CSR) and corresponding private key. Use the
tools/setup/apns/prep-certscript from the Zulip server tree:$ tools/setup/apns/prep-cert request /tmp/apns.key /tmp/apns.csr -
Greg is authorized in Apple Developer to upload the CSR and obtain the actual certificate, so you should send
apns.csrto him and ask him to do that. He'll follow the steps at https://developer.apple.com/account/resources/certificates/add with:- Cert type: “Apple Push Notification service SSL (Sandbox)" (not "Sandbox & Production")
- App ID: 66KHCWMEYB.org.zulip.Zulip
to obtain a certificate file
aps_development.cer, and send it back to you. -
Combine the certificate with the key using the same tool:
$ tools/setup/apns/prep-cert combine \ /tmp/apns.key /tmp/aps_development.cer zproject/apns-dev.pemThe file
zproject/apns-dev.pemis the output of all the steps up to this point. You can now delete the other files/tmp/apns.key,/tmp/apns.csr, and/tmp/aps_development.cer. -
Generate a bouncer public key and the corresponding private key by running the following management command on the dev server:
$ ./manage.py manage_push_registration_encryption_keys --addOutput is a public/private keypair, base64-encoded:
Added a new key pair: - Public key: <etc.> - Private key: <etc.>You'll use the "Public key" value in a later step.
-
Restart
tools/run-devto let the server pick up the changes: the push registration encryption keys generated in step 4, and the filezproject/apns-dev.pemfor communicating with the APNs sandbox server. -
Then in the app, source
bouncerPublicKeyinlib/model/push_device.dartfrom the "Public key" value from step 4, instead of the corresponding value for the production bouncer.Then hot restart the app to let the device registration take place with the new public key.
You should now be getting notifications on your iOS development build! (On a real device, though, a debug build shows E2EE notifications only as "New notification"; see the troubleshooting entry below.)
If it's not working, first check that mobile notification settings are on, using the web app's settings interface. See also the troubleshooting items below.
If none of those resolve the issue, please ask for help in #mobile-dev-help on chat.zulip.org, so we can debug.
When running the debug build (flutter run) of the app on a
real iPhone, the FlutterEngine in the NotificationService extension
will not start, which will result in the NotificationService
extension falling back to displaying the original notification's
"alert", which for E2EE notifications says "New notification".
And the following error will be logged:
Cannot create a FlutterEngine instance in debug mode without Flutter tooling or Xcode.
To launch in debug mode in iOS 14+, run flutter run from Flutter tools, run from an IDE with a Flutter IDE plugin or run the iOS project from Xcode.
Alternatively profile and release mode apps can be launched from the home screen.
We think that on a real iPhone, iOS only allows JIT-compiled
code to run when a debugger is attached to the process.
This is generally the case for the Runner process (the main app)
when doing flutter run.
The NotificationService extension runs in a separate process,
so its debug-mode FlutterEngine will not start without
a debugger attached.
To work around this, run the release version of the app
(flutter run --release), or run the debug version on the
iOS Simulator.
Logs will not show up in flutter run or flutter logs
because the code running in the NotificationService target runs in
a separate process. And Flutter's tooling is not routing/reading
logs from that process.
One workaround is to use the Console app for reading the logstream from the NotificationService process by following these steps:
- Open the Console app.
- Select your device or simulator in the left sidebar.
- In the search bar, type "process:NotificationService" and press Return/Enter.
- Press Start (with a "play" icon) to start streaming the logs with the specified filter.
- Receive a notification, and that will result in some logs being emitted.
The Flutter tooling doesn't attach to the secondary FlutterEngine
running in the NotificationService extension when doing a
flutter run. There are no known workarounds for this;
the only way to update the running code is by doing a complete
build again (stopping flutter run and starting it again).
After the above setup is done, when the dev server tries to send a notification (e.g. because you had some other user send a DM to the user you've logged in as from the mobile app), you might get an error like this one (reformatted for readability):
INFO [zerver.lib.push_notifications] Sending push notifications to mobile clients for user 11
INFO [zr] 127.0.0.1 POST 400 8ms (db: 3ms/6q) /api/v1/remotes/push/notify [can_push=False/Missing data] (zulip-server:… via ZulipServer/11.0-dev+git)
INFO [zr] status=400, data=b'{"result":"error","msg":"Your plan doesn\'t allow sending push notifications.","code":"INVALID_ZULIP_SERVER"}\n', uid=zulip-server:…
WARN [django.server] "POST /api/v1/remotes/push/notify HTTP/1.1" 400 109
ERR [] Problem handling data on queue missedmessage_mobile_notifications
Traceback (most recent call last):
…
File "/srv/zulip/zerver/lib/remote_server.py", line 195, in send_to_push_bouncer
raise PushNotificationBouncerError(
zerver.lib.remote_server.PushNotificationBouncerError:
Push notifications bouncer error: Your plan doesn't allow sending push notifications.
It's not clear why this happens. Try stopping the server
(i.e. tools/run-dev) and starting it again. On the
one occasion this error has been seen
as of this writing, the error went away after doing so.
When the dev server tries to send a notification, you might get in the server log an error like this one (reformatted for readability):
INFO [zerver.lib.push_notifications] Sending push notifications to mobile clients for user 11
…
INFO [zerver.lib.push_notifications] APNs: Removing invalid/expired token FD524D9B9018CB7340051C26E4F99304F71D4B37E2164265AB0BBA3543225331 (BadDeviceToken)
…
INFO [zerver.lib.push_notifications] Deleting push tokens based on response from bouncer: Android: [], Apple: ['FD524D9B9018CB7340051C26E4F99304F71D4B37E2164265AB0BBA3543225331']
This can happen when you're using a release build of the app (such as the one from TestFlight or the App Store), which the server's sandbox-only certificate is unable to send to. To fix it, use a debug build of the app.
When the dev server tries to send a notification, you might get in the server log an error like this one (reformatted for readability):
INFO [zerver.lib.push_notifications] Sending push notifications to mobile clients for user 11
INFO [zilencer.views] Sending mobile push notifications for remote user 1db86aa5-1327-4c03-8db8-c8189aa23b0c:<id:11><uuid:29acb8b2-4500-4268-80e5-219fd9af2c2d>: 0 via FCM devices, 1 via APNs devices
INFO [zerver.lib.push_notifications] APNs: Sending notification for remote user 1db86aa5-1327-4c03-8db8-c8189aa23b0c:<id:11><uuid:29acb8b2-4500-4268-80e5-219fd9af2c2d> to 1 devices (skipped 0 duplicates)
WARN [zerver.lib.push_notifications] APNs: Failed to send for user <id:11><uuid:29acb8b2-4500-4268-80e5-219fd9af2c2d> to device 34FBEF8A439C8F9B6CC6AEFC42CDFD2A9618FFA3E0C8C50B9A56CB96D3F2D727: TopicDisallowed
This can happen when the app's bundle ID (which gets used as an APNs "topic") is different from the one your server's APNs certificate is for. To fix it, either get a certificate for the desired topic, or edit the app's bundle ID to match the certificate.
To edit the bundle ID, try a command like:
$ perl -i -0pe 's/org\.zulip\.Zulip/OTHER_NAME/g' \
-- $(git grep -l org.zulip.Zulip)
Make a release build of the app, and upload it as an alpha. Update your device to the alpha from TestFlight, and test there.
This works OK, but it's slow: about 5m to build and upload, and another few on Apple's servers before it's available in alpha. In total perhaps 10m from "save and hit go" to actually getting to test.
The long iteration cycle can be tolerable for changes that are very likely to need only zero to one revision -- because they're very small, or already well tested on Android -- but makes serious development basically infeasible.
For making a throwaway alpha build like this approach calls for, you'll want to set the iOS build number.
In a Flutter app, this is set through pubspec.yaml, as
the portion after the + in the version field.
(The portion before the + is the user-facing version number.)
The build number needs to be set so that the new build can coexist on TestFlight with previous and future builds. Specifically, the rule appears to be:
-
The user-facing version number must be strictly greater than the last one published to the App Store.
-
The user-facing version number can stay the same across a series of alpha and/or beta releases, but the build number must strictly increase.
So if the last version in main is already in the App Store (and not still in alpha or beta), then we'll want to:
-
Increment the user-facing version number.
-
To avoid confusion when we next release, use a build number starting with "0." -- e.g.,
0.0.1or0.20260414.1.
For example if pubspec.yaml currently has a line like this,
reflecting a version that's already in the App Store:
version: 30.0.271+271
then you might edit it to read like this:
version: 30.0.272+0.20260414.1
Better yet, for many development purposes, would be to be able to dispense with a local dev server, and still get notifications from zulipchat.com and chat.zulip.org in a development build of the app.
The ingredient that'd be needed for that is to get a production Zulip server, or the production bouncer it talks to, to optionally send notifications through Apple's sandbox instance of APNs.
-
One way to arrange that might be to have the production bouncer talk to either the production or staging instance of APNs; the client tell the Zulip server which one to use for that client's notifications, and the Zulip server pass that information on to the bouncer.
-
Another might be to leave the bouncer out of it, and have the Zulip server talk directly to the sandbox APNs when the client asks it to, rather than talk to the bouncer.