Skip to content

Yellow-Dog-Man/resonite-wiki

BranchesTags

Repository files navigation

Resonite Wiki

A collection of files required to spin up the Resonite Wiki using docker compose.

Goals

  1. Open Source
    • Except for Secrets of Course
  2. Fully Documented
  3. Avoid Modifying the Containers at runtime.
  4. Proper Secret Management
  5. Resolving All public issues involving the wiki
  6. Documented
  7. Simple (As possible)

Context

Our current wiki(wiki.resonite.com) is running an older setup that mixes Docker hosting with extensive editing and tweaking of the Docker container.

Changes to the docker container at runtime are difficult to manage and can be lost during rebuilding the container etc.

We're rebuilding this setup, to align with the listed goals.

You can read more about this in a bunch of GitHub Issues:

Where we lamented about this issue before building this repo to fix it.

Files

  • extensions/
    • Contains all our Mediawiki Extensions. These are Git Submodules
  • skins/
    • Contains all our Skins. These are git Submodules
  • docker-compose.yaml
    • Compose file that sets everything up
  • Dockerfile
    • Contains our custom dockerfile for the mediawiki installation.
    • This can bake Extensions and skins into the docker container, avoiding the overhead and creating a stable image
  • config
    • A collection of configuration files for multiple services.
    • wiki
      • managed and linked into the wiki
      • LocalSettings.php is linked into the container via docker in the usual mechanism.
      • Orchestrated for separation of concerns and maintenance
      • Feel free to re-organize this, the initial split is arbitrary.
  • scripts
    • Helpful scripts to handle some automated tasks.

Images

Images in mediawiki are served over Cloudflare R2 with a CDN.

  • Asset Uploads and Management(PUT,PATCH,HEAD etc) use the AWS Mediawiki extension to upload to R2
    • Cloudflare R2 has an S3 Compatible API
  • Asset Retrieval (GET) for end users, uses Cloudflare's CDN.
graph LR;

U[Upload];
D[Download];
M[Management/Updates];

CFCDN[Cloudflare CDN];
CFS3[Cloudflare S3 API];
R2[R2 Bucket];

D --> MW[media.wiki.resonite.com] --> CFCDN --> R2;
U --> CFS3 --> R2;
M --> CFS3;
Loading

CDN

We use Cloudflare for our CDN. The wiki is routed via CF to improve page load time.

Cache Rules

Written out in pseudo-code, can be applied via Cloudflare Dashboard and eventually via API maybe?

Cache static assets aggressively

When incoming requests match:
  Hostname equals "wiki.resonite.com"
  AND (URI Path starts with "/images/" OR URI Path contains "/load.php")

Then:
  Cache eligibility: Eligible for cache
  Edge Cache TTL: 1 month

Never cache special/dynamic pages

When incoming requests match:
  Hostname equals "your-wiki.com"
  AND (URI Path contains "Special:" OR URI Path contains "api.php")

Then:
  Cache eligibility: Bypass cache

Cache article pages conservatively

When incoming requests match:
  Hostname equals "your-wiki.com"
  AND URI Path starts with "/wiki/"

Then:
  Cache eligibility: Eligible for cache
  Edge Cache TTL: Respect origin

Full CDN Diagram

A full diagram showing CDN routing:

graph LR;


CFCDN[Cloudflare Image CDN];
CFCDN2[Cloudflare Wiki CDN];
CFS3[Cloudflare S3 API];
R2[R2 Images Bucket];

WEB[Web];
W[Wiki];

SQL[Maria DB];
ES[Open Search];

WEB -- Read Images --> MW[media.wiki.resonite.com] --> CFCDN --> R2;
WEB -- Write Images --> W --> CFS3;
CFS3-->R2;

WEB --Read Articles--> CFCDN2 --> W;
WEB --Write Articles --> W;
W --Purge Cache--> CFCDN2;
W <--> SQL;
SQL --> R2B[R2 Backups Bucket]
W <--Search--> ES;
Loading

Cron

Many scheduled or cron related tasks are handled by ofelia. Such as:

  • The MediaWiki Job Queue
    • php /var/www/html/maintenance/run.php runJobs --maxtime=3600
  • automysqlbackup which creates backups
  • The RClone Sync process that sends SQL backups to R2. See #backups

Search

Search requires 3 Extensions:

  • Advanced Search
    • Gives the power users the ability to search really well, UI stuff
  • Cirrus Search
    • Search infrastructure and search index management
  • Elastica
    • Basically an SDK for Elasticsearch

Search via Cirrus, supports ElasticSearch and OpenSearch, we have chosen OpenSearch for now.

Config

Bare bones, we have no idea what we're doing. PR's welcome.

Init

Once everything is up and running:

TODO: update cirrus docs on mw docs, it uses old script paths

php maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --startOver
php maintenance/run.php CirrusSearch:ForceSearchIndex
php maintenance/run.php CirrusSearch:UpdateSuggesterIndex

Citizen Integration

Debugging

Run this: curl resonite-wiki-opensearch:9200/_cat/indices?v, to check for connectivity and indexes.

PDF Export

PDF export uses the ElectronPdfService extension backed by a Proton service. Proton is a Node.js service that renders wiki pages to PDF using headless Chromium via Puppeteer. This provides proper rendering of templates, styling, and all page content.

The request flow is:

  1. User triggers a PDF download from the wiki sidebar or Special:DownloadAsPdf page
  2. ElectronPdfService extension redirects to /api/rest_v1/page/pdf/{PageName}
  3. Apache proxies the request to the Proton container via mod_proxy
  4. Proton fetches the page HTML from the wiki, launches Chromium, and generates a PDF
  5. The PDF is streamed back to the user

Services

  • resonite-wiki-proton - The Proton container running on port 3030
  • Configuration: config/proton/config.yaml
  • Dockerfile: Dockerfile.proton

Database

Seeding

To create a compatible backup from the original wiki:

mysqldump -h [other-db-host] -u [other-db-user] -p[other-db-password] [other-db-name] | gzip > daily_wiki_db_$(date +%Y%m%d_%H%M%S).sql.gz

Manually upload it to the R2_BACKUP_BUCKET_NAME, start the docker stack and the old data will be ingested.

Public Seeding

We're unable to make the full SQL dump available to the public, it contains backups of user data.

TODO: We can use Data-Dump extensions for .XML downloads of the content, or expose selected table backups. LONG-TERM-TODO: separate data and account SQL dumps, treat them differently.

Backups

When the docker compose profile backups is include in startup: docker compose up --profile backups:

  1. Every day at 12:00AM server time, an automated SQL backup is performed.
    • This creates a tarbell of the database
  2. Every day at 01:00AM server time, an automated script runs, which syncs the database backups to Cloudflare R2

The delay between the two operation is to give the SQL backup time to finish. It usually takes a minute or two though.

We use automysqlbackup which creates and automatically rotates backups into a folder structure for latest, daily, weekly and monthly backups.

Manual Backup Commands

  • Make a backup with: docker exec resonite-wiki-utilities /usr/local/bin/manualBackup.sh
  • Sync the backup to r2 with: docker exec resonite-wiki-utilities rclone sync /backup r2:wiki-backups/database-backups --transfers 4 --checkers 8 --log-level INFO

Environment Variables

The automysqlbackup container uses these environment variables for configuration:

  • DBHOST: MySQL server hostname (resonite-wiki-database)
  • USERNAME: MySQL username for backups (root)
  • PASSWORD_FILE: Path to file containing MySQL root password
  • BACKUPDIR: Backup directory inside container (/backup)
  • CRON_SCHEDULE: Cron schedule for automated backups (0 0 * * *)
  • LATEST: Create latest backup symlink (yes)
  • DBNAMES: always set this to wiki_db

Bucket Configuration

  • Bucket Name: wiki-backups
  • Lifecycle policies:
    • Transition to Long Term storage after 5 days
    • Delete after 1 year.

Backup Syncing

Backup syncing is handled by RClone. We use the sync, command from RClone which will delete and move around files to match the Automysqlbackup folder structure.

Commands

  • docker compose up - starts up everything with defaults
  • docker compose up --profile * - include all profiles
    • recommended to use runAll.sh

TODO

  • On First install, restore the most up to date backup of DB
  • Backup
    • Take regular automatic backups and send them to R2
  • Images
    • Images are now on R2
  • Advanced Search Setup
    • Elastic Search is needed Q.Q
  • Cloudflare setup
    • Got started here with the R2 images but, need to do more to CDN the rest.
    • External Stuff
      • Caddy Setup
      • CF Rules
      • Switchover Caddy stuff, "wiki2.resonite.com"
  • Requested extensions from GH
  • Swap to Ophelia from MWJobrunner and other cronjobs, ophelia is newer. MW Job Runner bulky no need. Ophelia make it better/easier
  • Check Extensions list again
  • CheckGH issue list once more too
  • Swap to https://github.com/netresearch/ofelia
  • Test Backup command
    • Ofelia handles this but I haven't actually tested it yet, I must.
  • Isolate and minimize Env vars
    • We are currently sharing .env with all containers this gives everyone everything which is bad
  • Standardize R2 config, we've got it in two places, look for password file options?
    • Combining the container would be easier.
  • Fix composer cache, copy composer.local to installation in docker file before run composer update FASTER

Immediately After Upgrade

These items require us to isolate Wiki Data to the new wiki rather than continuing to copy across a backup from old to new. Rather than doing hacky stuff, its easier to wait till we're on new.

After "Upgrade"/Long term Issues

These ones take a bunch more effort, which means they are separate initiatives.

Switchover plan

  1. Mark current wiki as read-only.
  2. Start up new wiki as wiki2.resonite.com, it will be rw.
  3. Collect feedback, prioritize Reading content
  4. Shutdown old wiki
  5. Rename wiki2 -> wiki

Resources

Composer Stuff

Some useful composer commands, used to find dependencies within extensions/skins.

  • grep -r "composer/installers.*1\.\*,>=1.0.1" extensions/*/composer.json skins/*/composer.json
  • grep -r "johnkary/phpunit-speedtrap" extensions/*/composer.json skins/*/composer.json
  • grep -r "firebase/php-jwt.*5\.2" extensions/*/composer.json skins/*/composer.json
  • grep -r "firebase/php-jwt.*6.10.0" extensions/*/composer.json skins/*/composer.json
  • grep -r "symfony/yaml.*6" extensions/*/composer.json skins/*/composer.json

Logs

  • docker compose logs -f --tail=20

Run maintenance via docker

  • docker exec resonite-wiki php /var/www/html/maintenance/run.php namespaceDupes --fix
  • docker exec resonite-wiki php /var/www/html/maintenance/run.php rebuildLocalisationCache
    • You have to do this one like regularly >.<
  • docker exec resonite-wiki php /var/www/html/maintenance/run.php update

Drop Down styling

https://github.com/StarCitizenTools/mediawiki-skins-Citizen/blob/main/resources/skins.citizen.scripts/dropdown.js

Mediawiki Log Settings

Do not enable these unless there is a problem, they make a lot of noise.

// Slow DB
$wgDebugDumpSql = true;
$wgDebugLogGroups['DBQuery'] = "/var/log/mediawiki/db-queries.log";
$wgDebugLogGroups['slow-queries'] = "/var/log/mediawiki/slow-queries.log";

// Log requests
$wgDebugLogGroups['request'] = "/var/log/mediawiki/requests.log";

//Cache/session issues:
$wgDebugLogGroups['ObjectCache'] = "/var/log/mediawiki/cache.log";
$wgDebugLogGroups['SessionManager'] = "/var/log/mediawiki/session.log";

// If users are having login/permission problems
$wgDebugLogGroups['authentication'] = "/var/log/mediawiki/auth.log";

// Verbose debugging - generates LOTS of output
$wgDebugLogFile = "/var/log/mediawiki/debug.log";

Manually add tables for CheckUser

docker exec resonite-wiki cat /var/www/html/extensions/CheckUser/schema/mysql/tables-generated.sql | docker exec -i <db_container_name> mysql -u<user> -p<pass> wiki_db

For some reason our install refused to install this automatically in the Update Script so we used the above to install the tables. I imagine its because our database is just screwed up from the old setup.

OAuth Notes

Users with duplicate accounts / "1" accounts

run this: select user_name, wsoauth_remote_name, wsoauth_provider_id from wsoauth_multiauth_mappings, user where user_id = wsoauth_user and user_name like '%1';

Then run this: DELETE wsoauth_multiauth_mappings FROM wsoauth_multiauth_mappings, user WHERE user_id = wsoauth_user AND user_name LIKE '% 1';

to fix.

Copying Config files into Docker

Our .dockerignore filters out most config files, preventing them from being COPY'ied to containers. This is usually because mounting the container via volumes is a better strategy.

Should you need to override this, you will have to add an exception to the .dockerignore file for your config. This is common for Apache tweaks to the main docker container, and the whole Proton/PDF Setup.

TODO: Resolve this smartly using paths etc.

About

Open source orchestration/setup for wiki.resonite.com. (Docker Compose Stuff Mostly)

Resources

Readme
Activity
Custom properties

Stars

6 stars

Watchers

0 watching

Forks

2 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages