Add postgres init container to resolve permissions for some k3s deployments by rooftopcellist · Pull Request #1805 · ansible/awx-operator

rooftopcellist · 2024-04-02T21:47:38Z

Alternate PR for #1799

Resolves:

Postgres 15 pod: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied" #1770
Upgrade from 2.12.2 to 2.13.1 failed - mkdir: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied #1775

This approach allows users to simply set:

spec:
  postgres_data_volume_init: true

This removes the unused postgres_init_container_resource_requirements parameter which was never removed when the last postgres init container was removed. I cannot think of a good reason to not just use postgres_resource_requirements.

cc @kurokobo

ISSUE TYPE

New or Enhanced Feature

Add postgres init container if postgres_data_volume_init is true This allows users to run arbitrary commands This is aimed to solve the issue where users may need to chmod or chown the postgres data volume for user 26, which is the user that is running postgres in the sclorg image. For example, one can now set the follow on the AWX spec: spec: postgres_init_container_commands: | chown 26:0 /var/lib/pgsql/data chmod 700 /var/lib/pgsql/data In addition, remove the unneeded postgres_init_container_resource_requirements parameter Signed-off-by: Seth Foster <[email protected]> Co-authored-by: Christian M. Adams <[email protected]>

roles/installer/templates/statefulsets/postgres.yaml.j2

config/manifests/bases/awx-operator.clusterserviceversion.yaml

config/crd/bases/awx.ansible.com_awxs.yaml

rooftopcellist · 2024-04-03T16:39:36Z

@kurokobo I just realized that in your awx-on-k3s repo, there are a couple places where you reference postgres_init_container_resource_requirements

If a user has this specified on their AWX spec on upgrade or fresh install, they will get this error:

Error from server (BadRequest): error when creating "base": AWX in version "v1beta1" cannot be handled as a AWX: strict decoding error: unknown field "spec.postgres_init_container_resource_requirements"

It's probably safer for me to just add back in that parameter and mark it as deprecated, and add a backwards compat shim. I'll add that now.

…w param Deprecate postgres_init_container_resource_requirements param in favor of postgres_resource_requirements param Co-authored-by: craph <[email protected]> Co-authored-by: kurokobo <[email protected]>

rooftopcellist · 2024-04-03T17:16:07Z

I just added the changes mentioned so that the validator does not error if the user has postgres_init_container_resource_requirements defined. It also accepts that parameter and uses it preferentially over postgres_resource_requirements for the init container only if provided. Though I marked it as deprecated so that we can eventually remove it as there is no real need to have two separate variables for this imo.

Testing

I tested this out with the awx-on-k3s repo with the follow AWX spec:

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  # all of the defaults from https://github.com/kurokobo/awx-on-k3s/blob/main/base/awx.yaml

  postgres_init_container_resource_requirements: {}
  postgres_data_volume_init: true

I confirmed that the deployment came up successfully, and that the pg init param works:

$ kubectl -n awx get pod awx-postgres-15-0 -o yaml | grep resources: -B1
...
    name: init
    resources: {}

Then also tested with:

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  # all of the defaults from https://github.com/kurokobo/awx-on-k3s/blob/main/base/awx.yaml

  # postgres_init_container_resource_requirements: {} #commented out
  postgres_data_volume_init: true
  postgres_init_container_commands: |
    chown 26:0 /var/lib/pgsql/data
    chmod 700 /var/lib/pgsql/data

I confirmed that the deployment came up successfully.

Finally, I did one more with postgres_resource_requirements set to something custom just to make sure there were no regressions introduced there.

spec:
  postgres_resource_requirements:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      cpu: 500m
      memory: 512Mi

After the deployment finished, I checked that the requests and limits were set correctly.

$ kubectl -n awx get pod awx-postgres-15-0 -o yaml | grep resources: -B1 -A6
      protocol: TCP
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
      requests:
        cpu: 100m
        memory: 256Mi
--
    name: init
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
      requests:
        cpu: 100m
        memory: 256Mi

rooftopcellist · 2024-04-03T18:44:14Z

As follow-up I opened a PR to update the main branch of the awx-on-k3s repo cc @kurokobo

Add postgres_data_volume_init parameter to base/awx.yaml kurokobo/awx-on-k3s#330

kurokobo · 2024-04-04T01:41:08Z

@rooftopcellist

It's probably safer for me to just add back in that parameter and mark it as deprecated, and add a backwards compat shim. I'll add that now.

Thanks for the confirmation and sending PR, my guide is only unofficial one, so I think my guide should respect the Operator's implementation, and I don't think the Operator needs to adapt my guide. But yes, just marking the param as deprecated is safer. Thanks!

xamindar · 2024-10-06T07:14:32Z

Thank you, adding this finally got that dang pod to work. This needs to be default:
postgres_data_volume_init: true

kcjones91 · 2024-12-09T20:42:35Z

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx-demo
spec:
  admin_user: admin
  service_type: nodeport
  postgres_security_context_settings:
  fsGroup: 26
  postgres_data_volume_init: true
  postgres_init_container_commands: |
    chown 26:0 /var/lib/pgsql/data
    chmod 700 /var/lib/pgsql/data

I am running into this on my end. Probably an easy fix?

kubectl logs awx-demo-postgres-15-0 -n awx-dev -c init
chown: changing ownership of '/var/lib/pgsql/data': Permission denied
chmod: changing permissions of '/var/lib/pgsql/data': Permission denied

swipe@swipe-worker-1:/mnt$ ls -la /mnt/data/postgres/
total 0
drwxrwxrwx. 2 swipe swipe  6 Dec  9 15:48 .
drwxr-xr-x. 3 swipe swipe 22 Dec  9 13:17 ..

rooftopcellist changed the title ~~Add postgres init container~~ Add postgres init container to resolve permissions for some k3s deployments Apr 2, 2024

rooftopcellist requested review from TheRealHaoLiu and dsavineau April 2, 2024 21:48

This was referenced Apr 2, 2024

Add postgres init container #1799

Closed

Postgres 15 pod: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied" #1770

Closed

rooftopcellist force-pushed the k8s-job-permissions branch from c53fa96 to 634f52f Compare April 2, 2024 22:10

kurokobo reviewed Apr 3, 2024

View reviewed changes

roles/installer/templates/statefulsets/postgres.yaml.j2 Outdated Show resolved Hide resolved

craph reviewed Apr 3, 2024

View reviewed changes

config/manifests/bases/awx-operator.clusterserviceversion.yaml Outdated Show resolved Hide resolved

craph reviewed Apr 3, 2024

View reviewed changes

config/crd/bases/awx.ansible.com_awxs.yaml Outdated Show resolved Hide resolved

fosterseth approved these changes Apr 3, 2024

View reviewed changes

Correct the postgres path in description and the condition for the ne…

fbf62d3

…w param Deprecate postgres_init_container_resource_requirements param in favor of postgres_resource_requirements param Co-authored-by: craph <[email protected]> Co-authored-by: kurokobo <[email protected]>

rooftopcellist force-pushed the k8s-job-permissions branch from d434ceb to fbf62d3 Compare April 3, 2024 16:55

rooftopcellist merged commit a5211fe into ansible:devel Apr 3, 2024

rooftopcellist mentioned this pull request Apr 3, 2024

Add postgres_data_volume_init parameter to base/awx.yaml kurokobo/awx-on-k3s#330

Merged

rooftopcellist mentioned this pull request Apr 3, 2024

Upgrading to PostgreSQL 15 and moving to sclorg images ansible/galaxy-operator#80

Merged

This was referenced Apr 4, 2024

Unable to launch managed postgres db in cluster ansible/eda-server-operator#138

Closed

Add logic for upgrading from Postgres 13 to 15 ansible/eda-server-operator#122

Merged

This was referenced Apr 9, 2024

mkdir: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied #1813

Closed

Clean deployment on K3s with 2.14.0 fails with postgres 15 #1806

Closed

This was referenced Apr 11, 2024

Implement init container for PSQL to avoid permission error on specific implementation of PVs ansible/eda-server-operator#192

Closed

AWXBackup: mkdir: cannot create directory '/backups/tower-openshift-backup-2024-04-17-141257': Permission denied #1830

Open

hhk7734 mentioned this pull request May 2, 2024

Add postgres_pod_security_context #1809

Closed

kurokobo mentioned this pull request May 12, 2024

Assign ownership of backup pvc to postgres image's uid #1854

Open

fosterseth mentioned this pull request May 14, 2024

Upgrade from 2.12.2 to 2.13.1 failed - mkdir: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied #1775

Open

3 tasks

kcjones91 mentioned this pull request Dec 9, 2024

Add postgres init container to resolve permissions for some k3s deployments: init container commands fail with permission denied #1999

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add postgres init container to resolve permissions for some k3s deployments#1805

Add postgres init container to resolve permissions for some k3s deployments#1805
rooftopcellist merged 2 commits intoansible:develfrom
rooftopcellist:k8s-job-permissions

rooftopcellist commented Apr 2, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rooftopcellist commented Apr 3, 2024

Uh oh!

rooftopcellist commented Apr 3, 2024

Uh oh!

rooftopcellist commented Apr 3, 2024

Uh oh!

kurokobo commented Apr 4, 2024

Uh oh!

xamindar commented Oct 6, 2024

Uh oh!

kcjones91 commented Dec 9, 2024 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

rooftopcellist commented Apr 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ISSUE TYPE

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rooftopcellist commented Apr 3, 2024

Uh oh!

rooftopcellist commented Apr 3, 2024

Uh oh!

rooftopcellist commented Apr 3, 2024

Uh oh!

kurokobo commented Apr 4, 2024

Uh oh!

xamindar commented Oct 6, 2024

Uh oh!

kcjones91 commented Dec 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

rooftopcellist commented Apr 2, 2024 •

edited

Loading

kcjones91 commented Dec 9, 2024 •

edited

Loading