Skip to content

Extract "package manager" functionality into plugins#2557

Merged
psss merged 20 commits intomainfrom
package-managers
Mar 19, 2024
Merged

Extract "package manager" functionality into plugins#2557
psss merged 20 commits intomainfrom
package-managers

Conversation

@happz
Copy link
Copy Markdown
Contributor

@happz happz commented Dec 9, 2023 •
edited by psss
Loading

More and more we see the need for primitives like "install a package". Plugins start to require packages to be present on guests, we already do have prepare/install plugin, and the upcoming all mighty install plugin. To support all these functions, tmt.package_managers would provide necessary primites, allowing implementations to be shipped as plugins.

The patch starts building the primitives, converting tmt.steps.prepare.install to use them in the process.

Pull Request Checklist

  • implement the feature
  • extend the test coverage
  • include a release note

@happz
Copy link
Copy Markdown
Contributor Author

happz commented Dec 22, 2023

So, adding a prototype support for apt to help me identify parts where rpm/yum/dnf are assumed. And it works :)

/plan
    discover
        how: fmf
        order: 50
        directory: /home/happz/git/tmt/tmp
        summary: 1 test selected
            /test
    provision
        how: local
        order: 50
        guest: localhost
        multihost name: default-0
        arch: x86_64
        distro: Ubuntu 22.04.3 LTS
        kernel: 5.10.102.1-microsoft-standard-WSL2
        package manager: apt
        selinux: no
        is superuser: no
        summary: 1 guest provisioned
    prepare
        queued push task #1: push on default-0
        
        push task #1: push on default-0
    
        queued prepare task #1: requires on default-0
        
        prepare task #1: requires on default-0
        how: install
        summary: Install required packages
        name: requires
        order: 70
        package: 1 package requested
            tree
            cmd: dpkg-query --show tree || sudo apt install -y  tree
            err: dpkg-query: no packages found matching tree
            err: 
            err: WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
            err: 
            out: Reading package lists...
            out: Building dependency tree...
            out: Reading state information...
            out: The following package was automatically installed and is no longer required:
            out:   podman-machine-cni
            out: Use 'sudo apt autoremove' to remove it.
            out: The following NEW packages will be installed:
            out:   tree
            out: 0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
            err: dpkg-preconfigure: unable to re-open stdin: No such file or directory
            out: Need to get 0 B/47.9 kB of archives.
            out: After this operation, 116 kB of additional disk space will be used.
            out: Selecting previously unselected package tree.
            out:
                (Reading database ... 
                (Reading database ... 5%
                (Reading database ... 10%
                (Reading database ... 15%
                (Reading database ... 20%
                (Reading database ... 25%
                (Reading database ... 30%
                (Reading database ... 35%
                (Reading database ... 40%
                (Reading database ... 45%
                (Reading database ... 50%
                (Reading database ... 55%
                (Reading database ... 60%
                (Reading database ... 65%
                (Reading database ... 70%
                (Reading database ... 75%
                (Reading database ... 80%
                (Reading database ... 85%
                (Reading database ... 90%
                (Reading database ... 95%
                (Reading database ... 100%
                (Reading database ... 66655 files and directories currently installed.)
            out: Preparing to unpack .../tree_2.0.2-1_amd64.deb ...
            out: Unpacking tree (2.0.2-1) ...
            out: Setting up tree (2.0.2-1) ...
            out: Processing triggers for man-db (2.10.2-1) ...
            out: 
            out: Failed to retrieve available kernel versions.
            out: 
            out: Failed to check for processor microcode upgrades.
            out: 
            out: No services need to be restarted.
            out: 
            out: No containers need to be restarted.
            out: 
            out: User sessions running outdated binaries:
            out:  happz @ /dev/pts/0: bash[31121]
            out:  happz @ /dev/pts/1: node[1295,31204,31252,31293,31304,31394,31431], python[31333,31334,31489], sh[31194,31195,31200,31488]
            out:  happz @ /dev/pts/7: bash[31368]
            out: 
            out: No VM guests are running outdated hypervisor (qemu) binaries on this host.
    
        queued pull task #1: pull on default-0
        
        pull task #1: pull on default-0
    
        summary: 1 preparation applied
    execute
        queued execute task #1: default-0 on default-0
        
        execute task #1: default-0 on default-0
        how: tmt
        order: 50
        exit-first: false
            test: /test
                cmd: dpkg-query --show tree
                out: tree       2.0.2-1
                00:00:00 pass /test (on default-0) [1/1]

    
        summary: 1 test executed
    report
        how: display
        order: 50
            pass /test
                output.txt: /var/tmp/tmt/run-441/plan/execute/data/guest/default-0/test-1/output.txt
                content: tree   2.0.2-1

        summary: 1 test passed
    finish
    
    Prune '/plan' plan workdir '/var/tmp/tmt/run-441/plan'.
        summary: 0 tasks completed

total: 1 test passed

Unfortunately, I can't run tmt's tests because:

  1. there's no tmt package for Ubuntu, and
  2. there's no beakerlib package for Ubuntu :)
/plans/features/core
summary: Verify core functionality
    discover
        how: fmf
        directory: /home/happz/git/tmt
        hash: fb6235f1
        filter: tier: 0, 1
        summary: 1 test selected
            /tests/core/ls
    provision
        how: local
        guest: localhost
        multihost name: default-0
        arch: x86_64
        distro: Ubuntu 22.04.3 LTS
        kernel: 5.10.102.1-microsoft-standard-WSL2
        package manager: apt
        selinux: no
        is superuser: no
        summary: 1 guest provisioned
    prepare
        queued push task #1: push on default-0
        
        push task #1: push on default-0
    
        queued prepare task #1: default-0 on default-0
        queued prepare task #2: default-1 on default-0
        queued prepare task #3: requires on default-0
        
        prepare task #1: default-0 on default-0
        how: install
        package: 2 packages requested
            jq
            python3-pip
            cmd: dpkg-query --show jq python3-pip || sudo apt install -y  jq python3-pip
        
        prepare task #2: default-1 on default-0
        how: shell
        overview: 2 scripts found
        script: pip3 install --user yq || pip3 install yq
            cmd: /var/tmp/tmt/run-442/plans/features/core/tree/tmt-prepare-wrapper.sh-default-1-default-0
        script: yq --help
            cmd: /var/tmp/tmt/run-442/plans/features/core/tree/tmt-prepare-wrapper.sh-default-1-default-0
        
        prepare task #3: requires on default-0
        how: install
        summary: Install required packages
        name: requires
        package: 2 packages requested
            beakerlib
            tmt
            cmd: dpkg-query --show beakerlib tmt || sudo apt install -y  beakerlib tmt
        fail: Command 'dpkg-query --show beakerlib tmt || sudo apt install -y  beakerlib tmt' returned 100.
    finish
    
        summary: 0 tasks completed

plan failed

The exception was caused by 1 earlier exceptions

Cause number 1:

    prepare step failed

    The exception was caused by 1 earlier exceptions

    Cause number 1:

        Command 'dpkg-query --show beakerlib tmt || sudo apt install -y  beakerlib tmt' returned 100.

        stdout (3 lines)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        Reading package lists...
        Building dependency tree...
        Reading state information...
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

        stderr (7 lines)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        dpkg-query: no packages found matching beakerlib
        dpkg-query: no packages found matching tmt

        WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

        E: Unable to locate package beakerlib
        E: Unable to locate package tmt
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

@LecrisUT
Copy link
Copy Markdown
Member

LecrisUT commented Jan 10, 2024

Both beakerlib and tmt should be straightfoward packageable for debian/ubuntu. Are there any contacts that can help with that?

This PR would also help with tmt github action

@lukaszachy
Copy link
Copy Markdown
Contributor

lukaszachy commented Jan 30, 2024

@happz Shouldn't be the package_manager property of Guest? For my use case I'd need to use it outside of prepare-install. For the rsync installation use case it can happen in provision/execute/finish as well.

@happz
Copy link
Copy Markdown
Contributor Author

happz commented Jan 30, 2024

@happz Shouldn't be the package_manager property of Guest? For my use case I'd need to use it outside of prepare-install. For the rsync installation use case it can happen in provision/execute/finish as well.

I believe it is, see https://github.com/teemtee/tmt/pull/2557/files#diff-f684593942660856c3bdc810b597c3956b4107716a554eece2c2bf08344a9503R635. There's also Guest.facts.package_manager which is a string, name of the manager as discovered by fact collection, but once that one is known, Guest.package_manager will return a proper instance of tmt.package_managers.PackageManager. As you can see at https://github.com/teemtee/tmt/pull/2557/files#diff-f684593942660856c3bdc810b597c3956b4107716a554eece2c2bf08344a9503R1108, I think I even made it to the point where rsync installation would use this attribute.

@lukaszachy
Copy link
Copy Markdown
Contributor

lukaszachy commented Jan 30, 2024

Ouch. I completely missed that I am looking on different file... Thus this PR is usable for my use case ;)

@lukaszachy lukaszachy mentioned this pull request Jan 30, 2024
Merged
8 tasks
@happz happz added this to the 1.32 milestone Feb 7, 2024
@happz happz mentioned this pull request Feb 8, 2024
Closed
@lukaszachy
Copy link
Copy Markdown
Contributor

lukaszachy commented Feb 8, 2024

Unfortunately, I can't run tmt's tests because:

1. there's no tmt package for Ubuntu, and
2. there's no beakerlib package for Ubuntu :)

I guess this asks for "I have all bits installed already use case" and/or better check (e.g command -v for binaries or existence of /usr/share/beakerlib/beakerlib.sh for beakerlib...)

@happz
Copy link
Copy Markdown
Contributor Author

happz commented Feb 8, 2024

/packit test

lukaszachy
lukaszachy reviewed Feb 8, 2024
View reviewed changes
LecrisUT
LecrisUT reviewed Feb 8, 2024
View reviewed changes
@LecrisUT
Copy link
Copy Markdown
Member

LecrisUT commented Feb 8, 2024

Unfortunately, I can't run tmt's tests because:

  1. there's no tmt package for Ubuntu, and
  2. there's no beakerlib package for Ubuntu :)

What about making an implementation for homebrew? Packaging there should be relatively fast.

@lukaszachy
Copy link
Copy Markdown
Contributor

lukaszachy commented Feb 9, 2024

What about making an implementation for homebrew? Packaging there should be relatively fast.

Could you help with that? Not many macos users in the tmt devel audience.

@LecrisUT
Copy link
Copy Markdown
Member

LecrisUT commented Feb 9, 2024

What about making an implementation for homebrew? Packaging there should be relatively fast.

Could you help with that? Not many macos users in the tmt devel audience.

Yep, I can help there. Granted I don't have macs anymore, but I still maintain some homebrew packages now and then

@happz happz marked this pull request as ready for review February 14, 2024 12:58
martinhoyer
martinhoyer reviewed Feb 14, 2024
View reviewed changes
@happz happz force-pushed the package-managers branch 3 times, most recently from b3ec95b to 29f7341 Compare February 17, 2024 11:39
@happz happz mentioned this pull request Feb 19, 2024
Merged
2 tasks
@happz
Copy link
Copy Markdown
Contributor Author

happz commented Mar 16, 2024

/packit build

@happz happz force-pushed the package-managers branch from bf380ea to 5248903 Compare March 18, 2024 20:27
psss
psss requested changes Mar 19, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@psss psss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Tests are green \o/ Just two comments.

@psss
Copy link
Copy Markdown
Member

psss commented Mar 19, 2024

The only failing test is /tests/prepare/multihost which looks like the race condition issue again:

I've run the test manually and it's green, thus ready to merge.

@psss psss merged commit fb9c7df into main Mar 19, 2024
@psss psss deleted the package-managers branch March 19, 2024 14:35
@psss psss self-assigned this Mar 19, 2024
@psss psss mentioned this pull request Mar 25, 2024
Closed
@LecrisUT LecrisUT mentioned this pull request Apr 2, 2024
Open
@psss psss linked an issue Apr 8, 2025 that may be closed by this pull request
Add support for non-rpm based distros #1951
Closed
@psss psss mentioned this pull request Apr 8, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@LecrisUT LecrisUT LecrisUT approved these changes

@lukaszachy lukaszachy lukaszachy approved these changes

@psss psss psss approved these changes

@thrix thrix Awaiting requested review from thrix

@janhavlin janhavlin Awaiting requested review from janhavlin

@martinhoyer martinhoyer Awaiting requested review from martinhoyer

Assignees

@psss psss

Labels

ci | full test Pull request is ready for the full test execution status | blocking other work An important pull request, blocking other pull requests or issues step | prepare Stuff related to the prepare step

Projects

None yet

Milestone

1.32

Development

Successfully merging this pull request may close these issues.

Add support for non-rpm based distros

5 participants

@happz @LecrisUT @lukaszachy @psss @martinhoyer