IPS:

It Sucks Less Than You Think

Eric Sproul
OmniTI

Wednesday, December 18, 13 1

 What Is IPS?
Image Packaging System, aka "pkg(5)"
Created by Sun for OpenSolaris
Now used by OmniOS, OpenIndiana, Oracle Solaris 11
Transactional, metadata-driven and integrated with ZFS
Network-based, extensive search grammar
Changes-only updates

Wednesday, December 18, 13 2

 Motivations
Unify packaging and OS patching
Be smf(5)- and ZFS-aware
Verify correct installation
Optimize for the update case
Ease developer burden
Add dependency-based network retrieval

Wednesday, December 18, 13 3

Unify: patches could touch multiple packages; express all updates as the same type of op.
Developer burden: auto-gen. deps, eliminate build system, enforce guidelines in tools
 IPS: The Good

Every package 100% described by metadata

Updating requires fetching only changes
Get a new BE automatically, when needed
Automatic fetching of dependencies

Wednesday, December 18, 13 4

Metadata: enables verification that current state matches intent

baked into the package itself; repo-wide catalog only used for better performance
 IPS: The Not-So-Good

No single-file on-disk format (except archives)

Latency-sensitive
No pre- or post-install scripting*

* This is actually a good thing! Tasks usually scripted are now first-class actions

Wednesday, December 18, 13 5

Scripting: opaque, open-ended, unverifiable; e.g. packages may be in shared contexts

(virtualization)
 A Few IPS Commands

pkg(1) :: installation and information client

pkgsend(1) :: publication client
pkgrecv(1) :: raw contents retrieval utility
pkg.depotd(1M) :: repository server

Wednesday, December 18, 13 6

These are the most common. There are many more.

 IPS Concepts
FMRI :: Fault Management Resource Identifier
Manifest :: describes a specific version of a package
Publisher :: entity that provides one or more packages
Repository :: location for publishing and retrieving pkgs
Image :: location where packages may be installed
Boot Environment :: (BE) bootable instance of an image

Wednesday, December 18, 13 7

FMRI: other things that have them: SMF services, fmd/fmadm

Image: typically you have one, at /. OmniOS zones each have one.
 FMRIs in IPS
pkg://omnios/web/[email protected],5.11-0.151006:20130703T175442Z

pkg Scheme
omnios Publisher
web Category
curl Package Name
7.31.0,5.11-0.151006:20130703T175442Z Version

Wednesday, December 18, 13 8

category/name is arbitrarily deep; name is basename

 FMRIs in IPS
Publisher name is optional:
pkg://omnios/web/curl Must be preceded by 'pkg://' if present

pkg:/web/curl Note the use of only one '/' after the scheme

Scheme is also optional:

/web/curl Leading '/' anchors the name at publisher root

web/curl Anything ending in '/web/curl'

curl Anything named 'curl' or ending in '/curl'

Wednesday, December 18, 13 9

 Versions in IPS
Strictly numeric comparison
Comparison is left to right

7.31.0,5.11-0.151006:20130703T175442Z

7.31.0 Component Version ("upstream version")

5.11 Build Version (OS version, `uname -r`)

0.151006 Branch Version (vendor-specific version)

20130703T175442Z Timestamp (ISO 8601)

Wednesday, December 18, 13 10

 Versions in IPS
pkg://omnios/web/[email protected],5.11-0.151006:20130703T175442Z

But, that's hard to read!?

Version strings are for machines, not people!

Rarely do you need to worry

about anything but the
component version

Wednesday, December 18, 13 11

May look ugly, but version strings are for machines, not people!
 Package Manifest

Describes a specific version of a package

Collection of actions that deliver files, dirs, links,
dependencies, etc. via attributes
Attributes are key-value pairs
Viewable with `pkg contents -m <name>`

Wednesday, December 18, 13 12

 Package Manifest
set name=pkg.fmri value=pkg://omnios/web/[email protected],5.11-0.151006:20130703T175442Z
set name=pkg.summary value="curl - command line tool for transferring data with URL syntax"
set name=pkg.descr value="curl - command line tool for transferring data with URL syntax"
set name=publisher [email protected]
dir group=bin mode=0755 owner=root path=usr/bin/amd64
file 3a8938b01cf732fc0b4838218d94508fca75e54c
chash=d923dfc752598ed149a64c873065fc71cbbf83fb
elfarch=i386 elfbits=64 elfhash=aabff399422fb0e74df8ffb4356d7bee97db89a5
group=bin mode=0755 owner=root
path=usr/bin/amd64/curl
pkg.csize=100864 pkg.size=174672
...
link path=usr/lib/amd64/libcurl.so target=libcurl.so.4.3.0
...
depend fmri=library/security/[email protected] type=require
depend fmri=library/zlib type=require
depend fmri=web/ca-bundle type=require

Wednesday, December 18, 13 13

set action is pkg-level metadata; extensible to arbitrary k/v pairs

file action has positional parameter first, sha1 cksum of original file
elfhash: "interesting" sections of ELF header-- ones mapped into memory, affecting
executable behavior (.text, .data, etc.)
 Dependencies
Require :: provides essential functionality; including a
version sets a "floor"
Optional :: non-essential, but if installed, must meet
version constraint, if any (same as require)
Exclude :: conflicts; may not be installed with this
package (these are evil, avoid them)
Incorporate :: like optional, but sets "ceiling" as well as
"floor" to the given degree of precision

Wednesday, December 18, 13 14

There are a few other esoteric types

 Dependencies
# any version of foo
library/foo
require
# foo >= 2
optional library/foo@2
exclude
# foo >= 2.1
library/[email protected]

Wednesday, December 18, 13 15

 Dependencies
# foo 2.x, not 1.9 or 3.x
library/foo@2

# foo 2.1.x, not 2.0 or 2.2

incorporate library/[email protected]

# foo 2.1.2 only

library/[email protected]

Wednesday, December 18, 13 16

 Dependencies
Packages containing only incorporate
dependencies are called "incorporations"

Used to ensure a compatible set

of installed software

Used carefully, they can be very handy:

omniti/incorporation/perl-516-incorporation

Wednesday, December 18, 13 17

 Dependencies
$ pkg contents -mr perl-516-incorporation
set name=pkg.fmri
value=pkg://perl.omniti.com/omniti/incorporation/[email protected],5.11-0.151002:20120725T211803Z
set name=pkg.summary value="Constrains omniti/runtime/perl to version 5.16.x"
set name=pkg.descr value="Constrains omniti/runtime/perl to version 5.16.x"
set name=pkg.human-version value=5.16
set name=publisher [email protected]
depend fmri=omniti/runtime/[email protected] type=incorporate

Version of omniti/runtime/perl must be 5.16.x

Module dist pkgs have their own versions, but require
the incorporation matching the perl they were built with

Wednesday, December 18, 13 18

 Publisher

An entity that provides packages

Named for products ("omnios")
or domain style ("ms.omniti.com")
One publisher can have multiple URLs
List current publishers: `pkg publisher`
Configure publishers: `pkg set-publisher ...`

Wednesday, December 18, 13 19

 Repository

Location to which packages are published

Can be used locally (file://)
or remotely (http://) via pkg.depotd(1M)
Created and managed by pkgrepo(1)

Wednesday, December 18, 13 20

 Image

Location where packages can be installed

May be rooted at arbitrary points in the filesystem tree
Default image rooted at '/'
Have properties that govern policy; see pkg(1)

Wednesday, December 18, 13 21

 Boot Environment

Bootable instance of an image

Can be auto-created according to image policy
Can be manually created
Created and managed by beadm(1M)

Wednesday, December 18, 13 22

 Use Cases
Install
Update
List/Info
Inventory
Search
Audit

Wednesday, December 18, 13 23

 Use Cases: Install
# dry run, verbose
pkg install -nv foo
# latest
When "foo" pkg install foo
is not installed # latest available 2.x
pkg install foo@2
# exact version
pkg install [email protected]

Wednesday, December 18, 13 24

 Use Cases: Update
# dry run, verbose
pkg update -nv foo
# latest available
Assuming pkg update foo
"foo 2.1"
is installed # stay within 2.x line
pkg update foo@2
# downgrade
pkg update [email protected]

Wednesday, December 18, 13 25

 Use Cases: List/Info
# all installed packages
pkg list
# list packages matching "foo"
pkg list foo
# detailed information
pkg info foo
# same, but remote
pkg info -r foo

Wednesday, December 18, 13 26

 Use Cases: Inventory
# file/directory paths only
pkg contents foo

# raw manifest
pkg contents -m foo

# list deps
pkg contents -t depend -o fmri

Wednesday, December 18, 13 27

unless specified with -o, default output is the path attribute

 Use Cases: Search

Powerful due to package metadata

Local or remote
Expressive grammar
Results sometimes non-obvious

Wednesday, December 18, 13 28

non-obvious, until you understand what is being searched

 Use Cases: Search
# 'tmux' as any value
pkg search tmux

INDEX ACTION VALUE PACKAGE

basename file usr/bin/tmux pkg:/terminal/[email protected]
basename file usr/bin/tmux pkg:/terminal/[email protected]
basename file usr/bin/tmux pkg:/terminal/[email protected]
pkg.fmri set omnios/terminal/tmux pkg:/terminal/[email protected]
pkg.fmri set omnios/terminal/tmux pkg:/terminal/[email protected]
pkg.fmri set omnios/terminal/tmux pkg:/terminal/[email protected]

Wednesday, December 18, 13 29

what we're searching is actions

 Use Cases: Search
# same as before, but show only pkg name
pkg search -p tmux

PACKAGE PUBLISHER
pkg:/terminal/[email protected] omnios
pkg:/terminal/[email protected] omnios
pkg:/terminal/[email protected] omnios

Wednesday, December 18, 13 30

 Use Cases: Search
pkg_name:action_type:key:token

pkg_name :: the value of pkg.fmri

action_type :: file, dir, link, depend, set, etc.
key :: attribute name within the selected action
token :: attribute value, i.e., "what you're searching for"

Wednesday, December 18, 13 31

 Use Cases: Search
pkg_name:action_type:key:token

Blank fields implicitly wild-carded

Leading colons optional
`pkg search tmux` is effectively: `pkg search ':::tmux'`

Wednesday, December 18, 13 32

any package name, any action type, any attribute whose value is the string 'tmux'
 Use Cases: Search
This answer:
$ pkg search 'dir::pgsql*'
INDEX ACTION VALUE PACKAGE
...
basename dir opt/pgsql925 pkg:/omniti/database/postgresql-925/[email protected]
...

results from this manifest entry:

dir group=bin mode=0755 owner=root path=opt/pgsql925

Wednesday, December 18, 13 33

basename is a pseudo-attribute that matches within path

 Use Cases: Search
Packages that deliver perl .so files
$ pkg search -o pkg.name 'file:path:*perl*.so'
PKG.NAME
omniti/perl/db_file
omniti/perl/b-callchecker
omniti/perl/bsd-resource
omniti/perl/clone
...

Wednesday, December 18, 13 34

Simple globbing in token field

 Use Cases: Search
Reverse dependencies
$ pkg search -H -o pkg.name 'depend::web/curl'
developer/versioning/git
developer/versioning/mercurial
entire
incorporation/jeos/omnios-userland

What r151006 packages incorporate on web/curl,

and at what version?
$ pkg search -o pkg.fmri,fmri '*-0.151006:depend:incorporate:web/curl'
PKG.FMRI FMRI
pkg:/incorporation/jeos/omnios-userland@11,5.11-0.151006:20130506T214442Z web/curl@7,5.11-0.151006
pkg:/incorporation/jeos/omnios-userland@11,5.11-0.151006:20130716T202721Z web/curl@7,5.11-0.151006
pkg:/incorporation/jeos/omnios-userland@11,5.11-0.151006:20131030T205312Z web/curl@7,5.11-0.151006

Wednesday, December 18, 13 35

-H eliminates the header line

 Use Cases: Audit
# check installed state of all pkgs
pkg verify

# check state of a single package

pkg verify <pkg>

# repair installed state of curl

pkg fix <pkg>

Wednesday, December 18, 13 36

 Use Cases: Audit
# pkg verify -v curl
PACKAGE STATUS
pkg://omnios/web/curl OK

# rm /usr/share/man/man3/libcurl.3

# pkg verify -v curl

PACKAGE STATUS
pkg://omnios/web/curl ERROR
file: usr/share/man/man3/libcurl.3
Missing: regular file does not exist

Wednesday, December 18, 13 37

Using -v to show output; normally no output unless there's a problem

 Use Cases: Audit
# pkg fix curl
Verifying: pkg://omnios/web/curl ERROR
file: usr/share/man/man3/libcurl.3
Missing: regular file does not exist
Created ZFS snapshot: 2013-10-16-02:07:42
Repairing: pkg://omnios/web/curl

DOWNLOAD PKGS FILES XFER (MB)

Completed 1/1 1/1 0.0/0.0

PHASE ACTIONS
Update Phase 1/1

PHASE ITEMS
Image State Update Phase 2/2

Wednesday, December 18, 13 38

 Creating IPS Packages
IPS does not impose a build framework
(think rpmbuild, debuild)

Build software however you wish

Place build product in a proto area
Create manifest
Publish to a repo

Wednesday, December 18, 13 39

proto area: destination directory with a mockup of the final layout

 Creating IPS Packages
pkgsend(1) both creates manifests and
publishes packages
pkgsend generate /path/to/proto > /tmp/manifest.p5m
add FMRI, any other 'set' actions to manifest
pkgsend publish -s <repo_url> -d /path/to/proto \
/tmp/manifest.p5m

Wednesday, December 18, 13 40

 Creating IPS Packages

Adding the 'set' stuff is tedious

May want to make other changes/additions to manifest
This needs to be automated!
Use pkgmogrify(1)

Wednesday, December 18, 13 41

 Creating IPS Packages
pkgmogrify(1)
Programmatic transformations of manifest contents
Macro replacements
Include other manifests or manifest fragments
Transformation of actions
By convention, we store these directives in a .mog file
beside our build scripts

Wednesday, December 18, 13 42

 Creating IPS Packages
pkgmogrify: Add actions
group
gid=90
groupname=postgres
user
ftpuser=false
gcos-field="PostgreSQL Reserved UID"
group=postgres
login-shell=/usr/bin/pfksh
password=NP
uid=90
username=postgres
home-dir=/home/postgres
license
COPYING
license=GPLv2

Wednesday, December 18, 13 43

license can also cause the license to be displayed and/or require acceptance
 Creating IPS Packages
pkgmogrify: Transform actions
<transform dir path=opt/riak/data.* -> set owner riak>

<transform dir path=opt/riak/data.* -> set group riak>

<transform file path=opt/riak/etc/.*\.args -> set mode 0644>

<transform file path=opt/apache22/libexec/amd64/libphp5.so ->

edit path libphp5.so libphp5.53.so>

<transform file path=opt/elasticsearch/config/elasticsearch.yml ->

set preserve true>

<transform file path=opt/omni/lib/ruby/gems/1.9/cache.* -> drop>

<transform file path=(var|lib)/svc/manifest/.*\.xml ->

add restart_fmri svc:/system/manifest-import:default>

Wednesday, December 18, 13 44

Default ownership is root:bin for all files, dirs

Last one is from the global transforms in our build system
 Creating IPS Packages
Tangent: renaming
pkg:/network/iftop
pkg:/omniti/network/iftop

Forgot to follow naming convention

Users may have installed it, can't just abandon it

Wednesday, December 18, 13 45

 Creating IPS Packages
Tangent: renaming

Solution: publish a "rename package"

Transitional package that allows update to new name
set name=pkg.fmri
value=pkg://ms.omniti.com/network/[email protected],5.11-0.151006:20130816T191418Z
set name=pkg.renamed value=true
set name=variant.opensolaris.zone value=global value=nonglobal
depend fmri=pkg://ms.omniti.com/omniti/network/iftop type=require

Wednesday, December 18, 13 46

pkg removes the old package provided nothing else requires it

 Creating IPS Packages
Create a repo with pkgrepo(1)

# zfs create data/myrepo

# pkgrepo create /data/myrepo
# pkgrepo set -s /data/myrepo publisher/prefix=myrepo.example.com

May now use file:///data/myrepo to publish packages

publisher/prefix sets the default publisher name

Wednesday, December 18, 13 47

In theory, a repo can house pkgs for multiple publishers

In practice, we don't do it-- it's too confusing
 Creating IPS Packages
Create an archive with pkgrecv(1)
$ pkgrecv -s http://pkg.omniti.com/omnios/release/ -d web_curl.p5a -a web/curl
Retrieving packages for publisher omnios ...
Retrieving and evaluating 1 package(s)...
DOWNLOAD PKGS FILES XFER (MB)
Completed 1/1 88/88 1.3/1.3

ARCHIVE FILES STORE (MB)

web_curl.p5a 158/158 1.5/1.5

$ scp web_curl.p5a me@my-other-box:

# pkg install -g web_curl.p5a web/curl

Wednesday, December 18, 13 48

archives can contain multiple packages

watch out for dependency issues-- same rules apply on destination system
 Questions?
Further reading

Man pages: pkg(5), pkg(1), pkgsend(1), pkgrecv(1),

pkgmogrify(1), pkgrepo(1)
http://omnios.omniti.com/wiki.php/
GeneralAdministration#PackageManagement
http://omnios.omniti.com/media/ipsdevguide.pdf
http://web.archive.org/web/20100105071515/http://
blogs.sun.com/sch/entry/pkg_1_a_no_scripting

Wednesday, December 18, 13 49