Skip to content

The proxy snapshotter does not send the namespace in Context #6061

@Kern--

Description

@Kern--

Description

When an out-of-process snapshotter tires to retrieve the containerd namespace from context (using namespaces.Namespace or namespace.NamespaceRequired) the namespace is always empty.

Looking at how namspaces.Namespace works, it first tries to pull the namespace directly out of the context, and then if that fails, it checks grpc and ttrpc headers. When the client sets a namespace, the gprc and ttrpc headers are set so that it can be read inside containerd after transiting grpc/ttrcp.

The issue is that grpc has separate incoming and outgoing metadata, so when the client sets outgoing metadata, containerd receives it as incoming metadata. When the proxy snapshotter in containerd then forwards to the out-of-process snapshotter, the outgoing metadata is empty and the out-of-process snapshotter cannot read the namespace.

I believe this is a bug because in-process snapshotters do have access to the namespace. The incoming grpc/ttrpc headers will be set and so the fallback logic will find the namespace. It just happens that none of the built-in snapshotters use the namespace.

Steps to reproduce the issue

  1. Create a stub out-of-process snapshotter that logs the namespace
    relevant code:
func (s *UnimplProxySnapshotter) Prepare(ctx context.Context, key, parent string, opts ...snapshots.Opt) ([]mount.Mount, error) {
	namespace, err := namespaces.NamespaceRequired(ctx)
	if err != nil {
		fmt.Printf("Err getting namespace: %s\n", err)
	} else {
		fmt.Printf("Namespace: %s\n", namespace)
	}
	return []mount.Mount{}, nil
}
Full example
package main

import (
	"context"
	"fmt"
	"net"
	"os"
	"os/signal"
	"syscall"

	snapshotsapi "github.com/containerd/containerd/api/services/snapshots/v1"
	"github.com/containerd/containerd/contrib/snapshotservice"
	"github.com/containerd/containerd/mount"
	"github.com/containerd/containerd/namespaces"
	"github.com/containerd/containerd/snapshots"
	"golang.org/x/sync/errgroup"
	"google.golang.org/grpc"
)

const listenAddr = "./snapshotter.sock"

func main() {
	stop := make(chan os.Signal, 1)
	signal.Notify(stop, syscall.SIGINT, syscall.SIGTERM, syscall.SIGPIPE, syscall.SIGHUP, syscall.SIGQUIT)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	group, ctx := errgroup.WithContext(ctx)

	rpc := grpc.NewServer()

	snap := UnimplProxySnapshotter{}

	service := snapshotservice.FromSnapshotter(&snap)
	snapshotsapi.RegisterSnapshotsServer(rpc, service)

	listener, err := net.Listen("unix", listenAddr)
	if err != nil {
		fmt.Printf("failed to listen socket at %s: %s\n", listenAddr, err)
		return
	}

	group.Go(func() error {
		return rpc.Serve(listener)
	})

	group.Go(func() error {
		defer func() {
			rpc.Stop()
		}()

		for {
			select {
			case <-stop:
				cancel()
				return nil
			case <-ctx.Done():
				return ctx.Err()
			}
		}
	})

	if err := group.Wait(); err != nil {
		fmt.Printf("error running snapshotter: %s\n", err)
	}
}

type UnimplProxySnapshotter struct {
}

func (s *UnimplProxySnapshotter) Stat(ctx context.Context, key string) (snapshots.Info, error) {
	return snapshots.Info{}, nil
}

func (s *UnimplProxySnapshotter) Update(ctx context.Context, info snapshots.Info, fieldpaths ...string) (snapshots.Info, error) {
	return snapshots.Info{}, nil
}

func (s *UnimplProxySnapshotter) Usage(ctx context.Context, key string) (snapshots.Usage, error) {
	return snapshots.Usage{}, nil
}

func (s *UnimplProxySnapshotter) Mounts(ctx context.Context, key string) ([]mount.Mount, error) {
	return []mount.Mount{}, nil
}

func (s *UnimplProxySnapshotter) Prepare(ctx context.Context, key, parent string, opts ...snapshots.Opt) ([]mount.Mount, error) {
	namespace, err := namespaces.NamespaceRequired(ctx)
	if err != nil {
		fmt.Printf("Err getting namespace: %s\n", err)
	} else {
		fmt.Printf("Namespace: %s\n", namespace)
	}
	return []mount.Mount{}, nil
}

func (s *UnimplProxySnapshotter) View(ctx context.Context, key, parent string, opts ...snapshots.Opt) ([]mount.Mount, error) {
	return []mount.Mount{}, nil
}

func (s *UnimplProxySnapshotter) Commit(ctx context.Context, name, key string, opts ...snapshots.Opt) error {
	return nil
}

func (s *UnimplProxySnapshotter) Remove(ctx context.Context, key string) error {
	return nil
}

func (s *UnimplProxySnapshotter) Walk(ctx context.Context, fn snapshots.WalkFunc, filters ...string) error {
	return nil
}

func (s *UnimplProxySnapshotter) Close() error {
	return nil
}
  1. Register the proxy snapshotter in containerd config
[proxy_plugins]
  [proxy_plugins.customsnapshot]
    type = "snapshot"
    address = "/home/ec2-user/snapshotter/snapshotter.sock"
  1. Attempt to pull an image using the snapshotter with a custom namespace
sudo ctr --namespace test i pull docker.io/library/busybox:latest --snapshotter customsnapshot
  1. Observe that the snapshotter does not receive the namespace:
$ ./main
Err getting namespace: namespace is required: failed precondition

Describe the results you received and expected

I received no namespace in the out-of-process snapshotter. I expected to be able to read the namespace.

What version of containerd are you using?

containerd github.com/containerd/containerd v1.5.0-508-gd0bedc5bd.m d0bedc5.m

Any other relevant information

N/A

Show configuration if it is related to CRI plugin.

N/A

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions