OpenAPI support for kubectl #44531

pwittrock · 2017-04-15T22:30:14Z

Support for openapi spec in kubectl.

Includes:

downloading and caching openapi spec to a local file
parsing openapi spec into binary serializable datastructures (10x faster load times 600ms -> 40ms)
caching parsed openapi spec in memory for each command

NONE

k8s-reviewable · 2017-04-15T22:31:20Z

This change is

pwittrock · 2017-04-17T16:49:57Z

@k8s-bot verify test this

W0416 18:58:18.522] fatal: unable to access 'https://bitbucket.org/bertimus9/systemstat.git/': The requested URL returned error: 502
W0416 18:58:18.522] godep: error downloading dep (bitbucket.org/bertimus9/systemstat): exit status 128
W0416 18:58:24.761] godep: Error downloading some deps. Aborting restore and check.
W0416 18:58:24.765] !!! Error in ./hack/godep-restore.sh:28

pwittrock · 2017-04-17T18:05:24Z

@caesarxuchao - plz review the client-go changes
@mbohlool - plz review the openapi parsing
@fabianofranz / @monopole - plz review for kubectl

mbohlool · 2017-04-17T18:38:22Z

pkg/kubectl/cmd/util/helpers.go

@@ -401,6 +401,11 @@ func AddValidateFlags(cmd *cobra.Command) {
 	cmd.MarkFlagFilename("schema-cache-dir")
 }

+func AddOpenAPIFlags(cmd *cobra.Command) {
+	cmd.Flags().String("schema-cache-dir", fmt.Sprintf("~/%s/%s", clientcmd.RecommendedHomeDir, clientcmd.RecommendedSchemaName), fmt.Sprintf("If non-empty, load/store cached API schemas in this directory, default is '$HOME/%s/%s'", clientcmd.RecommendedHomeDir, clientcmd.RecommendedSchemaName))


I've seen three reference to schema, openapi, and swagger in code/comments/messages. should we just use one? I suggest openapi for all of them.

This doesn't introduce a new flag, it is just allows registering the cache dir without registering the validation flags. See 5 lines above this one.

mbohlool · 2017-04-17T18:50:02Z

pkg/kubectl/cmd/util/openapi/openapi.go

+
+// isPrimitive returns true if s is a primitive type
+func (o *Resources) isPrimitive(s spec.Schema) bool {
+	return !o.isArray(s) && !o.isMap(s) && !o.isDefinitionReference(s) && len(s.Type) > 0


The assumption of a schema is primitive if it does not have a ref may not be true all the time. I would say a primitive schema is one with no "properties". does that make sense?

Updated the checks to be more explicit.

There is a unit check that test that every definition encountered has exactly 1 of (isPrimitive, isArray, isMap, isDefinitionReference) which should give some confidence that these are working as intended.

mbohlool · 2017-04-17T18:57:51Z

pkg/kubectl/cmd/util/openapi/openapi_cache.go

+}
+
+// encodeSpec binary encodes the openapi spec
+func (c *cachingOpenAPIClient) encodeSpec(parsed *Resources) ([]byte, error) {


Can't we just encode/decode spec.swagger?

No. :( As discussed offline, the Ref type which contains the json reference has private fields that are dropped by binary serialization.

monopole · 2017-04-17T20:21:01Z

Reviewed 3 of 3 files at r1, 19 of 19 files at r2, 6 of 6 files at r3.
Review status: all files reviewed at latest revision, 20 unresolved discussions.

pkg/kubectl/cmd/testing/fake.go, line 406 at r3 (raw file):

func (f *FakeFactory) SwaggerSchema(schema.GroupVersionKind) (*swagger.ApiDeclaration, error) {
	return nil, nil
}

should the swagger stuff be marked deprecated and/or be given a TODO(#issue) to track its removal?

pkg/kubectl/cmd/util/factory.go, line 445 at r3 (raw file):

	cacheFile := path.Join(fullDir, prefix, groupVersion, schemaFileName)

	abs, _ := filepath.Abs(cacheDir)

this can be scoped to the 'if' body

pkg/kubectl/cmd/util/factory_object_mapping.go, line 442 at r3 (raw file):

// openAPIData returns an openAPIData containing metadata and structural information about Kubernetes
// models and operations.  Will try to cache the spec locally if possible.

pls add note about what (if anything) will make the cache stale, and how it is freshened

test coverage for that?

pkg/kubectl/cmd/util/helpers.go, line 405 at r3 (raw file):

func AddOpenAPIFlags(cmd *cobra.Command) {
	cmd.Flags().String("schema-cache-dir", fmt.Sprintf("~/%s/%s", clientcmd.RecommendedHomeDir, clientcmd.RecommendedSchemaName), fmt.Sprintf("If non-empty, load/store cached API schemas in this directory, default is '$HOME/%s/%s'", clientcmd.RecommendedHomeDir, clientcmd.RecommendedSchemaName))

BTW , go fmt will retain newlines added for readability.

pkg/kubectl/cmd/util/openapi/openapi.go, line 30 at r3 (raw file):

)

const gvkOpenAPIKey = "x-kubernetes-group-version-kind"

Please comment the const. Does the x mean something? could this string be arbitrary, e.g. peach-banana-sandwich?

pkg/kubectl/cmd/util/openapi/openapi.go, line 40 at r3 (raw file):

}

// KindDefinition defines a Kubernetes object Kind

Seems odd to have PeachDefinition instead of just Peach. The doc doesn't help.
Maybe for doc inspiration see https://docs.oracle.com/javase/7/docs/api/java/lang/Class.html

Went digging in apimachinery/blob/master/pkg/runtime/schema/group_version.go and am even more confused about Kind

pkg/kubectl/cmd/util/openapi/openapi.go, line 42 at r3 (raw file):

// KindDefinition defines a Kubernetes object Kind
type KindDefinition struct {
	// Name is the name of the type

BTW, this looks like L65, hence earlier request for commentary on Kind.
"TypeDefinition" below raises the same questions

pkg/kubectl/cmd/util/openapi/openapi.go, line 48 at r3 (raw file):

	IsResource bool

	// GroupVersionKind is the group version kind of a resource.

// GroupVersionKind (does this useful thing in the context) of a resource.

pkg/kubectl/cmd/util/openapi/openapi.go, line 63 at r3 (raw file):

}

// TypeDefinition defines a field definition

"defines a Type"

and again wondering why not just "Type" and how this differs from "Kind".

pkg/kubectl/cmd/util/openapi/openapi.go, line 193 at r3 (raw file):

// isArray returns true if s is an array type.
func (o *Resources) isArray(s spec.Schema) bool {
	return len(s.Type) > 0 && s.Type[0] == "array"

BTW: Could "ARRAY" show up?

pkg/kubectl/cmd/util/openapi/openapi.go, line 196 at r3 (raw file):

}

// isMap

isObject? (given s.Type[0] == "object")

pkg/kubectl/cmd/util/openapi/openapi.go, line 232 at r3 (raw file):

}

// getElementType returns the type of an element for arrays and maps

Comment looks odd given !o.isArray returns error

pkg/kubectl/cmd/util/openapi/openapi.go, line 259 at r3 (raw file):

// getGroupVersionKind implements openAPIData
// getGVK parses the gropuversionkind for a resource definition from the x-kubernetes

sic (gropuversionkind)

More and more want a primer in this doc about Group, Kind, Version, Type, Field
(or a ptr to a primer).

Doc under https://github.com/kubernetes/apimachinery/blob/master/pkg/runtime/schema/group_version.go doesn't help
as much as i'd hoped.

pkg/kubectl/cmd/util/openapi/openapi_cache.go, line 48 at r3 (raw file):

// newCachingOpenAPIClient returns a new discovery.OpenAPISchemaInterface
// that will read the openapi spec from a local cache if it exists, and
// if not fetch an openapi spec using client

using client.

People are not picky about sentence formation?

pkg/kubectl/cmd/util/openapi/openapi_cache_test.go, line 60 at r3 (raw file):

		result, err := instance.openAPIData()
		Expect(err).To(BeNil())
		expectEqual(result, expectedData)

FYI: make an Equals method for Resources, test it in openapi_test.go, and use it directly here? instead of the expectEqual utility below?

pkg/kubectl/cmd/util/openapi/openapi_test.go, line 67 at r3 (raw file):

	})

	It("should find the definition fields", func() {

BTW: nifty framework 👍 tests easy to read

staging/src/k8s.io/client-go/discovery/discovery_client.go, line 380 at r3 (raw file):

	// WARNING: We don't use the loads.Analyze because this is very expensive
	// and loads.Analyze takes 2x as long (it Unmarshal's 2x)
	// Warning: this is very expensive (~600ms)

comment above function?
loads.Analyzed called below, seems to contradict comment (though technically it differs by one letter)
Warning: twice?

Comments from Reviewable

arschles · 2017-04-17T21:00:32Z

pkg/kubectl/cmd/util/openapi/doc.go

+package openapi
+
+// The openapi package is a collection of libraries for fetching the openapi spec
+// from a Kubernetes server and then indexing the type definitions.


if these are meant to be Godoc-compatible comments, they need to go above the package keyword

pwittrock · 2017-04-18T00:42:58Z

Review status: 20 of 28 files reviewed at latest revision, 21 unresolved discussions.

pkg/kubectl/cmd/testing/fake.go, line 406 at r3 (raw file):

Previously, monopole (Jeff Regan) wrote…

should the swagger stuff be marked deprecated and/or be given a TODO(#issue) to track its removal?

Added

pkg/kubectl/cmd/util/factory.go, line 445 at r3 (raw file):

Previously, monopole (Jeff Regan) wrote…

this can be scoped to the 'if' body

Done

pkg/kubectl/cmd/util/factory_object_mapping.go, line 442 at r3 (raw file):

Previously, monopole (Jeff Regan) wrote…

pls add note about what (if anything) will make the cache stale, and how it is freshened

test coverage for that?

Updated with a comment.

Test coverage for the caching logic is in the openapi_cache_test.go file

pkg/kubectl/cmd/util/helpers.go, line 405 at r3 (raw file):

Previously, monopole (Jeff Regan) wrote…

BTW , go fmt will retain newlines added for readability.

Added some newlines for readability

pkg/kubectl/cmd/util/openapi/doc.go, line 20 at r3 (raw file):

Previously, arschles (Aaron Schlesinger) wrote…

if these are meant to be Godoc-compatible comments, they need to go above the package keyword

Done thx.

pkg/kubectl/cmd/util/openapi/openapi.go, line 30 at r3 (raw file):

Previously, monopole (Jeff Regan) wrote…

Please comment the const. Does the x mean something? could this string be arbitrary, e.g. peach-banana-sandwich?

Done

pkg/kubectl/cmd/util/openapi/openapi.go, line 40 at r3 (raw file):

Previously, monopole (Jeff Regan) wrote…

Seems odd to have PeachDefinition instead of just Peach. The doc doesn't help.
Maybe for doc inspiration see https://docs.oracle.com/javase/7/docs/api/java/lang/Class.html

Went digging in apimachinery/blob/master/pkg/runtime/schema/group_version.go and am even more confused about Kind

Updated to drop the Definition piece. GVK deserves a good doc.

pkg/kubectl/cmd/util/openapi/openapi.go, line 42 at r3 (raw file):

Previously, monopole (Jeff Regan) wrote…

BTW, this looks like L65, hence earlier request for commentary on Kind.
"TypeDefinition" below raises the same questions

Yes it is a bit confusing. Added a comment to clarify

pkg/kubectl/cmd/util/openapi/openapi.go, line 48 at r3 (raw file):

Previously, monopole (Jeff Regan) wrote…

// GroupVersionKind (does this useful thing in the context) of a resource.

Updated the comment

pkg/kubectl/cmd/util/openapi/openapi.go, line 63 at r3 (raw file):

Previously, monopole (Jeff Regan) wrote…

"defines a Type"

and again wondering why not just "Type" and how this differs from "Kind".

Updated the comment. Let me know if this makes more sense.

pkg/kubectl/cmd/util/openapi/openapi.go, line 193 at r3 (raw file):

Previously, monopole (Jeff Regan) wrote…

BTW: Could "ARRAY" show up?

Changed to compare against lowercase. I don't think that this should actually show up, but provides a bit of future proofing.

I've changed the matching logic a bit so it should match one of the primitive types "string", "boolean", "integer" or "array", "object", or be empty for definition references. The unit test parses the k8s swagger.json and validates that each definition matches exactly one of primitive, array, map, definitionReference, so we should catch if there is anything that isn't looking correct.

pkg/kubectl/cmd/util/openapi/openapi.go, line 196 at r3 (raw file):

Previously, monopole (Jeff Regan) wrote…

isObject? (given s.Type[0] == "object")

Updated with a comment to explain this. Also moved the strings into constants with descriptions.

pkg/kubectl/cmd/util/openapi/openapi.go, line 232 at r3 (raw file):

Previously, monopole (Jeff Regan) wrote…

Comment looks odd given !o.isArray returns error

Done

pkg/kubectl/cmd/util/openapi/openapi.go, line 259 at r3 (raw file):

Previously, monopole (Jeff Regan) wrote…

sic (gropuversionkind)

More and more want a primer in this doc about Group, Kind, Version, Type, Field
(or a ptr to a primer).

Doc under https://github.com/kubernetes/apimachinery/blob/master/pkg/runtime/schema/group_version.go doesn't help
as much as i'd hoped.

GroupVersionKind is a more general API concept. There is a little bit here. Could probably deserve something more in depth.

pkg/kubectl/cmd/util/openapi/openapi_cache.go, line 48 at r3 (raw file):

Previously, monopole (Jeff Regan) wrote…

using client.

People are not picky about sentence formation?

Sometimes they are :)

pkg/kubectl/cmd/util/openapi/openapi_cache_test.go, line 60 at r3 (raw file):

Previously, monopole (Jeff Regan) wrote…

FYI: make an Equals method for Resources, test it in openapi_test.go, and use it directly here? instead of the expectEqual utility below?

Thx

staging/src/k8s.io/client-go/discovery/discovery_client.go, line 380 at r3 (raw file):

Previously, monopole (Jeff Regan) wrote…

comment above function?

loads.Analyzed called below, seems to contradict comment (though technically it differs by one letter)

Warning: twice?

:P. Changed the logic after I started caching. Updated the comments to be correct.

Comments from Reviewable

mengqiy

Other LGTM

mengqiy · 2017-04-20T22:50:37Z

pkg/kubectl/cmd/util/factory.go

@@ -33,6 +33,7 @@ import (
 	"github.com/spf13/cobra"
 	"github.com/spf13/pflag"

+	"github.com/golang/glog"


nit: This line should be moved into github import group

mengqiy · 2017-04-20T22:51:35Z

pkg/kubectl/cmd/util/factory_object_mapping.go

@@ -28,6 +28,8 @@ import (

 	"github.com/emicklei/go-restful/swagger"

+	"sync"


nit: should move this up.

caesarxuchao · 2017-04-21T01:47:11Z

staging/src/k8s.io/client-go/discovery/discovery_client.go

@@ -91,6 +94,11 @@ type SwaggerSchemaInterface interface {
 	SwaggerSchema(version schema.GroupVersion) (*swagger.ApiDeclaration, error)
 }

+type OpenAPISchemaInterface interface {


nit: for consistency, could you put a comment for the interface?

caesarxuchao · 2017-04-21T01:53:00Z

staging/src/k8s.io/client-go/discovery/discovery_client.go

+// OpenAPISchema fetches the open api schema using a rest client and parses the json.
+// Warning: this is very expensive (~1.2s)
+func (d *DiscoveryClient) OpenAPISchema() (*spec.Swagger, error) {
+	data, err := d.restClient.Get().AbsPath("/swagger.json").Do().Raw()


This comment is off-topic. /swagger.json is not shown as an path when i visit the root of the server. It only shows "/swaggerapi/" which leads me to swagger-1.2. @mbohlool do you think we should fix that?

caesarxuchao · 2017-04-21T01:54:45Z

staging/src/k8s.io/client-go/discovery/discovery_client_test.go

+}
+
+func TestGetOpenAPISchema(t *testing.T) {
+	expect := spec.Swagger{}


Can we add some contents to the spec?

caesarxuchao

A couple of nits for the changes in staging/client-go.

deads2k · 2017-04-21T14:54:50Z

pkg/kubectl/cmd/util/factory_object_mapping.go

+	openAPIGetter openAPIGetter
+}
+
+type openAPIGetter struct {


please avoid the anonymous inclusion in the struct. It makes the code below harder to read.

deads2k · 2017-04-21T14:55:56Z

pkg/kubectl/cmd/util/factory_object_mapping.go

+	})
+
+	// Delegate to the OpenAPIGetter
+	return f.openAPIGetter.OpenAPIData()


This method is on the Getter I guess?

deads2k · 2017-04-21T14:58:31Z

pkg/kubectl/cmd/util/factory_object_mapping.go

+		}
+
+		// Create the caching OpenAPIGetter
+		f.openAPIGetter.Getter = openapi.NewOpenAPIGetter(cacheDir, versionString, discovery)


Why does it care about the server version? Why doesn't it use the discovery method and rely on the caching of the discovery client?

Does the discovery client cache the result to disk?

Does the discovery client expose a method to that will give the client stable a hash / key for for exposed API?

monopole · 2017-04-21T16:24:44Z

Review status: 20 of 28 files reviewed at latest revision, 24 unresolved discussions.

pkg/kubectl/cmd/util/openapi/openapi_cache.go, line 48 at r3 (raw file):

Previously, pwittrock (Phillip Wittrock) wrote…

Sometimes they are :)

OK :) it all /lgtm

Comments from Reviewable

monopole · 2017-04-21T16:27:41Z

Reviewed 4 of 8 files at r4, 4 of 4 files at r5.
Review status: all files reviewed at latest revision, 23 unresolved discussions.

Comments from Reviewable

pwittrock · 2017-04-21T21:48:14Z

Review status: 22 of 28 files reviewed at latest revision, 23 unresolved discussions.

pkg/kubectl/cmd/util/factory.go, line 36 at r5 (raw file):

Previously, mengqiy (Mengqi Yu) wrote…

nit: This line should be moved into github import group

Done.

pkg/kubectl/cmd/util/factory_object_mapping.go, line 64 at r4 (raw file):

Previously, deads2k (David Eads) wrote…

please avoid the anonymous inclusion in the struct. It makes the code below harder to read.

Done.

pkg/kubectl/cmd/util/factory_object_mapping.go, line 477 at r4 (raw file):

Previously, deads2k (David Eads) wrote…

This method is on the Getter I guess?

Done.

pkg/kubectl/cmd/util/factory_object_mapping.go, line 31 at r5 (raw file):

Previously, mengqiy (Mengqi Yu) wrote…

nit: should move this up.

Done.

pkg/kubectl/cmd/util/openapi/openapi_cache.go, line 82 at r4 (raw file):

Previously, fabianofranz (Fabiano Franz) wrote…

Same here.

Done.

staging/src/k8s.io/client-go/discovery/discovery_client.go, line 97 at r5 (raw file):

Previously, caesarxuchao (Chao Xu) wrote…

nit: for consistency, could you put a comment for the interface?

Done.

staging/src/k8s.io/client-go/discovery/discovery_client.go, line 380 at r5 (raw file):

Previously, caesarxuchao (Chao Xu) wrote…

This comment is off-topic. /swagger.json is not shown as an path when i visit the root of the server. It only shows "/swaggerapi/" which leads me to swagger-1.2. @mbohlool do you think we should fix that?

ACK

staging/src/k8s.io/client-go/discovery/discovery_client_test.go, line 354 at r5 (raw file):

Previously, caesarxuchao (Chao Xu) wrote…

Can we add some contents to the spec?

Done.

Comments from Reviewable

pwittrock · 2017-04-21T21:48:43Z

PTAL

pwittrock · 2017-04-21T22:25:10Z

@k8s-bot gce etcd3 e2e test this

caesarxuchao · 2017-04-21T23:31:10Z

client-go changes lgtm.

pwittrock · 2017-04-25T14:32:01Z

@caesarxuchao @deads2k Can I get an approval for client-go?

deads2k · 2017-04-25T17:15:12Z

/approve

monopole · 2017-04-26T18:35:32Z

/lgtm

k8s-github-robot · 2017-04-26T23:10:17Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, monopole, pwittrock

Needs approval from an approver in each of these OWNERS Files:

~~hack/OWNERS~~ [deads2k]
~~pkg/kubectl/OWNERS~~ [deads2k,pwittrock]
~~staging/src/k8s.io/apiserver/OWNERS~~ [deads2k]
~~staging/src/k8s.io/client-go/OWNERS~~ [deads2k]
~~staging/src/k8s.io/kube-aggregator/OWNERS~~ [deads2k]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

k8s-github-robot · 2017-04-26T23:59:16Z

Automatic merge from submit-queue

pwittrock self-assigned this Apr 15, 2017

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Apr 15, 2017

k8s-github-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Apr 15, 2017

pwittrock force-pushed the kubectl-openapi branch from 3a00572 to 5a06596 Compare April 15, 2017 22:30

k8s-github-robot added the release-note-none Denotes a PR that doesn't merit a release note. label Apr 15, 2017

pwittrock force-pushed the kubectl-openapi branch from 5a06596 to 3df90a2 Compare April 15, 2017 22:32

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 16, 2017

pwittrock force-pushed the kubectl-openapi branch 2 times, most recently from 1c4ef67 to bf281e5 Compare April 16, 2017 18:55

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 16, 2017

pwittrock force-pushed the kubectl-openapi branch 4 times, most recently from f12943c to 20e7af1 Compare April 17, 2017 00:12

pwittrock force-pushed the kubectl-openapi branch 2 times, most recently from e758ec4 to 18d9b0b Compare April 17, 2017 18:03

pwittrock assigned fabianofranz, monopole, caesarxuchao and mbohlool and unassigned pwittrock Apr 17, 2017

mbohlool suggested changes Apr 17, 2017

View reviewed changes

arschles reviewed Apr 17, 2017

View reviewed changes

pwittrock force-pushed the kubectl-openapi branch from 18d9b0b to 4ec6ee0 Compare April 17, 2017 21:36

pwittrock requested a review from mengqiy April 17, 2017 22:16

mbohlool approved these changes Apr 20, 2017

View reviewed changes

mengqiy reviewed Apr 20, 2017

View reviewed changes

caesarxuchao reviewed Apr 21, 2017

View reviewed changes

deads2k reviewed Apr 21, 2017

View reviewed changes

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 25, 2017

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 25, 2017

pwittrock added 4 commits April 25, 2017 10:44

kubectl OpenAPI godeps updates

7070419

kubectl OpenAPI support

212c2a3

kubectl OpenAPI whitelist pkg/kubectl/cmd/util/openapi for linting

6cbe37b

kubectl OpenAPI bazel updates

21e239f

pwittrock force-pushed the kubectl-openapi branch from 5704b9e to 21e239f Compare April 25, 2017 17:46

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 25, 2017

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 26, 2017

k8s-github-robot merged commit 433aec1 into kubernetes:master Apr 26, 2017

		@@ -28,6 +28,8 @@ import (

		"github.com/emicklei/go-restful/swagger"

		"sync"

OpenAPI support for kubectl #44531

OpenAPI support for kubectl #44531

Conversation

pwittrock commented Apr 15, 2017

k8s-reviewable commented Apr 15, 2017

pwittrock commented Apr 17, 2017

pwittrock commented Apr 17, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

monopole commented Apr 17, 2017

Choose a reason for hiding this comment

pwittrock commented Apr 18, 2017

mengqiy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

caesarxuchao Apr 21, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

caesarxuchao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

monopole commented Apr 21, 2017

monopole commented Apr 21, 2017

pwittrock commented Apr 21, 2017

pwittrock commented Apr 21, 2017

pwittrock commented Apr 21, 2017

caesarxuchao commented Apr 21, 2017

pwittrock commented Apr 25, 2017

deads2k commented Apr 25, 2017

monopole commented Apr 26, 2017

k8s-github-robot commented Apr 26, 2017

k8s-github-robot commented Apr 26, 2017

caesarxuchao Apr 21, 2017 •

edited

Loading