Kubernetes service can't support SCTP protocol #44485

tonyaw · 2017-04-14T05:49:24Z

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):
No.

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):
service sctp

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
FEATURE REQUEST

Kubernetes version (use kubectl version):

kubectl version

Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", BuildDate:"2017-04-03T20:44:38Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-03-28T16:24:30Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: hardware configuration
OS (e.g. from /etc/os-release):

cat /etc/os-release

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Kernel (e.g. uname -a):

uname -a

Linux foss-ssc-3 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Install tools:
kubeadm
Others:

What happened:
Kubernetes service can't support SCTP as protocol:
Error from server (Invalid): error when creating "tonyaw-svc.yaml": Service "tonyaw-rf" is invalid: spec.ports[0].protocol: Unsupported value: "SCTP": supported values: TCP, UDP

What you expected to happen:
protocol should support TCP, UDP and SCTP

How to reproduce it (as minimally and precisely as possible):
kind: Service
apiVersion: v1
metadata:
name: tonyaw-rf
spec:
clusterIP: None
selector:
app: myapp
ports:
- name: app
protocol: SCTP
port: 7890
targetPort: 7890

Anything else we need to know:

The text was updated successfully, but these errors were encountered:

0xmichalis · 2017-04-14T17:00:53Z

@kubernetes/sig-network-feature-requests

thockin · 2017-04-14T17:07:06Z

I don't have any problem opening this discussion, but we have no tests for this, so it will have to be a top-to-bottom effort. * vet the iptables logic * API change (pretty easy) for service ports and pod container ports * verify that cloudprovider LBs support it * e2e tests and test tools

…

On Fri, Apr 14, 2017 at 10:01 AM, Michail Kargakis ***@***.*** > wrote: @kubernetes/sig-network-feature-requests <https://github.com/orgs/kubernetes/teams/sig-network-feature-requests> — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#44485 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVAZUbbnfWgR6U7BiRd9JTZK7mCKgks5rv6ZZgaJpZM4M9ej9> .

janosi · 2017-04-14T20:32:49Z

@thockin I think, there are 2 main use cases here:

services with ClusterIP
headless services

The actual example is about a headless service. More exactly, a headless service with a named port. The aim with this was to got an SRV record in kube DNS with protocol "_sctp". As the kube-proxy does not create iptables rules for headless services, nor it wants to start listening on the defined ports(and protocol), do you think it would mess up things to enable SCTP as protocol for headless services?
So far so good maybe?
But then enabling SCTP as protocol for services with ClusterIP may be more complicated. As you know kube-proxy wants to reserve the configured port with net.Listen(), but net.listen() does not support "SCTP" as protocol. As a workaround it may be possible to use Go syscall.Socket() + syscall.Bind() instead of net.Listen() (as the sole purpose is to reserve the port so no other app on the same host could use it).

About cloudprovider LB support: is it so, that all features must be supported by the cloud provider LB? Like it is in the case of loadBalancerIP attribute: according to the documentation it is OK that loadBalancerIP is optional - some cloud provider support that, some not. Wouldn't it be the same with protocol:SCTP?

thockin · 2017-04-14T20:46:33Z

On Fri, Apr 14, 2017 at 1:33 PM, Laszlo Janosi ***@***.***> wrote: @thockin I think, there are 2 main use cases here: services with ClusterIP headless services The actual example is about a headless service. More exactly, a headless service with a named port. The aim with this was to got an SRV record in kube DNS with protocol "_sctp". As the kube-proxy does not create iptables rules for headless services, nor it wants to start listening on the defined ports(and protocol). Do you think it would mess up things to enable SCTP as protocol for headless services?

Making ti work in some-but-not-all cases would be more likely to cause trouble, I think. A quick test shows iptables works with SCTP.

So far so good maybe? But then enabling SCTP as protocol for services with ClusterIP may be more complicated. As you know kube-proxy wants to reserve the configured port with net.Listen(), but net.listen() does not support "SCTP" as protocol. As a workaround it may be possible to use Go syscall.Socket() + syscall.Bind() instead of net.Listen() (as the sole purpose is to reserve the port so no other app on the same host could use it).

Yeah, I didn't want to listen() at all, and the first version did just what you say. The problem was that I could not find a way to trivially prove that a socket was in such a state in my e2e tests (netstat and ss don't show them!!) If you can solve that, I would love it.

About cloudprovider LB support: is it so, that all features must be supported by the cloud provider LB? Like it is in the case of loadBalancerIP attribute: according to the documentation it is OK that loadBalancerIP is optional - some cloud provider support that, some not. Wouldn't it be the same with protocol:SCTP?

I'd like to at least know what the state is. I know gogole cloud does not support it, for example, so we would have to at least check that code to make sure it fails in a reasonable way.

janosi · 2017-04-15T21:01:42Z

Regarding testing whether the socket is in use already: what about a small test tool program that tries to bind on the IP+port given as parameters? If the IP+port is already in use, bind() fails with error "address already in use". I tried it via binding via the syscall API first and then trying to re-run the same on the same IP+port. The 2nd bind attempt failed.
Regarding cloud providers vs SCTP: as I understand AWS supports SCTP in VPC (and ELB for VCP) via security group settings (where other protocols than TCP or UDP or ICMP can be configured), but not for classic EC2. Azure seems to support only TCP or UDP. Though, it only means, that k8s should not allow SCTP as protocol for ports when Service type value is set to LoadBalancer.

thockin · 2017-04-15T21:37:45Z

I was trying to avoid shipping another custom test tool, but maybe that's the end result. Re clouds, I guess we "just" need to verify that cloud-provider code sets an appropriate event when it decides to not create an LB. We'll also need to check setup code for providers wrt firewalls for more than TCP/UDP. I am not against this change, it's just quite a lot more than adding an enum value :) On Apr 15, 2017 2:02 PM, "Laszlo Janosi" <[email protected]> wrote: Regarding testing whether the socket is in use already: what about a small test tool program that tries to bind on the IP+port given as parameters? If the IP+port is already in use, bind() fails with error "address already in use". I tried it via binding via the syscall API first and then trying to re-run the same on the same IP+port. The 2nd bind attempt failed. Regarding cloud providers vs SCTP: as I understand AWS supports SCTP in VPC (and ELB for VCP) via security group settings (where other protocols than TCP or UDP or ICMP can be configured), but not for classic EC2. Azure seems to support only TCP or UDP. Though, it only means, that k8s should not allow SCTP as protocol for ports when Service type value is set to LoadBalancer. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#44485 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVEfmi9TnLbjhAhZ91hMXAFaplF9uks5rwTBGgaJpZM4M9ej9> .

laf0rge · 2017-05-05T18:10:11Z

A quick test shows iptables works with SCTP.

yes, since 2003. SCTP is not new: https://git.netfilter.org/iptables/commit/?id=8f578a09b56f010d5bcd30086a8f7c8132b35d92

kernel code for matching SCTP ports has been merged in 2004: https://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/commit/net/ipv4/netfilter/ipt_sctp.c?id=37306cb0d818cdc3846d8ffbfe2d717125ee6a9b

kernel connection tracking code for sctp has been merged also in 2004: https://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/commit/net/ipv4/netfilter/ip_conntrack_proto_sctp.c?id=21b7bdf3c5b4a99573fcdc4609c469cf779d99c4

Please also note there are various other L4 protocols like udp-lite, DCCP, ... - the planet doesn't consist only of TCP+UDP (like in 1990ies)

fejta-bot · 2017-12-24T14:28:39Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

rkbug · 2018-01-09T19:31:23Z

Are there any updates to this issue? Someone working on this?

fejta-bot · 2018-02-10T21:47:09Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

RafalKrych · 2018-02-17T14:58:44Z

I'm working as software architect for telco company, Our business is about delivering RAN software for 3G/4G/5G networks. All these networks are SCTP based. We are starting a new project which is about full cloudification of our software, likely by using containers & k8s. I mean here products which are used by like 1.5 billion people everyday. Is there any chance that SCTP protocol will receive full support in k8s? Including LoadBalancer? Of course we consider our customers (AT&T, T-mobile, Swisscom just to name a few) may use AWS or Google cloud for their deployments but most likely they will come with own hardware which will support SCTP loadbalancing. Is it possible that our developers can help you with adaptation for SCTP? I can look for this further if you are interested in such cooperation.

fejta-bot · 2018-03-19T16:11:50Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

praseodym · 2018-04-15T17:00:03Z

Docker 18.03 supports exposing SCTP ports; it would be great if Kubernetes would too.

stepin · 2018-05-15T13:52:49Z

/remove-lifecycle rotten

stepin · 2018-05-15T13:53:35Z

/reopen

k8s-ci-robot · 2018-05-15T13:53:36Z

@stepin: you can't re-open an issue/PR unless you authored it or you are assigned to it.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2018-06-28T20:51:11Z

@bowei: you can't re-open an issue/PR unless you authored it or you are assigned to it.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2018-06-28T20:51:26Z

@bowei: you can't re-open an issue/PR unless you authored it or you are assigned to it.

In response to this:

/assign
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Automatic merge from submit-queue (batch tested with PRs 67694, 64973, 67902). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. SCTP support implementation for Kubernetes **What this PR does / why we need it**: This PR adds SCTP support to Kubernetes, including Service, Endpoint, and NetworkPolicy. **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: Fixes #44485 **Special notes for your reviewer**: **Release note**: ```release-note SCTP is now supported as additional protocol (alpha) alongside TCP and UDP in Pod, Service, Endpoint, and NetworkPolicy. ```

yuyongzhang1 · 2018-12-11T18:31:50Z

RafalKrych/all,

I am looking to deploy sctp protocol based service in Kubernetes cluster, is k8s-sctp merge solve this problem?

which release support sctp?

Does Kuburnetes ingress support sctp as well?

Any pointers are very much appreciated.

-zyy

yuyongzhang1 · 2018-12-11T18:33:58Z

Anyone knows if GCE supports SCTP now?

Thanks a lot.

-zyy

janosi · 2018-12-11T19:26:35Z

@yuyongzhang1 SCTP is supported as alpha feature from k8s 1.12 - you can read the documentation here: https://kubernetes.io/docs/concepts/services-networking/service/#sctp-support

SCTP vs GCE: the GCE Load Balancer does not support SCTP. SCTP support in VMs depends on the VM's base image: the GCE minimalistic container image does not contain the SCTP kernel module, so kernel level SCTP is not supported there.

0xmichalis added kind/feature Categorizes issue or PR as related to a new feature. sig/network Categorizes an issue or PR as relevant to SIG Network. labels Apr 14, 2017

dcbw added area/kube-proxy and removed kind/feature Categorizes issue or PR as related to a new feature. labels May 18, 2017

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 24, 2017

hwinkel mentioned this issue Jan 5, 2018

SCTP support cloudnativelabs/kube-router#266

Closed

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 10, 2018

k8s-ci-robot closed this as completed Mar 19, 2018

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label May 15, 2018

janosi mentioned this issue Jun 11, 2018

SCTP support implementation for Kubernetes #64973

Merged

k8s-ci-robot assigned bowei Jun 28, 2018

k8s-ci-robot reopened this Jun 28, 2018

k8s-ci-robot unassigned bowei Jun 28, 2018

k8s-github-robot closed this as completed in #64973 Aug 28, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kubernetes service can't support SCTP protocol #44485

Kubernetes service can't support SCTP protocol #44485

tonyaw commented Apr 14, 2017

0xmichalis commented Apr 14, 2017

thockin commented Apr 14, 2017 via email

janosi commented Apr 14, 2017 •

edited

Loading

thockin commented Apr 14, 2017 via email

janosi commented Apr 15, 2017

thockin commented Apr 15, 2017 via email

laf0rge commented May 5, 2017 •

edited

Loading

fejta-bot commented Dec 24, 2017

rkbug commented Jan 9, 2018

fejta-bot commented Feb 10, 2018

RafalKrych commented Feb 17, 2018 •

edited

Loading

fejta-bot commented Mar 19, 2018

praseodym commented Apr 15, 2018

stepin commented May 15, 2018

stepin commented May 15, 2018

k8s-ci-robot commented May 15, 2018

k8s-ci-robot commented Jun 28, 2018

k8s-ci-robot commented Jun 28, 2018

yuyongzhang1 commented Dec 11, 2018

yuyongzhang1 commented Dec 11, 2018

janosi commented Dec 11, 2018

Kubernetes service can't support SCTP protocol #44485

Kubernetes service can't support SCTP protocol #44485

Comments

tonyaw commented Apr 14, 2017

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): service sctp

kubectl version

cat /etc/os-release

uname -a

0xmichalis commented Apr 14, 2017

thockin commented Apr 14, 2017 via email

janosi commented Apr 14, 2017 • edited Loading

thockin commented Apr 14, 2017 via email

janosi commented Apr 15, 2017

thockin commented Apr 15, 2017 via email

laf0rge commented May 5, 2017 • edited Loading

fejta-bot commented Dec 24, 2017

rkbug commented Jan 9, 2018

fejta-bot commented Feb 10, 2018

RafalKrych commented Feb 17, 2018 • edited Loading

fejta-bot commented Mar 19, 2018

praseodym commented Apr 15, 2018

stepin commented May 15, 2018

stepin commented May 15, 2018

k8s-ci-robot commented May 15, 2018

k8s-ci-robot commented Jun 28, 2018

k8s-ci-robot commented Jun 28, 2018

yuyongzhang1 commented Dec 11, 2018

yuyongzhang1 commented Dec 11, 2018

janosi commented Dec 11, 2018

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):
service sctp

janosi commented Apr 14, 2017 •

edited

Loading

laf0rge commented May 5, 2017 •

edited

Loading

RafalKrych commented Feb 17, 2018 •

edited

Loading