Swish (arxiv) is an activation function that has been shown to empirically outperform ReLU and several other popular activation functions on Inception-ResNet-v2 and MobileNet. On models with more layers Swish typically outperforms ReLU. Implementation is simple:

Sigma is just sigmoid.

Worth a PR?

cc @albanD @mruberry