It is not possible to use Sequential container with recurrent networks. TF offers this: MultiRNNCell. Would be great to tune Sequential for such cases or add a separate RNN container.