dont merge yet, we messed up image pulling

@kcmannem @deniseyu

This seems kinda roundabout to me. You are basically saying find me a worker and create a container, then you expose back the worker just so that we can look up the container all over again.

Why don't we just add a CreateContainer method in worker.Client, which returns a Creating container so that we can move closer to not caring about worker selection. Then we can move a bunch of the worker.FindOrCreateContainer logic, such as input streaming, into a method on the creating container itself (or something that takes in a creating container).

It will probably also help to separate the FindOrCreate logic into Find or Create and push that orchestration into core. Once we move to ephemeral check containers we will never be looking up an existing container, only ever creating new ones.

\cc @ddadlani

@pivotal-jwinters I think the goal is to get to what you were saying. I agree its kind of weird right now. But this wasn't meant to be that big refactor but rather way to get things correct again and ease some pain points for users. This gives them something while we can work on longer term solutions after.

@wagdav FYI

@vito @ddadlani would you please take a look at this PR? My understanding is that this would be a fix for #3301 and possibly other scheduler related problems. Thanks!

@wagdav this change exposed a larger issue where checks error every now and then. Another recent change in how check sessions are handled also makes it difficult to patch this issue. We are still working on a fix for that error, after which we will merge this into master

@wagdav we're gunna workaround the issue, this should be going in considering the tests pass.

@ddadlani fixed the tests, we accidentally commit changes to the docker-compose. its ready for review now.

fixes #3301

fixes #3301

@ddadlani your comment didn't really close issue #3301, was that intended? (I thought that GitHub should close the issue with this type of comments...)

@aledeganopix4d yes it should have. I think I had to comment that before I merged this PR, though. Thanks for pointing it out!