@longjon sweet!, very nice and small change compared to #520

@sguada this simplification to sharing could still be paired with a buffer owned by Net to solve the parallel execution issue. It will be much simpler patch thanks to @longjon

Re: moving the shared buffer to Net, do note that this is a tradeoff; it allows net concurrency at the cost of memory when nets are used sequentially. For example, this patch currently shares column buffer memory between all train and test nets during solving. (Also note that this will still break layer concurrency, when added in the future, either way.)

This might cause the following post-optimization crash

I1020 06:44:01.849261  3396 caffe.cpp:121] Optimization Done.
F1020 06:44:02.442028  3396 syncedmem.cpp:16] Check failed: error == cudaSuccess (29 vs. 0)  driver shutting down
*** Check failure stack trace: ***
    @     0x7fc406272daa  (unknown)
    @     0x7fc406272ce4  (unknown)
    @     0x7fc4062726e6  (unknown)
    @     0x7fc406275687  (unknown)
    @           0x499d99  caffe::SyncedMemory::~SyncedMemory()
    @           0x4aa862  boost::detail::sp_counted_impl_p<>::dispose()
    @           0x4b3499  caffe::Blob<>::~Blob()
    @     0x7fc401b5f149  (unknown)
    @     0x7fc401b5f195  (unknown)
    @     0x7fc401b44ecc  (unknown)
    @           0x4167a7  (unknown)
    @              (nil)  (unknown)

but I don't have time to investigate at the moment so I'm just noting it here.

Yes, I've noticed that as well. I don't immediately see anything in our code that would cause that, so my guess is that cuda has some static variables that are being destroyed before the shared buffer.

There is almost no document about the behavior of static variable in CUDA. Another solution is to create a singleton class containing the shared buffer.

// http://stackoverflow.com/questions/1008019/c-singleton-design-pattern
class SharedBuffer {
    public:
        static SharedBuffer& instance() {
            static SharedBuffer  instance; // Guaranteed to be destroyed.
                                  // Instantiated on first use.
            return instance;
        }

        shared_ptr<Blob<Dtype> > buffer() { return buffer_; }
    private:
        SharedBuffer() {};     // Constructor? (the {} brackets) are needed here.
        // Dont forget to declare these two. You want to make sure they
        // are unaccessable otherwise you may accidently get copies of
        // your singleton appearing.
        DISABLE_COPY_AND_ASSIGN(SharedBuffer);
        shared_ptr<Blob<Dtype> > buffer_;
};

@futurely I don't think that solves the problem; when would the static SharedBuffer instance be destroyed (thus destroying buffer_)?

One solution is to explictly destroy the shared buffer with a clean-up function. But, that's somewhat irritating, and requires an extra line to be added to all Caffe invocations.

The static instance itself should also be a shared_ptr.

// http://stackoverflow.com/questions/1008019/c-singleton-design-pattern
class SharedBuffer {
    public:
        static shared_ptr<SharedBuffer> instance() {
            // Guaranteed to be destroyed. Instantiated on first use.
            static shared_ptr<SharedBuffer> instance(new SharedBuffer());                                   
            return instance;
        }
       ...
};

@futurely, I don't see how using a shared_ptr is any different... the lifetime of the SharedBuffer object is the same as the lifetime of the shared_ptr, right?

However, I think I see the point of the original code now, which might actually address the issue... if the static variable is method-local, it doesn't get allocated until the method is called, and therefore gets destroyed before any global static variables. (If I understand correctly, then, there's no need for a singleton class, just for an accessor function.)

I'm not going to bother updating an unmergeable PR to fix a post-optimization crash, but we can keep that trick in mind for the future.

There are still some chances to save this PR. It's all about the lifetime of static variable.

Yeah... that's what we're discussing here.

To be clear, the shared column buffer is still planned for merge, it just needs to be an option to avoid breaking concurrency. Until that's done there's no point in fixing the static destruction order issue, but let's keep it in mind for when it's ready (if it's still relevant then).

ArrayFire simply ignored this error.

Replaced by #2016.