This looks like a slightly frustrated comment I left some time ago 😄

We should probably figure it out now in this PR. :) I am not sure what the original intention of the code was---does the paper suggest that setting a controllable initial value of alpha is a good thing?

Also, this is unsafe: when a user deserializes a network from disk, this will overwrite alpha with whatever the userAlpha was. So either we need to ensure that userAlpha is always equal to what alpha(0) is when we serialize (and thus will also be equal when we deserialize), or find some other way to initialize it.

I know I am now leaving three comments here... sorry about that.

I realized the "clean" way to do this with the abstractions we have is to override CustomInitialize(); take a look at layer/layer.hpp or layer/batch_norm_impl.hpp for an example. That will allow setting the single weight of the network to userAlpha, and that will only be called when the network is being initialized---not when it's being just serialized or deserialized.

According to the paper mentioned in code, Yes, alpha is a trainable parameter and userAlpha is just for initialization. I am not sure but there is no need of using CustomInitialize(). I used alpha just like ratio in layer/dropout.hpp which is also trainable in the last commit. Please guide me if I am not able to understand.

This comment explains that the strategy here is incorrect. Please read it carefully and reference relevant code (and walk through the FFN code) if you don't believe me. We will need to use CustomInitialize().

In the new API this is actually not valid---when the layer object is constructed, its trainable weights are not valid. In fact even the size given by WeightSize() is not valid. Everything about the layer is invalid until ComputeOutputDimensions() and then SetWeightsPtr() is called. So, the process of setting the weight value in alpha can't be done until SetWeightsPtr().

Just to clarify, in the constructors, we do not need to deal with alpha at all---we can leave it empty. It will be set by SetWeights() before the network is used. 👍

Sorry if this is a little unclear, but consider the whole setup of the network to happen in two functions:

• SetWeights(): this function should call the MakeAlias() to set the internally-held alpha matrix (which is a single element) to be an alias of whatever memory is given.
• CustomInitialize(): this function should set the single value of the alpha matrix to the initialization value given by the user (userAlpha). This function will be called after SetWeights() is called, so there is no need to create the alias here.

There's no need to serialize alpha; the actual trained weights of the network (which includes alpha) are serialized by the FFN class. Here's how this works: when a network is deserialized (loaded), the FFN class will deserialize all the layers individually, and then deserialize the weights (all as one block), and then call SetWeights() on each layer to correctly set the layer weights. So you can see in that process that alpha will be correctly set in that case.

Just to clarify, in the constructors, we do not need to deal with alpha at all---we can leave it empty. It will be set by SetWeights() before the network is used. 👍

Do you want to also remove the unadapted PReLU tests from not_adapted/?