It would be nice to have support for an alternative to weight decay: the max column norm. It would basically mean, that after updating the weights, there's another step that rescales them according to their norm. This way the weights are only penalized if their norm exceeds a certain value.

I think I'd eventually implement this, but I'm not sure when I have time for this. If anyone else feels like it, he or she is welcome to take over ;)

Here's a snippet from the pylearn2 code implementing it.

        if self.max_col_norm is not None:
            assert self.max_row_norm is None
            W ,= self.transformer.get_params()
            if W in updates:
                updated_W = updates[W]
                col_norms = T.sqrt(T.sum(T.sqr(updated_W), axis=0))
                desired_norms = T.clip(col_norms, 0, self.max_col_norm)
                updates[W] = updated_W * (desired_norms / (1e-7 + col_norms))