Skip to content

Numpy-dev failure for modelling #7678

Closed

Description

@mhvk

Hey, I beat @pllim to it! numpy-dev seems to be failing consistently for modeling (e.g., https://travis-ci.org/astropy/astropy/jobs/407200918)


�[1m�[31m/home/travis/miniconda/envs/test/lib/python3.6/site-packages/scipy/optimize/minpack.py�[0m:394: TypeError
_____________ ERROR at setup of TestJointFitter.test_joint_fitter ______________

self = <class 'astropy.modeling.tests.test_fitters.TestJointFitter'>

�[1m    def setup_class(self):�[0m
�[1m        """�[0m
�[1m            Create 2 gaussian models and some data with noise.�[0m
�[1m            Create a fitter for the two models keeping the amplitude parameter�[0m
�[1m            common for the two models.�[0m
�[1m            """�[0m
�[1m        self.g1 = models.Gaussian1D(10, mean=14.9, stddev=.3)�[0m
�[1m        self.g2 = models.Gaussian1D(10, mean=13, stddev=.4)�[0m
�[1m        self.jf = JointFitter([self.g1, self.g2],�[0m
�[1m                                      {self.g1: ['amplitude'],�[0m
�[1m                                       self.g2: ['amplitude']}, [9.8])�[0m
�[1m        self.x = np.arange(10, 20, .1)�[0m
�[1m        y1 = self.g1(self.x)�[0m
�[1m        y2 = self.g2(self.x)�[0m
�[1m    �[0m
�[1m        with NumpyRNGContext(_RANDOM_SEED):�[0m
�[1m            n = np.random.randn(100)�[0m
�[1m    �[0m
�[1m        self.ny1 = y1 + 2 * n�[0m
�[1m        self.ny2 = y2 + 2 * n�[0m
�[1m>       self.jf(self.x, self.ny1, self.x, self.ny2)�[0m

�[1m�[31mastropy/modeling/tests/test_fitters.py�[0m:155: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
�[1m�[31mastropy/modeling/fitting.py�[0m:1219: in __call__
�[1m    self.fitparams, args=args)�[0m
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

func = <bound method JointFitter.objective_function of <astropy.modeling.fitting.JointFitter object at 0x7fe06e88cdd8>>
x0 = array([9.8, array([14.9]), array([0.3]), array([13.]), array([0.4])],
      dtype=object)
args = (array([10. , 10.1, 10.2, 10.3, 10.4, 10.5, 10.6, 10.7, 10.8, 10.9, 11. ,
       11.1, 11.2, 11.3, 11.4, 11.5, 11.6, 1...595e+00, -1.30030932e+00, -2.36494839e+00,
       -2.25212374e+00,  3.26000074e+00,  1.37397670e+00,  4.73034211e-02]))
Dfun = None, full_output = 0, col_deriv = 0, ftol = 1.49012e-08
xtol = 1.49012e-08, gtol = 0.0, maxfev = 1200, epsfcn = 2.220446049250313e-16
factor = 100, diag = None

�[1m    def leastsq(func, x0, args=(), Dfun=None, full_output=0,�[0m
�[1m                col_deriv=0, ftol=1.49012e-8, xtol=1.49012e-8,�[0m
�[1m                gtol=0.0, maxfev=0, epsfcn=None, factor=100, diag=None):�[0m
�[1m        """�[0m
�[1m        Minimize the sum of squares of a set of equations.�[0m
�[1m    �[0m
�[1m        ::�[0m
�[1m    �[0m
�[1m            x = arg min(sum(func(y)**2,axis=0))�[0m
�[1m                     y�[0m
�[1m    �[0m
�[1m        Parameters�[0m
�[1m        ----------�[0m
�[1m        func : callable�[0m
�[1m            should take at least one (possibly length N vector) argument and�[0m
�[1m            returns M floating point numbers. It must not return NaNs or�[0m
�[1m            fitting might fail.�[0m
�[1m        x0 : ndarray�[0m
�[1m            The starting estimate for the minimization.�[0m
�[1m        args : tuple, optional�[0m
�[1m            Any extra arguments to func are placed in this tuple.�[0m
�[1m        Dfun : callable, optional�[0m
�[1m            A function or method to compute the Jacobian of func with derivatives�[0m
�[1m            across the rows. If this is None, the Jacobian will be estimated.�[0m
�[1m        full_output : bool, optional�[0m
�[1m            non-zero to return all optional outputs.�[0m
�[1m        col_deriv : bool, optional�[0m
�[1m            non-zero to specify that the Jacobian function computes derivatives�[0m
�[1m            down the columns (faster, because there is no transpose operation).�[0m
�[1m        ftol : float, optional�[0m
�[1m            Relative error desired in the sum of squares.�[0m
�[1m        xtol : float, optional�[0m
�[1m            Relative error desired in the approximate solution.�[0m
�[1m        gtol : float, optional�[0m
�[1m            Orthogonality desired between the function vector and the columns of�[0m
�[1m            the Jacobian.�[0m
�[1m        maxfev : int, optional�[0m
�[1m            The maximum number of calls to the function. If `Dfun` is provided�[0m
�[1m            then the default `maxfev` is 100*(N+1) where N is the number of elements�[0m
�[1m            in x0, otherwise the default `maxfev` is 200*(N+1).�[0m
�[1m        epsfcn : float, optional�[0m
�[1m            A variable used in determining a suitable step length for the forward-�[0m
�[1m            difference approximation of the Jacobian (for Dfun=None).�[0m
�[1m            Normally the actual step length will be sqrt(epsfcn)*x�[0m
�[1m            If epsfcn is less than the machine precision, it is assumed that the�[0m
�[1m            relative errors are of the order of the machine precision.�[0m
�[1m        factor : float, optional�[0m
�[1m            A parameter determining the initial step bound�[0m
�[1m            (``factor * || diag * x||``). Should be in interval ``(0.1, 100)``.�[0m
�[1m        diag : sequence, optional�[0m
�[1m            N positive entries that serve as a scale factors for the variables.�[0m
�[1m    �[0m
�[1m        Returns�[0m
�[1m        -------�[0m
�[1m        x : ndarray�[0m
�[1m            The solution (or the result of the last iteration for an unsuccessful�[0m
�[1m            call).�[0m
�[1m        cov_x : ndarray�[0m
�[1m            Uses the fjac and ipvt optional outputs to construct an�[0m
�[1m            estimate of the jacobian around the solution. None if a�[0m
�[1m            singular matrix encountered (indicates very flat curvature in�[0m
�[1m            some direction).  This matrix must be multiplied by the�[0m
�[1m            residual variance to get the covariance of the�[0m
�[1m            parameter estimates -- see curve_fit.�[0m
�[1m        infodict : dict�[0m
�[1m            a dictionary of optional outputs with the key s:�[0m
�[1m    �[0m
�[1m            ``nfev``�[0m
�[1m                The number of function calls�[0m
�[1m            ``fvec``�[0m
�[1m                The function evaluated at the output�[0m
�[1m            ``fjac``�[0m
�[1m                A permutation of the R matrix of a QR�[0m
�[1m                factorization of the final approximate�[0m
�[1m                Jacobian matrix, stored column wise.�[0m
�[1m                Together with ipvt, the covariance of the�[0m
�[1m                estimate can be approximated.�[0m
�[1m            ``ipvt``�[0m
�[1m                An integer array of length N which defines�[0m
�[1m                a permutation matrix, p, such that�[0m
�[1m                fjac*p = q*r, where r is upper triangular�[0m
�[1m                with diagonal elements of nonincreasing�[0m
�[1m                magnitude. Column j of p is column ipvt(j)�[0m
�[1m                of the identity matrix.�[0m
�[1m            ``qtf``�[0m
�[1m                The vector (transpose(q) * fvec).�[0m
�[1m    �[0m
�[1m        mesg : str�[0m
�[1m            A string message giving information about the cause of failure.�[0m
�[1m        ier : int�[0m
�[1m            An integer flag.  If it is equal to 1, 2, 3 or 4, the solution was�[0m
�[1m            found.  Otherwise, the solution was not found. In either case, the�[0m
�[1m            optional output variable 'mesg' gives more information.�[0m
�[1m    �[0m
�[1m        Notes�[0m
�[1m        -----�[0m
�[1m        "leastsq" is a wrapper around MINPACK's lmdif and lmder algorithms.�[0m
�[1m    �[0m
�[1m        cov_x is a Jacobian approximation to the Hessian of the least squares�[0m
�[1m        objective function.�[0m
�[1m        This approximation assumes that the objective function is based on the�[0m
�[1m        difference between some observed target data (ydata) and a (non-linear)�[0m
�[1m        function of the parameters `f(xdata, params)` ::�[0m
�[1m    �[0m
�[1m               func(params) = ydata - f(xdata, params)�[0m
�[1m    �[0m
�[1m        so that the objective function is ::�[0m
�[1m    �[0m
�[1m               min   sum((ydata - f(xdata, params))**2, axis=0)�[0m
�[1m             params�[0m
�[1m    �[0m
�[1m        The solution, `x`, is always a 1D array, regardless of the shape of `x0`,�[0m
�[1m        or whether `x0` is a scalar.�[0m
�[1m        """�[0m
�[1m        x0 = asarray(x0).flatten()�[0m
�[1m        n = len(x0)�[0m
�[1m        if not isinstance(args, tuple):�[0m
�[1m            args = (args,)�[0m
�[1m        shape, dtype = _check_func('leastsq', 'func', func, x0, args, n)�[0m
�[1m        m = shape[0]�[0m
�[1m        if n > m:�[0m
�[1m            raise TypeError('Improper input: N=%s must not exceed M=%s' % (n, m))�[0m
�[1m        if epsfcn is None:�[0m
�[1m            epsfcn = finfo(dtype).eps�[0m
�[1m        if Dfun is None:�[0m
�[1m            if maxfev == 0:�[0m
�[1m                maxfev = 200*(n + 1)�[0m
�[1m            with _MINPACK_LOCK:�[0m
�[1m                retval = _minpack._lmdif(func, x0, args, full_output, ftol, xtol,�[0m
�[1m>                                        gtol, maxfev, epsfcn, factor, diag)�[0m
�[1m�[31mE               TypeError: Cannot cast array data from dtype('O') to dtype('float64') according to the rule 'safe'�[0m

Activity

  1. pllim commented on Jul 24, 2018

    @pllim
    Member
  2. saimn commented on Sep 10, 2018

    @saimn
    Contributor

    With Numpy 1.15:

    In [4]: x0 = [9.8, array([14.9]), array([0.3]), array([13.]), array([0.4])]
    
    In [5]: np.asarray(x0).flatten()
    Out[5]: array([ 9.8, 14.9,  0.3, 13. ,  0.4])
    

    And Numpy 1.16dev

    In [3]: x0 = [9.8, array([14.9]), array([0.3]), array([13.]), array([0.4])]
    
    In [4]: np.asarray(x0).flatten()
    Out[4]: 
    array([9.8, array([14.9]), array([0.3]), array([13.]), array([0.4])],
          dtype=object)
    
  3. pllim commented on Sep 10, 2018

    @pllim
    Member

    The 1.16dev behavior is not intuitive to me. Is that a bug? 😱

  4. mhvk commented on Sep 11, 2018

    @mhvk
    ContributorAuthor

    I'd say it is actually a little more correct: in the example, all the arrays are already 1-dimensional arrays, while the number is a scalar, so how can they be logically combined? 1.15 just made the 1-D arrays scalars, but I find that more surprising. One can see what the oddity is by making the number a list:

    x0 = [[9.8], array([14.9]), array([0.3]), array([13.]), array([0.4])]
    np.asarray(x0)
    array([[ 9.8],
           [14.9],
           [ 0.3],
           [13. ],
           [ 0.4]])
    

    Anyway, I think we need to adjust our code here to more clearly indicate what we want it to do. If we do just want an 1-D array out and our numbers are always scalars, then replacing by np.hstack would seem to do the trick... (since that implicitly upgrades scalars to 1-d arrays):

    x0 = [9.8, array([14.9]), array([0.3]), array([13.]), array([0.4])]
    np.hstack(x0)
    # array([ 9.8, 14.9,  0.3, 13. ,  0.4])
    
  5. astrofrog commented on Oct 26, 2018

    @astrofrog
    Member

    @nden - it looks like this is the only remaining failure in the Numpy-dev build. Do you think it's fixable for the v3.1 release to make sure things don't break once they release 1.16?

  6. nden commented on Oct 26, 2018

    @nden
    Contributor

    @astrofrog I'll try to take a look today. I wish I could just remove that class.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions