Add conditional to set _mv variable to np.uint64 data type#47
Add conditional to set _mv variable to np.uint64 data type#47robgpita wants to merge 2 commits intoDeltares:mainfrom
Conversation
…f large arrays. Ref Deltares#46
…ype, and _mv assertion
| self._mv = core._mv | ||
| if idxs_ds.dtype == np.uint32: | ||
| self._mv = np.uint32(self._mv) | ||
| if idxs_ds.dtype == np.uint64: |
There was a problem hiding this comment.
Just appears to be a dtype limitation?
There was a problem hiding this comment.
@fernando-aristizabal not exactly. The dtype assigned to self._mv needs to be the same as the dtype for idxs.ds. Those two dtypes need to line up, otherwise there were errors in numba code. In the downstream code, incorrect aliasing was occurring based on _mv's dtype. This was causing a mis-match in values between _mv and items in the idxs.ds array, which ultimately led to the IndexErrors being thrown.
In order to avoid the IndexErrors, for the large array indices case, where the idxs.ds.dtype is set to uint64, the dtype for _mv needs to be set to uint64 as well.
|
@robgpita and @fernando-aristizabal Many thanks for looking into this and proposing a solution. I've had to create a new branch with your commits in #56 to be able to merge it. With that merge of that branch I'm also closing this PR. Happy holidays! |
3 participants
Allow the processing of large rasters, shaped at (63708, 79780), for
Flwdirobject.Issue addressed
Closes #46
Explanation
As described in issue #46, when large rasters are used, an IndexError was thrown from underlying numba code.
The upstream_count function was the cause of the Indexing Error, due to an incorrect value set in the
_mvvariable when using anp.uint64data type.Checklist
mainAdditional Notes (optional)
The setting of the np.uint64 dtype in pyflwdir.py was not accounted for, and caused non-correct values in the
_mvvariable, when assigned in core.py. This was causing the C pointer references to not line up with the signed integer type that was being used in python. See numpy docs fornp.intp()The downstream effects