Repository navigation
API: provide examples of use of numpy random API via user stories #14778
Description
Gathered from gh-14604, gh-14517 and the discussions.
-
"To summarize I need to draw random ints of a given C type from continually changing ranges, either one-by-one or small batch-by-small batch."
-
Someone asked how to use random in a ufunc.
-
"An ideal API would allow projects like https://github.com/deepmind/torch-randomkit/tree/master/randomkit or numba to consume the code in NumPy without vendoring it."
-
"There are c++ applications which use boost::random, would be nice to be able to swap it for numpy.random."
-
"Using the existing distributions from Cython was a requested feature and an explicit goal, yes. There are users waiting for this." (mattip: but isn't this supported via Generator?)
-
"Numba would definitely appreciate C functions to access the random distribution implementations, and
-
have a side-project (numba-scipy) that is making the Cython wrapped functions in SciPy visible to Numba" (mattip: I think we do this after API: rearrange the cython files in numpy.random #14608 from _generator.so via these cdef extern declarations, but that requires the H file.
-
Someone who wants to write a new BitGenerator, i.e., the numpy/bitgenerator repo without needing to have the numpy code as a git submodule
Some of these are already be handled by gh-14604, but I am putting them here for completeness
Edit: turned into a checklist
Activity
rkern commentedon Oct 25, 2019
No. In the context of the discussion this statement was drawn from, this was referring to the functions in
distributions.c: efficient C-level functions for drawing numbers one-by-one without all of the overhead that theGeneratormethods have for cramming these numbers into numpy arrays. That's what people have been waiting for.mattip commentedon Nov 15, 2019
@rkern the way the code is currently structured in #14604 we do not ship
distributions.cdirectly, rather we expose all the funcitons in_generator.pyxfrom the shared object_generator.*.so. If we wish to officially support them from C we should either add a_generator.pxdwhich will expose them viacimport _generatoror ship a_generator.hheader file and a link library for MSVC. I think apxdfile would be better, what do you think?bashtage commentedon Nov 21, 2019
Coming from #14951, I think the C functions represent are a lot of the value add of a low-level interface. While some are pretty trival, many are not.
I don't think it adds much to maintenance since these functions are all used by
Generator. IMO the simple thing to do is to always have the set ofrandom_functions exactly match the set required inGenerator. Functions that are not inGeneratorshould probably not be exposed, and if not used inGeneratororRandomState, then they should be removed (don't think there are many if any like this).Did you change the export of the distributions in #14604? I couldn't figure out where I might find them.
mattip commentedon Nov 21, 2019
I removed unused functions and renamed some others more consistently (mainly
gauss_zig->normal) in #14604 which alsodistributions.htonumpy/core/include/numpy/random/distributions.hdistributions.pxdtomtrand.pyxand_generator.pyxdistributions.pxdThe functions are exported from the _generator c-extension shared object, so the issue is how best to access them from C/Cython/CFFI. xref #14954 (the first commit there shows there is a problem on windows)
bashtage commentedon Nov 21, 2019
The Windows .pyd files only export a single symbol,
PyInit_generator. If this is fixable, then the numba examples will work well.So that leaves Cython. I've always compiled the C source directly into Cython modules. Is there a reasonable alternative? Most examples that I found searching for Cython dll assume that you have the source of the DLL code, and don't actually rely on the DLL when the Cython extension module is built.
mattip commentedon Nov 22, 2019
The numpy code is only a click away. This has the added advantage of compiling the code directly, rather than calling into a DLL which not only adds runtime overhead but allows for compile-time optimizations.
rkern commentedon Nov 22, 2019
We need to expose the functions in the DLL/SO so that numba can use them. They have made an unsupported hack into the
mtrandDLL/SO, and we need to move them off onto something supported. Copying C source code from the numpy source dist doesn't help them.mattip commentedon Nov 22, 2019
The functions in
distributions.hare now exposed from the_generatorDLL/SO. Can you point to a numba issue or PR so we can help them work out how to do this moving forward?rkern commentedon Nov 22, 2019
I'm only reacting to @bashtage's comment: "The Windows .pyd files only export a single symbol, PyInit_generator. If this is fixable, then the numba examples will work well."
mattip commentedon Nov 22, 2019
The export on windows was a bug that is fixed in gh-14954
bashtage commentedon Nov 22, 2019
rgommers commentedon Nov 28, 2019
Since the last open PR was merged, this is probably the best issue for questions/comments. So here are some:
bitgenstruct. From that it's unclear what the difference is betweennext_uint64andnext_raw. It just refers to "raw value" but is that a differentuint64number than returned bynext_uint64?s_binomial_t, some_zignames, some other cryptic abbreviations (_mvhg,_btpe)cimport numpy.random._bit_generatorandcimport numpy.random._generatorSee extending for exampleslink on that same C API page is broken_standard? It only saves the user adding, 0.0, 1.0)to a function call (or outside of the function call) as far as I can tell. It's also unclear if the_standardfunctions are made available consistently (e.g.normalhas with/without_standard,lognormaldoes not)._generatorin that CFFI section.bashtage commentedon Nov 28, 2019
Next raw is the next number using the natural number of bits the bg uses. It can be 32, 53, 63 or 64 in the range of generators I've considered. It is used to directly test the correctness against a reference implementation.
6 remaining items
rgommers commentedon Nov 29, 2019
Right indeed. For the names that match Python it is what it is, I missed that. 1.18.0 only matter for the new names of functions that don't have a Python equivalent.
mattip commentedon Nov 29, 2019
For Cython, we could somehow extend the
__init__.pxdso the pattern would becimport numpy.randomwhich would expose all the gory details. I tried to do this and it was messy, but maybe doable. However CFFI can only access the shared object/DLL, which is named_generator.*.somattip commentedon Nov 29, 2019
The top half of that document is Cython-specific, the bottom half builds on it to document the available C functions. Perhaps it could be divided into two pages
mattip commentedon Nov 29, 2019
We should decide which of the points @rgommers raised are 1.18 blockers and break them out into separate issues to be resolved in the next couple of days.
mattip commentedon Nov 29, 2019
This is standard stuff for CFFI. All the C wrapper libraries have a boilerplate problem: ctypes must define
argsandretype, Cython needpxdfiles, SWIG needs interface files, CFFI needs to pre-parse headers.bashtage commentedon Nov 29, 2019
Should the C functions correct this? They could become random_student_t (or students_t) and random_cauchy.
bashtage commentedon Nov 29, 2019
Sorry, on the phone.
It should have said: CFFI is pretty easy if you only want to use one or a few functions.
See
https://github.com/numpy/numpy/blob/7548073e58e89b414ca999d666c19189291dc1e5/numpy/random/_examples/numba/extending_distributions.py
which is no longer in master (and isn't quite right, but easy to fix). There is no need for tons of boiler plate. TBH, I'm not sure why @mattip got rid of this simple example which is basic but follows the style pretty much every CFFI example on the internet.
Of course, NumPy could provide a components needed for CFFI interfacing, but this seems to me like committing support something that may not be worth the overhead since downstream might want to do things differently.
bashtage commentedon Nov 29, 2019
As for 1.18, I think only naming if essential, since changing it later is harder. I don't think it really is essential to rename some of the functions, but my vote would be
standard_t -> student_t
standard_cauchy -> cauchy
Remvoe all
_zigsince there are no other versions of these distributionsThe one feature that I would really like, and I think is needed to complete the story, is to have something like npymath.lib for the C functions, perhaps npyrandom.lib. This would allow C/Cython to link to these functions without needing to get the C source form somewhere. For my, this would allow what will be a slimmed-down version of
randomgento become a dependent on the random code in NumPy instead of having an out-of-sync copy. This is a bigger ask though, and can definitely wait.mattip commentedon Nov 29, 2019
Must be a glitch in the way you are looking at the repo. The file is there, it is referenced in the docs but not tested since it is complicated to run.
The example I provided shows how to use all the functions available in
distributions.h, and is closer to how I have seen CFFI used in industry. I agree that many CFFI examples and tutorials "cheat" by redefining only one or two functions without parsing a header, but I wanted to provide a more complete and powerful example. It seems I missed the mark, the attention was drawn to the boilerplate and not the actual code. I will refactor the parsing into a separate file.Then we would add
_legacyto the "bare" versions needed formtrand.pyx?mattip commentedon Nov 29, 2019
nvermind, that is wrong. Changing the names as suggested in gh-15007
mattip commentedon Nov 29, 2019
The Generator.standard_exponential method has a
zigkeyword. This uses therandom_standard_exponential_zig_fillandrandom_standard_exponential_fillfunctions fromdistributions.c. I will rename them torandom_standard_exponential_fillandrandom_standard_exponential_inv_fillrespectively, and droprandom_standard_exponential_inv_fillfrom the documentation.mattip commentedon Nov 29, 2019
Done in #15007 (pending review and passing tests)
jamesthomasgriffin commentedon Jan 29, 2020
The reference page: Cython API for random implies that it is possible to use a long list of functions "random_..." from Cython and points to Extending for examples of using these functions. However those examples only use the methods associated to bitgen_t.
I am unable to cimport any of those functions.
Perhaps "_generator.pxy" is required but is not present. I am able to generate random numbers from bitgen_t.next_double, following the Extending example above.
Would it be possible to clarify the documentation by including an example? Or have I misunderstood and those functions should be private, in which case could this be clarified?
Numpy/Python version information:
1.18.1 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 23:11:46) [MSC v.1916 64 bit (AMD64)]
mattip commentedon Jan 29, 2020
It seems the cython use case was indeed overlooked. There is an example using CFFI, but none for cython.
BUG: add missing c_distributions.pxd, enables cython use of random C-…