API: Allow dir(astropy) to list all subpackage names #17589

Activity

1. pllim

   added

   Feature Request

   on Dec 31, 2024
2. neutrinoceros commented on Jan 1, 2025

   

   neutrinoceros

   on Jan 1, 2025

   Contributor

   More actions

   The desired outcome is relatively straightfoward to achieve using a module-level __dir__:

   diff --git a/astropy/__init__.py b/astropy/__init__.py
   index a7f600ba4c..a8c8847b99 100644
   --- a/astropy/__init__.py
   +++ b/astropy/__init__.py
   @@ -237,3 +238,11 @@ for varname in dir():
            del locals()[varname]
    
    del varname, __module_type__
   +
   +def __dir__():
   +    local_dir = list(globals())
   +    force_includes = ["stats"]
   +    for incl in force_includes:
   +        if incl not in local_dir:
   +            local_dir.append(incl)
   +    return local_dir

   However, listing, e.g., stats in this fashion would imply that the following works:

   python -c "import astropy; astropy.stats" 
   

   but it doesn't, because stats is only loaded lazily (by design)

   Traceback (most recent call last):
     File "<string>", line 1, in <module>
   AttributeError: module 'astropy' has no attribute 'stats'
   

   Note that

   python -c "import astropy.stats; print(f'''{'stats' in dir(astropy) = }''')"
   

   prints, as of astropy 7.0.0

   'stats' in dir(astropy) = True
   

   My conclusion is I don't think we can satisfy the expectation here without a serious undermining of our lazy loading strategy.
3. mhvk commented on Jan 2, 2025

   

   mhvk

   on Jan 2, 2025

   Contributor

   More actions

   Can we not get the best of all worlds by also defining a module level __getattr__? Like,

   def __getattr__(attr):
       from importlib import import_module
       try:
           return import_module("astropy." + attr)
       except ImportError:
           pass
       raise AttributeError(...)
   
4. celik-muhammed commented on Jan 2, 2025

   

   celik-muhammed

   on Jan 2, 2025

   Last edited by celik-muhammed

   More actions

   dir(module) will return a list of names found in the module’s globals() 
   i.e., everything that is part of the module’s namespace.
   It also relies on __dir__() if defined.
   
   To customize the behavior of dir() for a module, we need to define __dir__().
   
   For example, if we define __dir__() as:
   def __dir__():
       return list(globals().keys()) + _available_submodules
       
   This could cause re-importing. _available_submodules is dynamically generated and returned as a plain value when accessed, but there is no caching or state to remember its previous value.
   
   To avoid this, we need to explicitly manage the state by checking globals() first, 
   then __all__, followed by __dir__(), and finally using __getattr__() for dynamic imports.
   
   This approach reduces complexity and ensures proper handling of dynamically generated submodules and attributes.
   
   
   here full exmple for define seperately __all__, and dynamic namespases _available_submodules 
   
   but my opinion all modules must add globals() then can listed in __all__  list not seperately to easy manage.
   
   
   # Define _available_submodules dynamically (example)
   _available_submodules = ['submodule1', 'submodule2']  # This can be dynamically generated as needed.
   
   # Define __all__ to explicitly list the submodules you want to expose
   __all__ = ['submodule1', 'submodule2']
   
   def __dir__():
       """
       Customize the behavior of dir() for this module. Returns a list of 
       attributes, including both predefined attributes in globals() and 
       dynamically generated submodules.
       """
       # Return predefined attributes and the list of available submodules
       return list(globals().keys()) + _available_submodules
   
   import importlib
   import sys
   import difflib
   def __getattr__(name):
       """
       Dynamically handle undefined attributes in the module. This function 
       attempts to dynamically import submodules or access attributes from 
       globals() and sys.modules.
       """
       
       # Check if the requested name is already in the global namespace
       if name in globals():
           return globals()[name]
   
       # Check if the submodule is already imported (in sys.modules)
       if name in sys.modules:
           return sys.modules[name]
   
       # Check if the requested name exists in the list of available submodules
       if name in _available_submodules:
           try:
               # Dynamically import the submodule
               module = importlib.import_module(f'.{name}', package=__name__)
               # Add to globals and sys.modules for future access
               globals()[name] = module
               sys.modules[name] = module
               return module
           except ModuleNotFoundError:
               pass  # If the module cannot be found, fall through to error handling
   
       # Generate suggestions for mistyped names based on globals() and __all__
       all_names = set(globals().keys()) | set(getattr(__name__, '__all__', []))
       suggestions = difflib.get_close_matches(name, all_names)
       suggestion_msg = f" Did you mean: {', '.join(suggestions)}?" if suggestions else ""
   
       # Raise an error if the attribute cannot be found
       raise AttributeError(f"Module '{__name__}' has no attribute '{name}'.{suggestion_msg}")
   
   
   
   
5. mhvk commented on Jan 2, 2025

   

   mhvk

   on Jan 2, 2025

   Contributor

   More actions

   See #17598 - it is a bit simpler than suggested above, since __getattr__ only gets called if a module is not already imported.

   👍React with 👍1Reacted by Muhammed ÇELİK
6. mhvk

   closed this as completedin #17598on Jan 5, 2025