24 Commits (23e2ca91c2b73280e84e805d04be3a768b584aee)

Author	SHA1	Message	Date
Matt Clay	379058e10f	Update collections.abc imports (#79911) ... * Update `collections.abc` imports - Use `six.moves` for modules and module_utils - Use `collections.abc` for controller code This avoids using `ansible.module_utils.common._collections_compat`, which was added before the vendored `six` was updated to provide these imports. * Update _collections_compat to use six.moves Also update the custom pylint rule to reflect this change.	3 years ago
Matt Martz	56d142350d	Add support for importlib.resources (#78915) ... * Add support for importlib.resources * Remove the importlib.resources imports * return the correct data * Some code comments, and re-order for consistency * Disallow traversing packages below an individual collection * Add a traversable class for namespaces * Re-use variable * Utilize itertools.chain.from_iterable Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua> * Simplify logic to check for packages from ansible loaders Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua> * Just a generator expression, instead of a generator * docstrings * Add comment about find_spec for our namespaces * Add some initial unit tests for importlib.resources * normalize * Utilize importlib.resources for listing collections * collections_path is already in config, just use config * install uses a different default for collections_path * Remove unused import * Remove duplicate __truediv__ * Bring back TraversableResources * Apply some small suggestions from code review Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua> Co-authored-by: Matt Davis <6775756+nitzmahone@users.noreply.github.com> * Remove cross contamination between plugin loader code and CLI code * Remove unused import Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua> Co-authored-by: Matt Davis <6775756+nitzmahone@users.noreply.github.com>	3 years ago
Brian Coca	415e08c297	Try to get best usable locale (#75033) ... specially for when you have parameters in unicode but need to scrape responses, C is still the fallback Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>	5 years ago
Sam Doran	abacf6a108	Use ArgumentSpecValidator in AnsibleModule (#73703) ... * Begin using ArgumentSpecValidator in AnsibleModule * Add check parameters to ArgumentSpecValidator Add additional parameters for specifying required and mutually exclusive parameters. Add code to the .validate() method that runs these additional checks. * Make errors related to unsupported parameters match existing behavior Update the punctuation in the message slightly to make it more readable. Add a property to ArgumentSpecValidator to hold valid parameter names. * Set default values after performining checks * FIx sanity test failure * Use correct parameters when checking sub options * Use a dict when iterating over check functions Referencing by key names makes things a bit more readable IMO. * Fix bug in comparison for sub options evaluation * Add options_context to check functions This allows the parent parameter to be added the the error message if a validation error occurs in a sub option. * Fix bug in apply_defaults behavior of sub spec validation * Accept options_conext in get_unsupported_parameters() If options_context is supplied, a tuple of parent key names of unsupported parameter will be created. This allows the full "path" to the unsupported parameter to be reported. * Build path to the unsupported parameter for error messages. * Remove unused import * Update recursive finder test * Skip if running in check mode This was done in the _check_arguments() method. That was moved to a function that has no way of calling fail_json(), so it must be done outside of validation. This is a silght change in behavior, but I believe the correct one. Previously, only unsupported parameters would cause a failure. All other checks would not be executed if the modlue did not support check mode. This would hide validation failures in check mode. * The great purge Remove all methods related to argument spec validation from AnsibleModule * Keep _name and kind in the caller and out of the validator This seems a bit awkward since this means the caller could end up with {name} and {kind} in the error message if they don't run the messages through the .format() method with name and kind parameters. * Double moustaches work I wasn't sure if they get stripped or not. Looks like they do. Neat trick. * Add changelog * Update unsupported parameter test The error message changed to include name and kind. * Remove unused import * Add better documentation for ArgumentSpecValidator class * Fix example * Few more docs fixes * Mark required and mutually exclusive attributes as private * Mark validate functions as private * Reorganize functions in validation.py * Remove unused imports in basic.py related to argument spec validation * Create errors is module_utils We have errors in lib/ansible/errors/ but those cannot be used by modules. * Update recursive finder test * Move errors to file rather than __init__.py * Change ArgumentSpecValidator.validate() interface Raise AnsibleValidationErrorMultiple on validation error which contains all AnsibleValidationError exceptions for validation failures. Return the validated parameters if validation is successful rather than True/False. Update docs and tests. * Get attribute in loop so that the attribute name can also be used as a parameter * Shorten line * Update calling code in AnsibleModule for new validator interface * Update calling code in validate_argument_spec based in new validation interface * Base custom exception class off of Exception * Call the __init__ method of the base Exception class to populate args * Ensure no_log values are always updated * Make custom exceptions more hierarchical This redefines AnsibleError from lib/ansible/errors with a different signature since that cannot be used by modules. This may be a bad idea. Maybe lib/ansible/errors should be moved to module_utils, or AnsibleError defined in this commit should use the same signature as the original. * Just go back to basing off Exception * Return ValidationResult object on successful validation Create a ValidationResult class. Return a ValidationResult from ArgumentSpecValidator.validate() when validation is successful. Update class and method docs. Update unit tests based on interface change. * Make it easier to get error objects from AnsibleValidationResultMultiple This makes the interface cleaner when getting individual error objects contained in a single AnsibleValidationResultMultiple instance. * Define custom exception for each type of validation failure These errors indicate where a validation error occured. Currently they are empty but could contain specific data for each exception type in the future. * Update tests based on (yet another) interface change * Mark several more functions as private These are all doing rather "internal" things. The ArgumentSpecValidator class is the preferred public interface. * Move warnings and deprecations to result object Rather than calling deprecate() and warn() directly, store them on the result object so the caller can decide what to do with them. * Use subclass for module arg spec validation The subclass uses global warning and deprecations feature * Fix up docs * Remove legal_inputs munging from _handle_aliases() This is done in AnsibleModule by the _set_internal_properties() method. It only makes sense to do that for an AnsibleModule instance (it should update the parameters before performing validation) and shouldn't be done by the validator. Create a private function just for getting legal inputs since that is done in a couple of places. It may make sense store that on the ValidationResult object. * Increase test coverage * Remove unnecessary conditional ci_complete * Mark warnings and deprecations as private in the ValidationResult They can be made public once we come up with a way to make them more generally useful, probably by creating cusom objects to store the data in more structure way. * Mark valid_parameter_names as private and populate it during initialization * Use a global for storing the list of additonal checks to perform This list is used by the main validate method as well as the sub spec validation.	5 years ago
Matt Davis	4c5ce5a1a9	module compat for py3.8+ controller (#73423) ... * module compat for py3.8+ controller * replaced internal usages of selinux bindings with internal ctypes binding (allows basic selinux operations from any Python interpreter), plus tests * added new respawn_module API to allow modules to import Python packages that are only available under a well-known interpreter, plus tests * added respawn logic to modules that need Python libs from a specific system interpreter (apt, apt_repository, dnf, yum) minimize internal HAVE_SELINUX usage spurious junk pep8 * pylint fixes * add RHEL8 Python 3.8 testing * more pylint * import sanity * unit tests * changelog update * fix a bunch of stuff * tweak changelog * fix setup_rpm_repo on EL8 * misc sanity/test fixes * misc feedback tweaks * fix import fallback in test module * fix selinux MU test * fix dnf tests to avoid python-dependent test packages * add trailing LFs to aliases * fix yum tests to avoid test package with Python deps * hack create_repo for EL6 to create noarch package	5 years ago
Matt Davis	c616e54a6e	refactor Python module_utils locator (#70610) ... * refactor Python module_utils locator * no longer recursive * embed special-case module code internally * share common code between collections/not cases * fixes #70134 * properly support subpackage redirection * adds support for FQCN redirect targets used by migration (expands to FQ Python name) * add tests * add changelog	6 years ago
Bob Weinand	f200487414	Fix filedescriptor out of range in select() when running commands (#65058) ... * Fix filedescriptor out of range in select() when running commands * Simplify the run_command() code Now that we're using selectors in run_command(), we can simplify some of the code. * Use fileobj.read() instead of os.read() * No longer use get_buffer_size() as we can just slurp all of the data instead. Also use a simpler conditional check of whether the selector map is empty Co-authored-by: Toshio Kuratomi <a.badger@gmail.com>	6 years ago
Sam Doran	3461c682c3	Add mechanism for storing warnings and deprecations outside of AnsibleModule (#58993) ... * Move warn() and deprecate() methods out of basic.py * Use _global_warnings and _global_deprications and create accessor functions - This lays the foundation for future functions being moved outside of AnsibleModule that need an interface to warnings and deprecations without modifying them. * Add unit tests for new warn and deprecate functions	6 years ago
Matt Clay	f510d59943	Support relative imports in AnsiballZ. (#61196)	7 years ago
Sam Doran	2d266fbc87	Fix unit tests to work with pytest >= 5.0 (#60246) ... pytest made a change in the way the message from ExceptionInfo objects was returned. https://docs.pytest.org/en/latest/changelog.html#pytest-5-0-0-2019-06-28	7 years ago
Matt Martz	2732cde031	Support using importlib on py>=3 to avoid imp deprecation (#54883) ... * Support using importlib on py>=3 to avoid imp deprecation * Add changelog fragment * importlib coverage for py3 * Ansiballz execute should use importlib too * recursive module_utils finder should utilize importlib too * don't be dumb * Fix up units * Clean up tests * Prefer importlib.util in plugin loader when available * insert the module into sys.modules * 3 before 2 for consistency * ci_complete * Address importlib.util.find_spec returning None	7 years ago
Sam Doran	ff88bd82b5	Move type checking methods out of basic.py and add unit tests (#53687) ... * Move check_type_str() out of basic.py * Move check_type_list() out of basic.py * Move safe_eval() out of basic.py * Move check_type_dict() out of basic.py * Move json importing code to common location * Move check_type_bool() out of basic.py * Move _check_type_int() out of basic.py * Move _check_type_float() out of basic.py * Move _check_type_path() out of basic.py * Move _check_type_raw() out of basic.py * Move _check_type_bytes() out of basic.py * Move _check_type_bits() out of basic.py * Create text.formatters.py Move human_to_bytes, bytes_to_human, and _lenient_lowercase out of basic.py into text.formatters.py Change references in modules to point to function at new location * Move _check_type_jsonarg() out of basic.py * Rename json related functions and put them in common.text.converters Move formatters.py to common.text.formatters.py and update references in modules. * Rework check_type_str() Add allow_conversion option to make the function more self-contained. Move the messaging back to basic.py since those error messages are more relevant to using this function in the context of AnsibleModule and not when using the function in isolation. * Add unit tests for type checking functions * Change _lenient_lowercase to lenient_lowercase per feedback	7 years ago
Sam Doran	43a44e6f35	Move utility functions out of basic.py (#51715) ... Move the following methods to lib/anisble/module_utils/common/validation.py: - _count_terms() - _check_mutually_exclusive() - _check_required_one_of() - _check_required_together() - _check_required_by() - _check_required_arguments() - _check_required_if - fail_on_missing_params() --> create check_missing_parameters()	7 years ago
Sam Doran	aba4bed803	Move _handle_no_log_values() out of basic.py (#48628) ... * Rename method and make private * Use is_iterable, combine transformations * Remove unused return_values from network modules * Improve docstrings in new functions * Add new PASS_VAR * Add unit tests for list_no_log_values * Fix unit tests for Python 2.6	7 years ago
Sam Doran	2a98faee2b	Move _handle_aliases() out of basic.py (#48578) ... Refinements: - return legal_inputs and update class properties - remove redundant arguments from method and handle in caller - add better exception types to method * Add unit tests for handle_aliases	7 years ago
Abhijeet Kasurde	bc6cd13874	Handle exception raised in recursive_finder API (#49590) ... User module can contain Indentation errors or syntax errors. Handle AST exceptions rather than showing traceback while importing such module. Fixes: #21707 Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>	7 years ago
Toshio Kuratomi	5844c8c7f0	Cleanups to the common.sys_info API ... * Move get_all_subclasses out of sys_info as it is unrelated to system information. * get_all_subclasses now returns a set() instead of a list. * Don't port get_platform to sys_info as it is deprecated. Code using the common API should just use platform.system() directly. * Rename load_platform_subclass() to get_platform_subclass and do not instantiate the rturned class. * Test the compat shims in module_utils/basic.py separately from the new API in module_utils/common/sys_info.py and module_utils/common/_utils.py	7 years ago
Toshio Kuratomi	61b1daa65f	Port from plaform.dist to ansible.module_utils.distro.linux_distribution ... ci_complete	7 years ago
Andreas Calminder	876b637208	move some of basic into common (#48078) ... * move file functions into common.file * move *_PERM_BITS and mark as private (_*_PERM_BITS) * move get_{platform, distribution, distribution_version} get_all_subclasses and load_platform_subclass into common.sys_info * forgot get_distribution_version, properly rename get_all_subclasses * add common/sys_info.py to recursive finder test * update module paths in test_platform_distribution.py * update docstrings, _get_all_subclasses -> get_all_subclasses * forgot to update names * remove trailing whitespace	7 years ago
Toshio Kuratomi	8ec973b453	Fix unittests for always adding basic to the AnsiBallZ zips	8 years ago
Toshio Kuratomi	9f7b0dfc30	Remove automatic use of system six ... * Enable the pylint no-name-in-module check. Checks that identifiers in imports actually exist. When we do this, we also have to ignore _MovedItems used in our bundled six. This means pylint won't check for bad imports below ansible.module_utils.six.moves but that's something that pylint punts on with a system copy of six so this is still an improvement. * Remove automatic use of system six. The exec in the six code which tried to use a system library if available destroyed pylint's ability to check for imports of identifiers which did not exist (the no-name-in-module check). That test is important enough that we should sacrifice the bundling detection in favour of the test. Distributions that want to unbundle six can replace the bundled six in ansible/module_utils/six/__init__.py to unbundle. however, be aware that six is tricky to unbundle. They may want to base their efforts off the code we were using: 2fff690caa/lib/ansible/module_utils/six/__init__.py * Update tests for new location of bundled six Several code-smell tests whitelist the bundled six library. Update the path to the library so that they work. * Also check for basestring in modules as the enabled pylint tests will also point out basestring usage for us.	9 years ago
Dag Wieers	4efec414e7	test/: PEP8 compliancy (#24803) ... * test/: PEP8 compliancy - Make PEP8 compliant * Python3 chokes on casting int to bytes (#24952) But if we tell the formatter that the var is a number, it works	9 years ago
Toshio Kuratomi	2fff690caa	Update module_utils.six to latest (#22855) ... * Update module_utils.six to latest We've been held back on the version of six we could use on the module side to 1.4.x because of python-2.4 compatibility. Now that our minimum is Python-2.6, we can update to the latest version of six in module_utils and get rid of the second copy in lib/ansible/compat.	9 years ago
Toshio Kuratomi	5c38f3cea2	Combine jimi-c and bcoca's ideas and work on hooking module-utils into PluginLoader. ... This version just gets the relevant paths from PluginLoader and then uses the existing imp.find_plugin() calls in the AnsiballZ code to load the proper module_utils. Modify PluginLoader to optionally omit subdirectories (module_utils needs to operate on top level dirs, not on subdirs because it has a hierarchical namespace whereas all other plugins use a flat namespace). Rename snippet* variables to module_utils* Add a small number of unittests for recursive_finder Add a larger number of integration tests to demonstrate that module_utils is working. Whitelist module-style shebang in test target library dirs Prefix module_data variable with b_ to be clear that it holds bytes data	9 years ago