What happens behind execution of an `import` statement?
Very often we in our daily usage of Python, we face errors related to $PATH, $PYTHONPATH, or other import related errors. A clear understanding of the import system helps us to fix these errors properly, and build extensions on top of the import system.
This talk follows this template to explain what goes on in the background when you execute
from spam import ham, or any other import statement. Additionally, it discusses import hooks(PEP-302) and how to use them.
Every Python script involves import statements. And if you develop software with Python, you might already have come across import and path related errors like ImportError, ModuleNotFoundError, etc.
It's important to know the import mechanics, both for general knowledge, and to develop custom plugins to tweak the import system. In this talk, we'd discuss in detail about how the import system works, its components and what tasks they perform respectively.
Who should attend this talk?
This talk is most suitable for beginner to intermediate level Pythonistas.
How does the talk proceed?
- The talk starts with packages in Python, and how to organize Python files in order to make them distributable, and importable.
- It proceeds to discuss about $PATH and $PYTHONPATH, and how they are used to enable system-wide imports, and how capable are Python's path based import handlers, if they let a programmer import from remote URIs, etc.
- Different types of packages such as regular packages, namespace packages, frozen modules, etc. are discussed.
After that, some key components are introduced i.e sys.modules and how modules are refreshed and regulated in runtime, what finders and loaders are, and what 'import protocol' is, sys.meta_path, sys.path_hooks.
The talk then goes in detail about finders, the default finders in Python, spec object, and its functioning. After finders, loaders are discussed with sample code, explaining the steps involved from processing a spec, and creating, executing, and loading a module to sys.modules.
- After finders and loaders, 'import hooks' are discussed i.e how Python allows injection of custom components to handle a pre-defined import statement.
- This follows a demonstration of import hooks, implementation of a meta_path finder Class which protects importing of modules from an http server with a token exchange. (So the source code is protected, and it can be logged who requested the source code by a signature to act against misuse.)
- The demonstration is followed by listing some important use cases of import hooks, and how folks have used it in the past.
What can someone get out of this talk?
An understanding of the import system's mechanics, which helps to quickly debug and fix path/import related errors. Knowledge of import hooks, to build custom import related plugins.
Time Duration - Topic
0 - 4 Introduction to packages and how to organize packages
4 - 7 Types of packages in python, and the default ones the import system supports.
7 - 12 Components of import system - sys.modules - finders and loaders - chronology of import related tasks
12 - 15 Finders, examples of finders, and how they function, find_spec function
15 - 17 sys.meta_path, sys.path_hooks in detail
17 - 19 Introduction to PEP 302, and import hooks
19 - 22 explain loaders with template code
23 - 25 code walkthrough and demonstration of an implementation of import hooks
25 - 26 briefly cover 'importing of submodules' topic with an example
26 - 28 present use-cases of import hooks, and usage in industry, conclude.
28 - 30 Q&A
- Syntactic knowledge of Python
- Should have come across import/$PYTHONPATH related errors
If you don't know either and still like the talk idea, please follow this gist to 'intentionally' come across the errors :)
Shivashis Padhi is a senior, majoring Computer Science and Engineering at National Institute of Technology, Tiruchirappalli, India.
With ~3 years of experience as a student software developer he has worked with multiple organizations(Python Software Foundation - GSoC'20, Delta Force), large and small scale startups(Grofers, Flytbase Labs, Gmetri), he lives by a simple policy, 'learn and build stuff to make the world a better place'. He's currently an intern with the Data Engineering team at Grofers building geospatial data-management tools. He spends some time contributing to MSS with an amazing set of people.
He also gave his first ever technical talk, about maps and Python during PyCon India'19.
Keeping work aside, he's a big fan of earth, computers, maps, classical music from different cultures, and history.