What happens behind execution of an `import` statement?

Introduction

Every Python script involves import statements. And if you develop software with Python, you might already have come across import and path related errors like ImportError, ModuleNotFoundError, etc.

It's important to know the import mechanics, both for general knowledge, and to develop custom plugins to tweak the import system. In this talk, we'd discuss in detail about how the import system works, its components and what tasks they perform respectively.

Who should attend this talk?

This talk is most suitable for beginner to intermediate level Pythonistas.

How does the talk proceed?

• The talk starts with packages in Python, and how to organize Python files in order to make them distributable, and importable.
• It proceeds to discuss about $PATH and $PYTHONPATH, and how they are used to enable system-wide imports, and how capable are Python's path based import handlers, if they let a programmer import from remote URIs, etc.
• Different types of packages such as regular packages, namespace packages, frozen modules, etc. are discussed.

• After that, some key components are introduced i.e sys.modules and how modules are refreshed and regulated in runtime, what finders and loaders are, and what 'import protocol' is, sys.meta_path, sys.path_hooks.

• The talk then goes in detail about finders, the default finders in Python, spec object, and its functioning. After finders, loaders are discussed with sample code, explaining the steps involved from processing a spec, and creating, executing, and loading a module to sys.modules.

• After finders and loaders, 'import hooks' are discussed i.e how Python allows injection of custom components to handle a pre-defined import statement.
• This follows a demonstration of import hooks, implementation of a meta_path finder Class which protects importing of modules from an http server with a token exchange. (So the source code is protected, and it can be logged who requested the source code by a signature to act against misuse.)
• The demonstration is followed by listing some important use cases of import hooks, and how folks have used it in the past.

What can someone get out of this talk?

An understanding of the import system's mechanics, which helps to quickly debug and fix path/import related errors. Knowledge of import hooks, to build custom import related plugins.

Outline

Time Duration - Topic

0 - 4 Introduction to packages and how to organize packages

4 - 7 Types of packages in python, and the default ones the import system supports.

7 - 12 Components of import system - sys.modules - finders and loaders - chronology of import related tasks

12 - 15 Finders, examples of finders, and how they function, find_spec function

15 - 17 sys.meta_path, sys.path_hooks in detail

17 - 19 Introduction to PEP 302, and import hooks

19 - 22 explain loaders with template code

23 - 25 code walkthrough and demonstration of an implementation of import hooks

25 - 26 briefly cover 'importing of submodules' topic with an example

26 - 28 present use-cases of import hooks, and usage in industry, conclude.

28 - 30 Q&A

Prerequisites:

• Syntactic knowledge of Python
• Should have come across import/$PYTHONPATH related errors

If you don't know either and still like the talk idea, please follow this gist to 'intentionally' come across the errors :)

Content URLs

WIP slides

Longer description of the talk idea as blogpost