Definition
Multiprocessing runs code in separate processes, each with its own Python interpreter and memory. Each has its own GIL, so heavy calculation can truly run in parallel across CPU cores. The concurrent.futures module offers a simple interface to it.
In simple words
If threads are one cook multitasking, processes are several cooks in several kitchens. They can all chop at once, but they have to carry food between kitchens, and that takes time.
Why it matters
It's how to use all your CPU cores for data processing in pure Python. PyTorch's DataLoader also uses worker processes to prepare batches.
How it works
| Threads | Processes | |
|---|---|---|
| Memory | Shared | Separate |
| Heavy Python maths | Limited by the GIL | Truly parallel |
| Start-up cost | Low | Higher |
| Sharing data | Easy, but risky | Needs copying |
from concurrent.futures import ProcessPoolExecutor
def heavy(n):
return sum(i * i for i in range(n))
if __name__ == "__main__": # required on Windows and macOS
with ProcessPoolExecutor() as pool:
print(list(pool.map(heavy, [10**6] * 4)))Common mistakes
- Using processes for tiny tasks, where the overhead outweighs the gain.
- Forgetting the __main__ guard.
Check yourself
Summary
- Processes run truly in parallel, with separate memory.
- Use them for heavy CPU work.
- Use the __main__ guard.
Finished reading?
Sources: Python docs: multiprocessing