Definition
Profiling measures where a program actually spends its time, so you can speed up the right part. timeit measures small snippets, and cProfile reports the time spent in each function. Caching, better data structures and vectorising are the usual fixes.
In simple words
Don't guess why your app is slow, measure it. Often 90% of the time goes into one small loop. Fixing anything else gains you nothing.
Why it matters
Programmers are famously bad at guessing where time goes. In data work, swapping a Python loop for a NumPy operation can make something 100 times faster.
How it works
- Make it correct first.
- Measure, to find the slow part.
- Choose better tools: a set in place of a list for lookups, or vectorising.
- Cache repeated work.
- Measure again.
In code
import timeit
items_list = list(range(10000))
items_set = set(items_list)
list_time = timeit.timeit(lambda: 9999 in items_list, number=1000)
set_time = timeit.timeit(lambda: 9999 in items_set, number=1000)
print("Set lookup is faster:", set_time < list_time)
import numpy as np
numbers = list(range(100000))
array = np.array(numbers)
loop_time = timeit.timeit(lambda: [n * 2 for n in numbers], number=10)
numpy_time = timeit.timeit(lambda: array * 2, number=10)
print("NumPy is faster:", numpy_time < loop_time)Output
Set lookup is faster: True NumPy is faster: True
Common mistakes
- Optimising before measuring.
- Sacrificing readability for a tiny gain in a part that rarely runs.
Check yourself
Summary
- Measure first, and fix the slow part.
- Better data structures and vectorising give the biggest gains.
Finished reading?
Sources: Python docs: The Python Profilers