PyCon 2019 | Statistical Profiling (and other fun with the sys module)
Speaker: Emin Martinian
Profiling involves computing a set of data about how often and how long various parts of your program are executed. Profiling is useful to understand what makes your program slow and how you can improve it. After a quick review of deterministic profiling tools and techniques, I will describe how you can do statistical profiling with existing packages or write your own from scratch.
Statistical profiling involves occasionally sampling what your program is doing instead of watching each line or function. A key feature of statistical profiling is that by using a moderate sampling frequency, you can profile your production code with almost no overhead. This lets you find the actual bottlenecks in real use cases.
The core technical focus of the talk is python’s sys module and how it lets you easily examine a running program. I also describe some tricks to be aware of related to threading, context switches, locks, and so on. At the conclusion of the talk, you will hopefully understand how to use an existing statistical profiler or write a customized version yourself.
Slides can be found at: https://speakerdeck.com/pycon2019 and https://github.com/PyCon/2019-slides
