Being able to profile your program's memory usage is yet another key skill that most senior developers will need in order to effectively debug issues with their systems. I've often seen developers pay little or no attention to the limitations of the hardware that they are developing on--I've also been one of the biggest perpetrators of this heinous crime.
I've loaded a server with a jenkins instance along with multiple heavy JVM-based systems as well as multiple cron jobs and whatever else I could fit in there, and seen it all come tumbling down after one particularly memory hungry program gobbled up all of the remaining memory available on that server. The main cause of the issue? I'd not paid attention to appropriate garbage collection within one of my applications, and over time it started throwing exceptions and dying on me slowly.
It was only once I'd been burned by this kind of issue that I started to seriously analyze the memory usage of the programs that I create. Within the Python ecosystem, we have a tool aptly called memory_profiler, which, very much like our line_profiler, generates informative tables based on the memory usage of each of the lines of our explicitly chosen functions.
In order to install the memory profiler, run the following command:
pip install -U memory_profiler
Again, we'll take the following program as our test bed:
import random
import time
@profile
def slowFunction():
time.sleep(random.randint(1,5))
print("Slow Function Executed")
def fastFunction():
print("Fast Function Executed")
def main():
slowFunction()
fastFunction()
if __name__ == '__main__':
main()
Thankfully, we are able to reuse the @profile annotation when it comes to memory profiling our slower function. In order to perform a memory profile, we can execute this:
$ python3.6 -m memory_profiler profileTest.py
Upon execution of this, you should then see a table, much the same layout as our line_profiler display in the console. This gives us the line numbers of the code that we are actively profiling, their associated memory usage, and the amount by which memory usage incremented when executing that line.
As you can see, our slowFunction in this particular program isn't exactly too demanding on our system, and thus, no real action needs to be taken.
Line # Mem usage Increment Line Contents
================================================
4 33.766 MiB 0.000 MiB @profile
5 def slowFunction():
6 33.777 MiB 0.012 MiB time.sleep(random.randint(1,5))
7 33.785 MiB 0.008 MiB print("Slow Function Executed")
