Many years and two jobs ago the company I was working for decided to use Python for game development – I talked about our experiences at the Game Developers Conference in 2002. We felt that the available Python debuggers (most of them in-process and prone to hanging) were inadequate, so we created our own, and open sourced it.
At my current job we use a lot of Python so I dusted off the debugger and updated the open source version. The updated version is up on source forge.
The debugger is called HAP which stands for the Humongous Addition to Python. It was written (mostly by Neal Josephson) while at Humongous Entertainment, and Ms. Hap was a character in one of our in-development games, so the name is an inside joke.
The debugger’s user interface was designed to be as Visual Studio compatible as possible – F5 to start or continue debugging, F10 to single-step, F9 to toggle breakpoints. It has the expected set of windows (call stack, locals, watch window, output window, etc.), a syntax highlighting editor, project files, Perforce integration, remote debugging support, and a bunch of other features that I’ve forgotten.
The screen shot below shows a bunch of the features during a typical debugging session:
There are a lot of other Python debuggers available now, including debugging extensions for Visual Studio, but I have a sentimental attraction to this one, and it works quite well.
Features to look for
You can create projects that contain multiple Python source files – handy when working on large projects – and it is through project settings that you set things like the startup directory and command line arguments.
The ‘Both’ window (Error and Output) lets you type arbitrary Python commands. I find this an excellent way to prototype regular expressions when I’m running through early versions of scripts.
Ctrl+F7 does a compile (syntax check) of the current Python file.
The default layout is missing many of the useful debugger windows. Set a breakpoint with F9, launch your script with F5, then use the View menu to show the debug windows you want. They will then show up automatically whenever you are debugging.
Hap uses sockets for its debugging protocol so the client can be on any machine. With a bit of work the client could be compiled for a game console, Linux, etc.
Known limitations and gotchas
The watch window is supposed to let you type in an arbitrary expression, but this doesn’t work. So, I add items to the watch window by selecting them in the source code and typing Shift+F9. Somebody should fix that…
The watch window also starts out with the ‘Name’ field being zero pixels wide. You need to click the left edge of the Value field and drag it to the right to expose the name field. You only have to do this once per machine and then it remembers it. Somebody should fix that…
The locals window displays variables by asking them to convert themselves to text and can hang on every single-step if you have large lists. The most common cause of this is from writing code like this:
lines = open(file).readlines()
for line in lines:
# Do stuff…
This is reasonable Python code, but the mini-hangs when processing huge files are annoying enough that I tend to use this style instead:
for line in open(file).readlines():
# Do stuff…
A few simple heuristics in the watch window code (look for large lists, dictionaries, or strings) would avoid most of these mini-hangs. Somebody should fix that…
The HAP client gets compiled to a specific version of Python – it currently references python27.dll – and using it with different versions requires a recompile. Somebody should fix that…
Bug fixes for this release
After many years of neglect Hap needed a bit of work to get it back into shape and stronger than before. Some of the main fixes include:
- Removed a requirement that the HapClient be built with the same version of VC++ as Python, caused by passing FILE pointers to Python
- Got everything to build mostly warning free with VS 2010
- Added VS 2008 and VS 2010 solution files (VS 2005 no longer supported). The VS 2010 solution is recommended for future development as the VS 2008 solution may be out of date already
- Fixed erroneous use of namespace aliases
- Fixed some const-correctness errors
- Added pre-build checks for the existence of $PYTHON_ROOT\include\python.h to make build setup easier
- Fixed the function declarations for OnNcHitTest to match the new correctness
- Added and used a HAP_MIN template function to avoid namespace problems with min and MIN
- Added _CRT_SECURE_NO_WARNINGS to the preprocessor definitions to suppress lots of warnings, which should probably be individually addressed at some point
- Got debug builds to work by changing _DEBUG to NDEBUG and specifying the release CRT. The only difference now between debug and release should be optimization. This change was needed because a true debug build of the debugger would require a debug build of Python, which is unwieldy. The P4ApiWrapper project doesn’t build in debug for some reason, but the others all do
- Rewrote makerelease.bat significantly
- Fixed bug with files not in folders not getting saved, by forcing them into folders
- Fixed an infinite loop when dragging folders to the project root
The Hap Python Debugger for Windows can be downloaded (source and binaries) from here. Give it a try and let me know what you think.
Random ASCII - tech blog of Bruce Dawson 
The mini-hangs in the Python code you mention are caused by readlines() reading the entire file up front. The style you’re using as a solution (removing the variable) doesn’t actually solve that; readlines() is still being called. Instead, you’d want to use something like this:
for line in open(file):
# Do stuff…
This will read from the file line by line.
The hangs that I see are definitely not from the reading of the lines. That is a one-time thing and I’m generally okay with it. The mini-hangs are because every time you stop in the debugger (on every breakpoint, after every single step) HAP displays all of the locals and globals. Displaying a ten million entry list takes a while.
That said, I’ll try your suggested change — it seems like a good idea.
Pingback: Bugs I Got Other Companies to Fix in 2013 | Random ASCII