... .  .    .       .joeberkovitz.com

Finally, a Working AS3 Code Coverage Technique!

March 29, 2008 on 7:37 am | In Flex, Programming |

[Since this was posted, an initial version of the coverage tool has been put up on Google Code which can be accessed here.]

I’ve been pining for an AS3 code coverage tool for so long, I feel sick. I hate not being able to know how much of my code is actually exercised by test cases, whether automated or manual. So I’ve been playing with an idea on how to get AS3 code coverage working for the last few days and, somewhat to my own amazement, it actually works! Thanks to Adobe’s release of the compiler as an open source distribution, I’ve been able to hack mxmlc to instrument each line of code, adding a call to a global function that logs the line number and the function name.

I invoke the modified compiler on my test program like this (note the additional library for coverage support)

mxmlc.exe -library-path+=../fcov/bin/fcov.swc -coverage src/CoverageTest.mxml
               -output bin/CoverageTest.swf

This generates a special debug version of the SWF that calls a special function in the fcov.swc library. As you can imagine, this piggybacks on the same compiler feature which supports breakpoints. However, the debugger is not involved in any way; there’s no need to run it.

The crude trace log looks like this:

[SWF] C:\work\eclipse\sandboxFlex3\CoverageTest\bin-debug\CoverageTest.swf
       - 575,011 bytes after decompression
coverage:  CoverageTest/CoverageTest 2
coverage:  CoverageTest/bar/get 7
coverage:  CoverageTest/bar/get 7
coverage:  CoverageTest/bar/get 7
coverage:  CoverageTest/___CoverageTest_Button2_click 17
coverage:  CoverageTest/___CoverageTest_Button2_click 17
coverage:  pkg:TestClass/TestClass 5
coverage:  pkg:TestClass/TestClass 6
coverage:  pkg:TestClass/TestClass 7
coverage:  pkg:TestClass/testFunction 8
coverage:  pkg:TestClass/testFunction 10
coverage:  pkg:TestClass/testFunction 11
coverage:  CoverageTest/___CoverageTest_Button2_click 17
coverage:  CoverageTest/___CoverageTest_Button3_click 18
coverage:  CoverageTest/___CoverageTest_Button3_click 18
coverage:  CoverageTest/testFunction 9
coverage:  CoverageTest/testFunction 11
coverage:  CoverageTest/testFunction 12
coverage:  CoverageTest/___CoverageTest_Button3_click 18
coverage:  CoverageTest/___CoverageTest_Button3_click 18
coverage:  CoverageTest/___CoverageTest_Button3_click 18
coverage:  CoverageTest/testFunction 9
coverage:  CoverageTest/testFunction 11
coverage:  CoverageTest/testFunction 12
coverage:  CoverageTest/___CoverageTest_Button3_click 18
coverage:  CoverageTest/___CoverageTest_Button1_click 16
coverage:  CoverageTest/___CoverageTest_Button1_click 16
coverage:  CoverageTest/bar/set 7
coverage:  CoverageTest/bar/get 7
coverage:  CoverageTest/bar/get 7
coverage:  CoverageTest/___CoverageTest_Button1_click 16

The coverage log includes correct line numbers and function names from both MXML and AS, including synthetic functions created by the MXML compiler for things like event scripts and Bindable variables.

This stuff is still under development so I’m not releasing it yet — the only way to distribute it cleanly is to pick up my entire custom Flex 3 SDK distribution, or to overlay my source patches on top of a distro that you check out from opensource.adobe.com, so that needs to be figured out.

And, of course, mere trace log output does not a good coverage tool make. My next steps are:

• Generate metadata that describes, independently of the execution log, what line numbers actually are executable in the source. Without this, one has no idea what fraction of the possible lines were actually run.
• Buffer the coverage log in an in-memory data structure and write it out in single chunks when desired, otherwise performance will be hammered.
• Create a simple command-line tool to interpret the coverage data and produce reports, probably just a Java program.
• Create a nice AIR-based tool for browsing the data, along the lines of ItDepends (and probably using a lot of the same code).

In case you are interested, here are the Flex 3 SDK diffs that were needed to accomplish this.