Version 1 of WASSEC (Web Application Security Scanner Evaluation Criteria) is (finally) out! I'm not going to say which section I wrote, but the document is (as far as I know) the first attempt to comprehensively list the features that should be considered when evaluating appsec scanning tools. Check it out. It's worth a read.
The WASSEC document be found here in both wiki and PDF formats: