Hi everyone,
I’m looking for some feedback from you all on static analysis tools today and whether you can implement these tools without breaking the bank. If you follow me on Twitter you might have noticed that I’ve started a vendor selection process for a static analysis tool to compliment our manual processes here at Realex Payments. I know these tools aren’t cheap but I had a budget in mind which I thought would get us a solution that meets most of our requirements – boy was I wrong! I’m being realistic with our requirements, I can’t justify the cost of implementing an all singing all dancing, integrated into “everything” static and dynamic analysis solution but the figures I had in my mind would make the static analysis solution the largest application security expense for the company after application security resource salaries.
Are commercial static analysis solutions only affordable by consultancies and large software companies and are they actually worth that much money? I’m not trying to cause an argument or a flame war; I’m genuinely interested in people’s opinions on this.
We have a very strong SDLC with security ingrained throughout and this includes the use of some open source/free static analysis tools such as Findbugs, RIPS and CodePro AnalytiX to compliment our manual reviews. We also use in house tools such as Agnitio and have a commitment to conduct manual reviews of all code changes before they are deployed into production and we are currently trying to hire another application security analyst to ensure the manual review process continues to be executed twice per project.
So now I’m looking for opinions from you all on what you have done from a static analysis point of view. What has worked for you? Are these commercial tools genuinely worth the money and do the downsides (false positive for example) outweigh the positives in your experience? What vendors have you used and would you recommend them to another company, if yes why? What do we gain from buying an expensive commercial solution versus buying cheaper solutions such as Klocwork Solo to use in conjunction with the tools mentioned above and expanding in house tools such as Agnitio to include automated code crawling capabilities?
On a slightly different note I wonder if the cost of these application security solutions has a bearing on the low application security spend pointed out by Gunnar Peterson and Jeremiah Grossman recently? If I’m a manager/CxO looking at the cost of these solutions I’d be expecting miracles from them, in fact I’d probably point to the (much) lower cost of our network security solutions and ask why there is such a big difference! If the manager/CxO can hire 4 or 5 members of staff for the cost of an application security solution guess where the money is going to go……
I’m looking forward to hearing what you all think about this!
SN
I can’t comment too much, except to say that I’m in the same boat as you. When it came to the crunch, the prices offered by the big guys was just way too much and we couldn’t justify it. In some cases it came to almost 50-75% of another full-time employee! In addition, when we ran extensive in-house trials WITH specialists from the vendors, they failed to deliver time after time with our apps.
So I guess I *can* comment. At this point in time, and the exorbitant prices from the vendors, I can’t imagine anyone using these systems, except if this is the only service they do, or they’re a HUGE software shop. I’m keen to see what other people say.
Thanks for sharing SN!
Hi SN,
Just thought I’d plug my own low tech solution, Graudit. It’s a slightly different approach, but if you take the time to learn it and apply more advanced techniques shown in the aux scripts it can be quite effective.
You can grab it from my website.
In all honesty, static source code analysis is only a part solution. It will not find compiler optimization bugs, logic flaws and a number of other issues. Dynamic invocation, lazy loading and proper taint analysis is only some of the issues you should still manually check for. A well trained employee with a wider tool and skill set is perhaps a much wiser investment than another automated scanner. Even if the false positives/false negatives or finding accuracy differs with a more expensive tool.
Just my $0.02
~Wireghoul
I’m in the same boat of you, and looking foward to see the comments posted here.
Take a look at the following paper. Maybe it can help you.
http://www.informit.com/articles/article.aspx?p=1680863
Cheers.
Hi Fabricio,
Thanks for the comment and the link, I’m interested to see what people think on this subject.
I have had a few comments here, some on Twitter and some via email and to be honest none of them are saying this blog is incorrect.
SN
Hi Wireghoul,
Thanks for the tool suggestion and the comment, I’m going to look at that tool today as it covers a few of the languages we use. I will give you some feedback once I’ve tried it out
I completely agree with what you are saying about it only being one part of a solution which includes a well defined SDLC, manual reviews, security testing and so on. It just happens to be a small part of the overall solution which comes with a huge price tag and (based on responses I’ve received so far) very few positive reviews.
SN
Hi Christian,
Thanks for comment, I’m glad to see that I’m not the only one thinking this way. The comments here, on Twitter and via email have all been very useful.
I’m not saying static analysis tools are rubbish but I’m certainly getting the impression that everyone thinks the price tags attached to them are way to big compared with the performance/results you get.
SN
Ahhh WG! I’ve been meaning to have a better look at GRAudit and you’ve just reminded me! Next week! Cheers,
Christian “xntrik” Frichot
Hi Christian!
At work we thought about purchasing a security static analysis solution, finally we understimate it because of expenses. There are cheaper options such as Checkmarx, and you can purchase one of those expensive solutions just for a specific project (the licensing types usually include that option). As well as this, quality assurance tools could help a bit, they have some security checks and tools to help the manual reviewer (such as a data flow diagram creator), so maybe you could join efforts with your software quality testing team.
Just my 2 cents. By the way, thank you for your blog, I’ve been following it for a while
Hi Des,
Thanks for the compliment and thank you for being a follower and commenter on the blog!
I think the comments I’ve received on this blog, Twitter and via email have made me realise that you can’t implement static analysis without breaking the bank. To be more specific you can’t implement it in a way which really makes it worth its price tag, cut down solutions, once off solutions etc don’t really lend to a strong and consistent approach to reviews.
SN
You don’t have enough source code and apps to scan if it doesn’t make monetary sense. In this situation, it’s easy: go with an application security consulting company and let them use the security-focused static analysis tools.
If you look at the numbers, it costs $1.5K per app per week for a simple Ounce license — which is $78K per year — about the same cost as a single Fortify auditor license. Appsec consulting companies (usually partnered with HP, IBM, Armorize, or Checkmarx) have at least one person on their team using these tools — and near constantly e.g. a few apps every week.
If your organization works at that pace, then these tools start to make sense for webapp projects. Different verticals require different tools — for example Klocwork is very focused on embedded systems by comparison. Checkmarx is heavily tailored to cloud-PaaS or custom languages in multi-tier environments. Fortify is the kitchen-sink for frameworks, especially managed code language support. An ISV with a very critical embedded systems research project is probably going to go with Klocwork, Coverity, or another static analysis tool that is even more focused on quality.
What I think you are completely missing in this post is the cost of training up to and implementing a static analysis tool, especially for security-related purposes. This is why you end up with a $3M bill to Cigital after the first year — and the whole time with minimal results and serious consequences to the dev timelines.
Some application security consulting shops don’t even concentrate heavily on security-focused static analysis — instead operating on the underlying principles of OWASP ESAPI and OWASP ASVS. Understanding your application security program and how the people and processes are working is essential. Tools are not essential at all. You can setup Yasca and prepare for analysis — get your code metrics going. Get build servers and the like. Get continuous integration and continuous deployment going where they need to go — with some solid release management. Understand your technical and software debts. There is tons of work you can do to prepare for the eventual security-focused static code scan — so do it.
The price for these tools is absolutely crippling, and after a few conversations with a couple of different vendors, I was left with a bad taste in my mouth. I am under the impression (and in agreement with Andre) that most of these tools are priced to facilitate partner relationships with 3rd party consultants as opposed to actual tools to integrate into the SDLC.
The value that these tools can bring to the table is huge, but until they become affordable, they are pretty much relegated to fringe status. The good news is that if you have progressed your app-sec practice to the point where the next logical step is adopting these tools, you are in much better shape than alot of other companies in the financial services space!
Hi Andre,
Thanks for the great comment, I think we have done almost everything you have listed around build servers, release management, metrics etc and now looking at what other improvements we can make to our SDLC.
SN
Hi Yvan,
Thanks for the comment, almost everyone has given the same kind of opinion on this blog post so that has made things a lot clearer for me
SN
Just as an aside.. we pretty much need a Burp Pro equivalent for Static Analysis – awesome, powerful in the right hands, and completely affordable!
Saw your post pop up in my twitter feed. I work for one of the big vendors in the space (IBM), specifically as a static analysis specialist. Figured I’d post some of my personal thoughts on adopting static analysis in general but if you want to talk specifics I’m happy to do that too, or put you in touch with one of my colleagues in Dublin if you want to speak to someone local.
First off, any company that doesn’t have a well developed security practice would probably do better with a combined push to implement security basics in the SDLC and black box testing (automated or manual) to light a fire to get things moving.
Sounds like you’re well beyond that, though, at which point it comes down to the number of applications you have and the problems you’re having reviewing them. I’d say that there are two major value based reasons people purchase static analysis tools (we’ll leave out the ones that aren’t directly value based like meeting compliance requirements or just generally wanting to be more secure, perfectly valid they may be).
The first would be to better utilize existing audit resources. I’ve done fully manual reviews in my days as a security consultant and these days it’s semi automated or fully automated (depending on coverage) and I would say that my coverage, speed, and results are vastly improved. One of the major reasons for that is that a lot of the applications I’m reviewing have had a focus on security for some time (established secure coding practices, design review, penetration tests, etc) so the vulnerabilities present in the application are not endemic, they won’t be immediately present through an architecture review or taking a look at some of the data flows. So, to find the vulnerabilities, you’re either looking through every entry point to its eventual destination, something that is a major slog without an automated tool, or you’re using an automated tool to speed the process along. Beyond that, offloading everything easily automatable leaves more time for business logic review, architecture review, etc.
The second reason is to improve remediation times. It looks like you can see this argument in plenty of different parts of the industry now, I know I’ve seen it from an article on InformIT by Gary McGraw, and I’ve seen it from competitive companies as well, but the gist is that it’s a lot easier for developers to fix an issue if they can look at the code itself rather than a URL and an attack. Getting information on the vulnerability doesn’t hurt either since outside of the security team the knowledge about particular vulnerabilities tends to be very shallow.
Oh, and a third reason that comes out after the static analysis tool has been in use for a bit, repeatable scans and time based reporting. Automating the process means that the work you put into understanding an application continues to get leveraged into the future and issues are found during development rather than in QA, during user acceptance, or as I used to see often, right before production.
Most of these solutions are being targeted to the enterprise – this is why HP and IBM have bought in recently – enterprises are where the money is. Prices continue to go up, so now commercial static analysis tools are out of the range of most SMBs and small software shops, except for maybe a small number of shops with specialist high-risk requirements who are willing to pay. Small shops are left to work with the free stuff, which doesn’t have a lot of momentum behind it (for example, Findbugs hasn’t been updated in a year and a half, now that the project lead is tied up working with Google, and it has never had much in the way of security checking).
Our commercial vendor just significantly changed their pricing at the end of last year, abandoning the SMB space, so we had to cut a deal with another vendor – you can always get a deal when you are in a competitive win back situation, but it probably won’t last long, I’ll probably have to go looking for something else next year or stick with the out of date freeware and maybe pick up some more Klocwork Solo licenses.
The real customers for this stuff is the enterprise: the big banks and insurance companies who need a checkmark and are used to paying big system integration fees to IBM and HP and Accenture for packaged solutions with training and consulting help. These companies will pay a lot, but they don’t pay list software price anyways: the software licenses are just a small part of the overall cost of the solution they are getting, and they will get dinged later with high maintenance fees to make up the difference.
I run a small shop. I don’t need the dashboards and integration with other tools and value-add modeling tools and application portfolio management and whatever else, and I don’t need training and consulting – we take care of all of that ourselves. All of this costs extra and what’s important is getting programmers to run the checks and take them seriously. What’s worked for me is to push it down to the individual level: get senior people to take it seriously, get people to use the tools and follow-up. Start with the high-risk code and focus on high-risk and clear findings and work from there.
My advice would be if you can, start small and build up. Stick with smaller and simpler tools if you can find them and get people to use them. The returns will be modest, but that’s all that you will get out of this technology anyways.