Security research, news and guidance

What I’ve learned from (nearly!) a year of open source

July 14, 2011  |  Written by Security Ninja  |   Application Security, Ninja News and Updates   |   26 Comments

Hi everyone,

As the days count down until I jet off to Las Vegas and release v2.0 of Agnitio I’ve been thinking about the past year and the lessons/things I’ve learned since I first released Agnitio – my first open source project.

Whilst Agnitio v1.0 was only released 7 months ago I announced that I was writing the tool at SecurityBSides Las Vegas last year and that it would be “free for anyone to download and use” (see slides 49, 50, 51 and 53 here). That particular statement makes me think of one thing in particular I learned and I will cover later in this post.

People will try to put you and your work down

This is the biggest and hardest lesson I probably learned over the past year. Most people who have used Agnitio have praised and enjoyed the application but there has of course been a small amount of people who provide nothing but negative feedback and trouble. The negative feedback and trouble has come from people in a wide range of positions in our industry including people high up in organisations such as OWASP who should be encouraging projects like this not trying to put them down. If you really want to see the personal agendas of some application security people release an open source application security tool! It showed me that we have too many people in application security who actually have no real desire to move application security forward, they gain too much from trotting out the same old lines and references and they don’t want anything to come along and disrupt that.

Let me give you some specific examples, first of all there was one particular person who appeared to want the Agnitio project banned and removed from Source Forge. He had no real reason other than I’d been slower than he wanted me to be in uploading my source code as well as the installer. This escalated to this person contacting Source Forge and reporting me for a violation of the TOS, I have to say this process was a pain in the backside to go through. If I hadn’t accidentally deleted the emails when I did a cleanup recently I would publish them to show you how it unfolded.

Another example was a blogger who praised Agnitio on his own blog and linked to an Angitio presentation on this site. I was then surprised to find this person then wrote comments which says Agnitio is a simple little database app, hard to use and saying I used the term “static analysis” (“analysing source code without executing the application” is exactly what Agnitio helps with but that was a different days discussion) just to attract people to the project and my blog. I won’t publish the private conversations I had with this person but the comments are public and on this website if you want to read them.

Comments on other blogs have also been quite funny and frustrating at the same time. Colin Watson is a fan of Agnitio and he wrote a blog post when v1.0 was released. You can see two comments on his blog which were certainly not posted to praise Agnitio. You will also see I tried to address both of the people who posted comments but unsurprisingly they never commented again or contacted me via Twitter, email etc. That is something I’d certainly say is one of my biggest frustrations so far, people give negative feedback but never elaborate on it when I’ve asked them to. If you think Agnitio is rubbish or lacking something I can’t make it better without good feedback!

Ultimately you have to be ready for people calling your baby ugly, it doesn’t matter how many hours you work on your project it will be called ugly. It hurts, it might make you angry but that’s life and I try to extract positives from these kinds of situations. I try to get more specific feedback and take the “customer is always right” stance,  sometimes this works out well and other times you get nowhere.

Know your license type and TOS well!

Open source licenses are something I’d never really paid much attention to and now I’ve picked one for Agnitio I hope to avoid reading about different license types for the rest of my life. It is important to understand which license is right for your project but I struggled to find any good guides for open source licensing n00bs. I did a lot of reading and narrowed it down to a couple of different licenses but ultimately I went with one of the most popular licenses on Source Forge: GPLv3.

As I mentioned above I had problems relating to one part of the Source Forge TOS which I’m happy to admit was my own fault. I failed to properly understand my obligations for hosting an application on Source Forge. This was the first open source application I’d released and I was naïve, it isn’t as simple as throwing an installer and some source code up on to Source Forge. You have to do these things in a particular way otherwise you get reported and go through a process where you aren’t really helped but expected to just understand and fix the thing you have been reported for.

I mentioned above that I wanted to make Agnitio “free for anyone to download and use” but things aren’t as simple as that. I’m not trying to blame anyone else here, I failed to understand what license was best for Agnitio and what exactly I needed to uploaded and host on Source Forge. This caused me grief that I could have avoided if I had read and understood the TOS. Then again maybe it wouldn’t have changed anything, even when the Source Forge rep told me the specific point I’d violated I still had to push them to be more specific before I could sort the problem out.

Test, test and test again

My respect for QA testers increased a lot over the past year after I started to try and draw up test plans and execute them for each Agnitio release. I’ve always taken the approach of testing as I develop and then I try to test “everything” once I’ve finished. I’m convinced this approach has prevented a lot of bugs getting into the final release of Agnitio but that doesn’t mean I’ve caught everything. There is one bug that I introduced in v1.0 that no one found until v1.2 had been released. The bug occurred when the user went to the verify report tab and clicked verify without browsing to report first, I hadn’t included a check to see whether the path was empty before trying to open a file:

If you are using v1.0, v1.1 or v1.2 you can try that yourself if you want to crash Agnitio :)

I had only tested whether this part of Agnitio functioned as I expected it to or how I expected a user to use the verify report function. If you are releasing your own open source application I’d encourage you engage with any QA people you know and learn how they approach testing, I know this helped me change how I tested Agnitio and hopefully find silly issues like the one above in future.

The other testing lesson I learned the hard way was that Agnitio would not only be used on English versions of Windows. Agnitio has been downloaded by people in over 90 countries which meant some users had problems using parts of it because I never tested it on a non English version of Windows. I had regexs which were wrong and file paths in the code in English which caused users problems. One example that sticks out in my mind is a regex I had written to validate the date field on the code review tab. I wrote this based on what I could see Agnitio creating when the calendar popup was used, a date value with numbers and / in it:

(!Regex.IsMatch(Date.Text, @”^[0-9/\b\s]*$”))

This caused users of some versions of Windows problems; the calendar popup takes its date format from the Windows date/time format. So some users had a date field with Jan/Feb/Mar and so on in it and not the month number that I expected.

I love open source development and the community support

I must say that I’m delighted with how Agnitio has been received and used, constructively criticised and improved over the past year. The negative people or incidents are the ones that will always jump out in my mind first but for every one of those there have been many, many more positive interactions.

I now have even more respect for the open source community and its contributors than I had a year ago. The most encouraging sign to me has been seeing how others are willing to give up some of their free time to try and make Agnitio better. I have had individuals give lots of useful feedback, contribute their own time, perform QA testing, translate Agnitio content and even offer some the use of their employees time to help Agnitio progress. I cannot thank those people enough for their support and help but Steven van der Baan, Ryan Dewhurst, Daniel Cornell, Angel Alonso, Tiago Henriques, Jim Bird, Fabricio Braz, Colin Watson and Nabil Ouchn have been stars in contributing to and supporting Agnitio!

A challenge to other application security people

I’ve said many times before I’m not a developer, I’ve been doing “things” with code since I started writing BASIC on my Spectrum as a child and I’ve always coded to make a particular task easier. Agnitio is no different; I pulled a lot of (ugly in places!) code together to make a task easier. The point I want to make here is that whilst I don’t think you have to have been (or currently be) a professional developer to be a good application security professional I think you have to be able to write code. To be more specific if you haven’t tried to write secure code, if you haven’t had to write code that others will review, if you haven’t written code that will be used by someone other than yourself you have to ask yourself a question; why should developers listen to you and your secure development ideas? If you haven’t done any of the above you don’t know how difficult things are to implement and you don’t know what makes secure coding difficult and painful.

If you are someone who blogs, tweets, talks at conferences about application security or even have an application security job ask yourself that question and be honest with your answer. Go further and ask yourself when was the last time you sat down with a developer and tried to help them write secure code? I don’t mean giving them some generic “OWASP top ten” training I mean actually sit in front of the IDE with them and help them write secure code or help them fix a vulnerability in the code. What about helping them prototype security functionality or designing a secure application with them? How many times have you made secure coding easier for your developers? If you don’t know what makes secure coding difficult and painful you will never make a difference.

What I’m trying to say is practice what you preach! Talk is cheap, we can talk of echo chambers all day long but the problem is we fill the echo chamber with talk. Even if you go outside the echo chamber and still only talk you aren’t helping.

I’m glad I’m not alone in saying this kind of thing; I think Andrew Wilson and David Shackleford made similar points recently.

I wanted to finish on a positive note so I will say I can’t wait to look back over the next 12 months which I hope will be even more positive than the last 12 months. I hope even more people download and use Agnitio and provide feedback, feature suggestions and whatever else they want to contribute!

SN

This entry was posted on July 14, 2011 at 12:11 pm and is filed under Application Security, Ninja News and Updates . You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

26 comments   >

  1. Mr. Mindaugas Vezelis says:

    Good man, do not give up.

  2. Daniel Galvin says:

    Fair play, SN. Don’t let the neysayers stop you from achieving your goals!

  3. sjk says:

    “It showed me that we have too many people in application security who actually have no real desire to move application security forward, they gain too much from trotting out the same old lines and references and they don’t want anything to come along and disrupt that.”

    This is the exact reason why I stopped coding Code Crawler.

    Add me to the Agnitio’s fan list and please let me know if you need some extra help, I’ll be glad to help.

    Cheers,
    Alessio

  4. Security Ninja says:

    Hi Alessio,

    I’m sorry to hear you experienced the same as I have. I think you have seen from past blog posts and some of the functionality I’m adding to v2.0 that I was a fan of Code Crawler and how it helped with code reviews.

    I will definitely be looking for extra help when I’ve released v2.0. I’m at a point now that I know I need extra eyes, hands and brains to take it forward :)

    SN

  5. Jeff Williams says:

    I feel your pain! Thank you for sharing so openly. I apologize for any negative feedback from OWASP members – it’s an occupational hazard that we tend to overfocus on problems. I’ve experienced some of the same on the projects I’ve contributed. I try to remember that for every noisy complainer, there are hundreds or thousands of quiet happy users! I hope you keep your efforts going – they are appreciated.

    –Jeff

  6. Andre Gironda says:

    Causing trouble and being cruel about appsec tools is my job.

    I’ve never said anything but positive about Agnitio!

    Let me know who it was! I need to sue them for my patebts

  7. Clerkendweller says:

    It takes courage to put your head above the parapet, but those who do so make such a difference. It’s easy to criticise things, and hard to contribute.

    Happy 7th month birthday Agnitio.

  8. Security Ninja says:

    Hi Colin,

    I completely agree, it is sad to see good contributors feeling like it’s too much hassle (see the comment from Alessio for example) because of those who snipe and criticise for the sake of criticising.

    As I’ve mentioned on this blog before and in this post I’m more than happy to receive criticism as long as its something I can use to improve Agnitio. Just saying “xyz” is rubbish doesn’t really help anyone.

    Thanks again for your support and feedback.

    SN

  9. Security Ninja says:

    You are very good at that job Andre ;)

    Thanks for the positive comments about Agnitio, it isn’t the flashiest application security tool out there but I hope people continue to benefit from using it.

    SN

  10. Security Ninja says:

    Hi Jeff,

    Thanks for the comment, we do indeed over focus on the problems and I do the same myself. I think it’s just human nature to take the negative comments to heart even when they are a tiny percentage of the total feedback we receive.

    SN

  11. Stefano says:

    Press on SN, that is just human tendency. Some encourage, some discourage, some helps, some just destroys. You got to wait for no encouragement to press on….keep walking man!

  12. James says:

    Well, this is a comment from someone who appreciates the work developers like you have done and are doing.

    I read a lot of blogs from many different people and different projects. What I have learned is that there are a lot of bloggers who write intentionally inflammatory blogs just to bring in readers and to be known. These bloggers, in my opinion, are a lot like the politicians; they will say what ever it takes to get noticed good or bad.

    So, my advise to you is to ignore those unprofessional glory hound bloggers and stick to the work you love to do. Just remember Blogging is all about the number of followers, not about the facts or accuracy of what is written.

  13. Juan Valencia says:

    The criticism is going to come regardless of what you do. Some will be constructive, and that is always appreciated, but some is just destructive. Whether you write open source software, or write novels, or sing, or act, or in my case write a few technical guides and personal thoughts, some people will just try to discourage you, they will criticize you, and try to make you feel like what you do is worthless.

    So my respects to you for not giving up even after the attacks and the attempts at discourage you and at remove you from source forge.

    Even if nobody else believes in your project, the one person who should always believe in it, is the important and relevant person for the project, you.

  14. JohnMc says:

    Fear not the naysayer. For their angst is that of one who lacking either the thought or the capability deride those who try.

  15. Security Ninja says:

    Hi JohnMc,

    Thanks for that, I really like that comment!

    SN

  16. Security Ninja says:

    Hi Juan,

    The first part of your comment is completely true, I actually love the constructive feedback but some people as you say will just attempt to be malicious to put others down.

    I certainly won’t be giving up any time soon because I’m passionate about this project and as you say the criticism will always be around :)

    SN

  17. Security Ninja says:

    Hi James,

    Thanks for commenting on this post. The inflammatory bloggers I tend to avoid as I can see they are very often just attention seeking. There are one or two who who are inflammatory but have a real reason for being that way, they are the minority though!

    All of the comments and feedback I’ve had on this blog just backs up my passion and belief in both Security Ninja and Agnitio so thanks again for the comment :)

    SN

  18. Travis Northrup says:

    Very nice article.

    Make sure not to forget to add the Google +1 to your sites articles, but since you currently don’t:

    +1

  19. John M says:

    The testing element really applies to all development – not just OSS.

    Date issues cause all sorts of fun for anyone developing for multiple nationalities. Numbers strings can also cause you issues as many countries use commas as the decimal place – e.g. “1.0” will show as “1,0” on Spanish settings

  20. Psiinon says:

    Welcome to the world of the developer :)
    Most of your users will never get in touch.
    Even if theres something that really annoys them and would take you 2 mins to fix, they just wont.
    Some people will complain about everything.
    Most people who do complain will think that a statement along the lines of “this sucks” is helpful feedback.
    And having told you that they will move on and never answer any follow up questions.
    You will never know that most of your users love one particular feature until you change it and then they’ll jump down your throat.
    But a small minority of users will give useful and helpful feedback – they are your biggest asset, treat them like royalty.
    I’m relatively new to the security world, but I’ve been a developer for over 20 years, and I’m afraid thats just the way it is, not just in the security space.
    Ignore the people who just pick holes, stick to your guns, but cultivate and listen to anyone who actually does give good feedback.
    And good luck with Agnitio!

    Psiinon

  21. Pingback: Links 19/7/2011: Why GNU/Linux Feels Better Than Mac OS X, Howard County Library Uses Ubuntu | Techrights

  22. Paul M says:

    In retail, the rule of thumb is that dissatisfied customers are 6 times more likely to comment.

    Evaluate your feedback by throwing out the pointless objections, for they are not the product of intelligent thought. Value reasoned insight, both positive and negative, for it will guide you well. And remember, every positive voice speaks for at least five more people!

    Good luck going forward.

  23. Security Ninja says:

    Hi Travis,

    Thanks for the comment, I will see if I can get the +1 feature on the site.

    SN

  24. Security Ninja says:

    Hi John,

    I completely agree with you on the testing. That should be the same regardless of license type :)

    I actually come a bit unstuck with that exact date issue when testing v2.0 on a Spanish version of Windows!

    SN

  25. Security Ninja says:

    Hi Psiinon,

    Thanks for the detailed comment, everything you say is correct. I suppose in the past everything I developed was just for personal use – especially in my past life as an infrastructure/operations security guy!

    I’m definitely ignoring those who deliver nothing constructive to the project. I have a small group of people who I’m trying to treat like royalty because of the great feedback and ideas they have given me so far :)

    SN

  26. Security Ninja says:

    Hi Paul,

    I like those numbers, I think I had read something similar about the reviews on hotel review websites.

    Thanks,

    SN

Leave a comment

VIDEOS & SLIDESHARES

Look at our latest security Videos & SlideShares

EVENTS & SEMINARS

Upcoming Security Events & Seminars

PODCASTS & DOWNLOADS

Check out our Podcasts & White Papers