I’m very happy to introduce Mark Hillick as a guest on the Security Ninja blog today. Myself and Mark have been talking about security for a few years now and we both appear to like and dislike the same things when it comes to the security industry. In 2011 we seem to have talked a lot about security education and more specifically the lack of relevant information security content in the large majority of university degrees.
Mark is a big fan of using hands on exercises like capture the flag competitions to teach people how to secure systems and applications but more importantly showing people what can go wrong if they don’t secure systems and applications. I’m also a fan of this kind of approach and although I prefer to focus a lot more on the “how to do things right” approach to security education they both work together very well.
After the 2011 HackEire CTF competition at IRISS Con Mark posted the results of the CTF and the results of a survey he created. The survey produced some very interesting answers as you can see in his blog post here and we wanted to discuss it a bit further in this blog post today.
We wanted to try and write this blog post as if we were having a conversation about these results and the general security education issues in real life, I hope you find this style interesting!
SN: So Mark, first of all thanks for sharing your survey results and thoughts with me today. Before we start can you tell me about the history of HackEire? Why did you create HackEire and how has the CTF changed since the first HackEire competition three years ago?
MH: Yeah, no problem. So over the years I did a few CTFs during SANS courses (mostly written by Ed Skoudis) and thoroughly enjoyed them. Additionally, I’ve taken part in various other challenges from ethicalhacker.net, forensicscontest.com and a few others. This gave me the opportunity to put me outside my comfort circle, put (what) theory (I knew) into practice, meet other security geeks (virtually usually) and ultimately learn and improve. I wasn’t aware of any such event in Ireland (and still am not, but I could be wrong) so I thought I’d give back to the community by sharing what little knowledge I possess, whilst this would also give me the opportunity to learn ? If I had have known how much work it’d involve, I wouldn’t have done it (joking :)). Projects like this also teach you that you can’t rely on most people to do what they say they’ll do whilst there are also many people who expect a commercial, corporate experience for free, whilst contributing nothing themselves. Despite these frustrations, it’s been incredibly rewarding for me personally.
To begin with, HackEire was four servers (over two vlans, one semi-hidden from the competition vlan) with each server containing one or two exploitable vulnerabilities (o/s, application, privilege escalation, free-flowing network traffic containing important information, weak passwords (sadly still prevalent) etc) running web applications, databases etc
2010 saw the CTF evolve with more network segmentation , different, more complex vulnerabilities as well as some simple tasks and a generic challenges page that contained questions about hacker trivia, packet capture analysis and basic reverse-engineering.
2011 took that evolution several scales up and I did a post that talked a little about how the design for HackEire had changed for 2011. Unfortunately it was essentially a two-man job (though @bobmcardle & @dsancho66 did a super job on the reverse-engineering challenges, while a buddy, Damian, helped with the development of the web app) so by the time the contest came around, the two of us were wrecked with our wives almost threatening divorce! I hope to do a few more posts on how the HackEire 2011 infrastructure was built because to be honest, it was a bit of an engineering challenge to say the least.
Despite being free to enter, we’ve always managed to hold the event in D4 and provided prizes though this year it required quite a bit of personal financing, something that is not viable in the long-run.
SN: Do you think the degrees offered by colleges/universities actually give students the knowledge they need to work in the IT roles companies currently want to fill?
MH: I think (generally speaking) these degrees give students enough to start in IT but not enough (in my humble opinion) enough to be walking into pen testing roles, consulting jobs or most info sec jobs (there are obviously exceptions to the rule, e.g. Ryan Dewhurst) but I find it funny that many of them come right out of college and are suddenly “Senior”.
I am far from being a HD Moore, Fyodor or Ed Skoudis but I studied Mathematics (Applied, Pure Maths & Theoretical Physics) at uni and that’s where I first experienced Unix, so I’m not like the child prodigies, I concentrated on sport when I was younger before realising I was far from being good enough. Anyway, Maths wasn’t IT but I enjoyed it, it tested me and I learnt a lot such that it formed a great basis for working in technology. However, I started my job at the bottom, configuring firewalls, proxy servers, directory servers and fixing end-user email and web problems. I learnt an awful lot there before moving on eventually to be the Team Lead for the team responsible for the design, implementation and maintenance of the complete Internet Infrastructure. I learnt immensely from “doing” and it wasn’t always doing the right thing. With this practical experience, I learnt far more than I ever did in college to the point that I now wonder, should I have gone straight into the real-world and started working in IT at 18?
I’ve come across an awful lot more colleagues that have a degree (who are ultimately wasters) in comparison to colleagues who don’t. A lot of folk who have a degree seem to be believe that they’re entitled to stuff, or maybe that’s the Celtic Cub Entitlement Program?, whereas colleagues who didn’t have a degree were more hard-working with more initiative but more significantly, for me, they had more innovation and came up with out-of-the-box solutions (possibly because nothing is drilled into them). Regarding the concept that with a degree, at least you’ve shown you can endure 4 years of crap, is it not better to get out after only wasting two years as opposed to wasting 4?
SN: As we have discussed in the past Mark I never went to college or university so I’m probably a bit biased when it comes to the “do a degree or get work experience” debate. I left school when I was 16 and went straight into work, right at the bottom of the ladder! I was certainly not a child prodigy either but I was fortunate to have been bought a ZX Spectrum by my parents before I’d even started school so I got a head start on a lot of people because of this. That planted a seed in me and from a very young age I knew I wanted to do a job that had something to do with computers!
Without giving a blow by blow run down of my career I was getting real world experience that I know others who did degrees still won’t have yet. I was an IT Manager at a property company before I was 21 managing a team of three people supporting 11 offices in 4 countries. I certainly agree with your thinking outside of the box comment above – when we were in remote places in foreign countries with no internet access (no Googling for the answers kids!) you just had to somehow make things work. The other thing I value apart from the experience is the fact I wasn’t locked into a curriculum, I learned whatever I wanted to learn which saved me from being forced to sit through outdated lectures.
SN: So based on your answer above if little Joe/Jill Bloggs came to speak to you today and asked whether they should get a degree/masters because they want to work in IT security what would you say to them?
MH: Little, eh? That’s a bit condescending, ain’t it It’s a difficult question without any simple answer and my answer is far from short.
If you look at Daniel Miessler’s post on hiring, he makes some excellent points (I think his site is awesome with so many varied topics and knowledgeable posts), however, I disagree with any guideline that says that you should prefer someone with a degree over someone without. That is possibly over-simplifying Miessler’s post, but for me, once you make any sort of guideline, people without the intelligence to see through things will blindly follow the guideline and yes, sadly many of those are in management
Additionally, if someone comes to me with a degree and says “Look, this shows I can endure four years of pain and mundaneness”, should I respect that or someone who had the ‘balls’ to get out after two?
This is just my personal experience but any non-graduate I’ve worked with or anyone that I’ve known who’ve done well in their career, they’ve been better work colleagues than those who have the degree.
Given that I’ve a Masters degree and certifications coming out my backside, that might seem contrary but these days, we need to look at things like:
- Do you have an Internet presence? Tweet, blog?
- Open source projects?
- Help on forums?
- Competed in any of the public challenges – honeynet, forensics, CTF, reverse-engineering?
- Ask the interviewee questions on what books he/she has read recently? What blogs do they read? Who do they follow on Twitter?
- Any contributions to Google Code, Sourceforge or GIT?
- Degree? If so, what and where’s it from? Was it done full-time or part-time? There’s a big difference between someone getting a 1st during full-time, versus a 2:1 during full-time but with an active sporting life or say a 2:2 but working full-time.
I don’t regret my degree and would probably do it again because I enjoyed abstract mathematics, quantum theory etc but it’s not overly relevant to my day job though I did show I could cram and retain large amounts of information about obscure topics.
Ha, ha, I still haven’t answered the question!!! My answer would be that it depends on the person – not everyone is suited to sitting in college doing a degree, whilst for other it’s the way they learn (some of us are visual, others like to touch whilst others rely on what they hear). If they truly believe that the degree will help them get to their goal quicker and they will learn more, than by all means, go to college. I think people should go with their heart, believe it or not, I did and it’s worked out so far.
From what I’ve seen in my interviews or more particularly in HackEire, college doesn’t seem to be teaching people the practical or relevant hands-on stuff. Do we have the correct people lecturing? Are they lecturing in the wrong way? Do they have the real-world experience to lecture?
A friend of mine was studying at night and the module was about system management. He had 10 years of experience in an American ISP (that was one of the original dotcom boomers) so he knew way more about how to manage systems, troubleshoot issues and scale and ended up having constant arguments with the lecturer. He still doesn’t have a degree but I’d have him in my team over the lecturer.
On the other hand, many companies won’t interview unless the candidate has a degree so in that case, yeah, get a degree!
I actually read CVs now from the bottom up, although that’s not very revolutionary – see here. I want to see the person loves technology, likes to play with it outside of work but also is well-rounded with other interests. Furthermore, for most folk that apply for jobs where I work, their degree is no longer that relevant, it’s more about what they’ve done in the last few years, do they know TCP, how to manage and scale systems, can they script and most important how do they handle themselves when the crap hits the fan?
SN: In your recent survey you asked a question which I strongly agreed with. You asked “Should we encourage folk to go into another discipline of IT before moving into computer/network security, e.g. development, system administration, operations etc (so he/she may gain knowledge)?” Could you explain why you feel this career path in security is valuable?
MH: I’ve probably answered that earlier to a degree but I believe that it’s better to learn by doing. For example, a lot of my life has been spent administering networks or systems, therefore, I know that sometimes for the sake of speed, performance, reliability, resilience or business reasons, the most secure solution is not the right way to go. On the other hand, I’ve seen folk who go straight into security, they’re brainwashed, accept nothing less than the 100% solution and end up getting a ‘waiver’ indemnifying them of any responsibility when they don’t get their way. It’s not a very constructive, team attitude, encouraging the development/infrastructure teams not to engage security. I ultimately believe that working outside security helps your understand technology better and also enables you to empathise with others more whilst it clearly provides you with more ‘skillz’ before moving into security.
SN: I’m not really sure where to start my input here so I will just jump in and say I agree that ideally no one’s first IT job should be in security. I feel that if you haven’t had experience in other roles first such as systems administration/networking/development you aren’t ready for anything other than junior security roles. The first half of my career was spent in non security roles learning a lot about networking and systems administration which I felt was the perfect grounding for a person looking to move into security roles. The problem is that we have companies needing security positions filled with very few people either having this experience or willing to accept they are not going to step straight into a senior security position.
I think security people who lack this real world experience are very easy to spot because every finding/issue is a blocker, every SQL Injection finding makes them run around like Chicken Little shouting the sky is falling. You can help prevent this by having something like the infosecmentors program internally but even then it’s far from ideal.
SN: We have to assume that some of the people currently working towards degrees will get their first job working in IT Security. So apart from taking part in the HackEire CTF what extra curricular study or research would you advise them to work on?
MH: See my previous answer concerning Twitter, blogging, GIT, public challenges, open source contributions (e.g. I’m currently trying to help out Doug Burks with his awesome Security Onion project).
Another viable option is doing some form of volunteer work, e.g. helping out with IT in a NGO (looks great on the CV) or how about on forums, such as Boards, Backtrack etc.
SN: We have both discussed the value of certifications privately and publicly on Twitter. Can you tell me what you think of certifications in general and how you think they can be made more relevant to the real world?
MH: I have done quite a few vendor certifications and found every single one of them to be useless. Most lack real-world experience, others include obscure questions about features that are never used and others have the answers on the Internet.
I’ve never been attracted to CISSP because I felt the various domains were too generic as you’ve to learn such a wide range and it doesn’t seem to delve deep enough for my liking; I’ve also met (at the start of my career) some complete muppets that held the CISSP and that probably made me biased; similarly with CEH, some folk who were completely lacking.
I’ve heard great things about some of the Red Hat certs (hands-on sys admin Linux work) and OSCP/OCSE. I rate the SANS/GIAC combination very highly though again I’m biased. The SANS course that I did with Arrigo Triulzi inspired me (seriously) because at that time, I was actually studying Neuromuscular Therapy because I was thinking of a career change but that course with Arrigo and a subsequent one with Ed Skoudis simply stunned me!!!
I’ve done a mixture of offensive and defensive (mostly defensive) but it’s not possible to regurgitate answers and the questions really make you think. The exams are open-book but you have to know your stuff in order to pass as there isn’t loads of time and the exams that I’ve done have very technical questions. I’m taking this a step further and paying myself to do the GSE certification, which involves a 3 hour paper (summarising GSEC, GCIA and GCIH) before doing the 2-day hands-on lab. For me the ‘lab’ aspect really shows that you can ‘walk the walk’ such that I’ll probably fail now as I’ve told your thousands of readers?
SN: I have a tough time giving my opinion on certifications because I know some of them can be very useful but my time as a certifications tutor really tainted all certifications for me. I actually shouldn’t say all certifications because I do think some of them are very good and very useful, mainly anything that requires hands on exercises to be completed.
When I was teaching certifications I mainly did Microsoft and Cisco certifications and the majority of people who were in my classes should never be working IT roles. I know that may sound harsh but I saw one person fail the Microsoft 70-270 exam (Windows XP) 11 times before he passed it. I also had one person go home from my class in the first break on the first day because it was too technical for him.
I left the certification industry two weeks after that because I realized I didn’t want to be part of something which churned out low quality certification passes and potentially very poor quality IT professionals.
SN: I noticed that a small minority (3 out of 59) said yes to question number 4 on your survey “Can you learn computer/network security simply by reading books?”. If you could spend five minutes with those three people what would your response be to that answer?”
MH: I’d show them one of two things, a shell prompt and ask them ‘What to do now?’ or a compromised system (preferably their own) and ask the same question?
“Mark currently leads the Networking & Cloud TRM team for Citrix Systems, in EMEA, where he concentrates on supporting and advising Citrix’s biggest networking customers across many industries. He was previously the Team Lead of the Internet Infrastructure team at an Irish financial institution he configured far too many firewalls (his own words).
Outside of work, Mark is better known as the creator and designer of the Hackeire CTF game. Additionally, Mark is a founding member of IRISS-CERT and was a volunteer Incident Handler with IRISS-CERT from 2008-2011.He has also presented at conferences and local chapter meet-ups, some of which can be found here.
From reading Mathematics at university, to collecting GIAC certifications, he’s spent far too many hours studying and would prefer to be hanging out in the ‘water’ somewhere. Mark is currently preparing for the GIAC GSE with SANS and he’s trialing a blog to follow his progress to help him study better :)”