Author: Federico Biancuzzi
NewsForge: Why BSD certification?
Dru Lavigne: While the BSD family of operating systems is well-known and respected for its maturity, security, and stability, there currently isn’t a mechanism to quantify the skills of those who use and administer BSD systems. The BSDCG wishes to address this need by first determining and then assessing the skillsets required to successfully administer BSD systems.
Here is a practical example. Let’s say you’re screening employment candidates for a position that requires configuration of Cisco routers. It is quite likely that your job advertisement will indicate that a CCNA certification is preferred (or required), because the CCNA represents a defined body of knowledge and a minimum required skillset. You can go to the Cisco Web site and see for yourself which objectives one needs to master in order to achieve a CCNA certification. Armed with that knowledge, you can sort the resumes into a CCNA pile and then skim through related job experience to make a short list of interview candidates.
Now, let’s say you need to hire a system administrator for your BSD servers. Until the upcoming certification goes live, you don’t have a predefined yardstick that states a prospective employment candidate has met a minimum defined knowledge base or skillset. While you can still use related job experience to make a short list of candidates, you will have to ask more probing questions at the interview to determine how the candidate learned his skills and whether the candidate has any obvious knowledge gaps.
NF: How did the project start?
DL: Interest in creating a standardized BSD examination has been around for quite a few years and there are numerous threads on the subject in the various BSD mailing lists and forums. As an IT instructor myself, I finally got to the point where I was tired of fielding the question, “Why isn’t there a BSD certification exam I can take?”, and my time and income had stabilized, so I could devote most of my time to a volunteer certification project and still manage to keep the bills paid.
Towards the end of December 2004 I emailed everyone I knew who had ever expressed interest in a BSD certification to see if they had the time to finally do something about it. Seventeen people responded favourably and we began to put together the resources we would need to get started: a Web site, a mailing list, an IRC channel, our ground rules, and a mission statement.
NF: Who’s involved?
DL: We’ve grown a lot in the past year and things evolve as we go through the various tasks needed to bring an examination to life. I won’t mention names as there are so many — most are mentioned on the Web site, and I fear I’ll inadvertently forget to mention someone. At the moment, there are:
- 17 members of the BSDCG (the original members minus one who is now on the Advisory Board)
- 5 members on the Advisory Board
- More than 80 translators for more than 20 languages
- 2 mailing lists with more than 1,000 subscribers who discuss the progress of the certification and volunteer to assist as required
We incorporated in the state of New Jersey in October and hope to receive IRS 501(c)(3) non-profit status by the end of January. We’ll be holding elections for the board of directors before the end of Q1 2006.
2006 will see even more changes. So far these announcements are on the horizon:
- A partner for psychometric assessment of the exam questions
- Partners for actual delivery of exams
- Corporate sponsorships
Part of creating the standard will include guidelines for testing and training centers. We would also like to create a plan for incorporating BSD certification into existing post-secondary programs.
NF: Do you plan to create official books and manuals?
DL: This has been the topic of much discussion. We originally thought we would stay away from this and just concentrate on defining the exam itself. One reason was accreditation; many accreditation bodies have rules prohibiting the organization that creates the exam from also creating, or even recommending, study materials. This makes sense from a conflict of interest perspective and is meant to prevent persons with “inside” information from having an advantage over other writers.
Since then, we’ve had a lot of requests and some convincing arguments for official courseware created by the BSDCG. In order to go this route, we would have to sort out two things. The first is a conflict of interest policy that assures the study material focuses on learning and understanding and is not a rehash of hidden exam questions or caters to the mentality of “read this book and pass the exam.”
The second is quite practical: money and time. A book is a huge time commitment and the BSDCG is made up of volunteers with day jobs and bills to pay. Monetary sponsorship, or lack thereof, to allow some BSDCG members to devote the time needed to create an official courseware will probably provide the ultimate answer to this question.
NF: Recently you started a fundraiser to pay the services of a psychometric agency and other startup costs. How is it going?
DL: This is my first fundraiser, so I’m still struggling with a mental image of standing on a street corner holding out a hat and smiling as people pass by. At the same time, I’m impressed at how much has already been accomplished and how generous the community has been.
As the annual report will show when it is published, we were extremely conservative in 2005, with our largest expenditure being the $500 fee for the IRS 501(c)(3) application. Thanks to generous donations of both hardware and bandwidth from FreeBSD Brasil LTDA and NYI, the fact that we only use BSD operating systems and open source applications, and that we administer our own systems on a rotating volunteer basis, our only operational cost has been the domain registration.
We still have some major startup costs to tackle, most notably the psychometric assessment of the exam questions. We spent Q4 2005 interviewing many agencies and have a short list of those with both experience in IT assessments and successful legal defenses. However, we won’t start the actual psychometric review until we have raised at least 85% of the projected $35,000 we need to ensure we’ll have all of the required funds when the bill comes due.
Once we partner with a psychometric agency, we’ll have a better idea of whether there will be any startup costs to deliver the exam itself or if that cost can be entirely incorporated into the exam price.
NF: What type of skills will the project certify?
DL: Our current mandate is to test system administration skills. The results from our Task Analysis Survey [PDF] show a need to test the skills of two separate audiences. The first audience covers those who are currently working as or who would like to find employment as an entry-level BSD systems administrator. The exam objectives [PDF] for the BSDA, the name of the exam intended to certify this audience, show that these testing candidates are expected to prove competency in basic Unix skills. They are also expected to be aware of the features found in the FreeBSD, OpenBSD, NetBSD, and DragonFly BSD operating systems.
The second audience covers those who have more experience administering BSD systems and who want to prove that they really know their stuff. We won’t be releasing the exam objectives for this audience until after the BSDA exam has gone live, but you can expect this to be a much more demanding exam.
There has also been interest expressed in an end-user exam that tests BSD desktop usage skills, as well as an exam for BSD developers. We will take a closer look at those once the first two exams have been released.
NF: Do you plan to include hands-on tests in the certification process?
DL: Exam methodology is a very important topic to the BSDCG as we want to create a certification that accurately assesses the skills, or lack thereof, of the testing candidate. This is one reason why a psychometric assessment is so important, as it maps each exam question to the exam objectives (known as a blueprint), assigns it a weight, and determines the percentage of candidates expected to answer that question correctly (known as a cut-off score). Psychometrics also deals with how the question is worded and presented so that, for example, even a multiple choice question can require the testing candidate to understand a concept rather than just memorize a tidbit of knowledge.
We won’t know the exact design of the questions until after the psychometric review for the exam is complete, though I suspect that the BSDA will be a mix of different types of questions. Once we know the design, we will explain it on the Web site and provide examples of each type of question, so the testing candidate knows what to expect on the actual exam.
Exam methodology also brings up the constraints of existing test delivery software. Once we know the design of the exam, we will have to compare our requirements to the current software offerings. There is a good chance that we will have to create our own customized component if we wish to deliver either some questions or the entire exam as a hands-on component.
NF: You talked about a “standardized BSD examination,” and this sounds like those Linux certifications that just cover one or a just few particular Linux distributions, but call themselves “Linux certification.” Why didn’t you choose to make separated certifications, one for each BSD project? After all, the four BSDs (FreeBSD, NetBSD, OpenBSD, and DragonFly BSD) are complete (and incompatible) operating systems, while Linux is just a kernel used by more that 350 distributions.
DL: Yes, I’ve always found the term “Linux Certification” a bit of an oxymoron as none of the Linux exams I have seen expect you to understand the kernel. I’ve taught dozens of Linux classes and the first question I always ask is “what is Linux?”, and so far only one student has answered that question correctly.
The question of four separate exams versus one integrated exam has been a lively topic within the BSDCG and was posed to the public in the Task Analysis Survey. A legitimate complaint is, “Why should I be forced to learn four separate operating systems when I only plan on using one?” The answer to that became more obvious once we ascertained that there were two different audiences.
As we created the BSDA exam objectives it became obvious that it was more beneficial to have one all-inclusive exam rather than four separate exams. The bulk of the BSDA is demonstrating proficiency in basic Unix and system administration skills, and most of these commands are the same on all four operating systems. Where there are differences, these are listed in the exam objectives and on the Web site in a Rosetta Stone-like chart [PDF]. We feel it is a good thing to know ahead of time that in the real world command A is slightly different on the different BSDs, as it can save some frustration when it doesn’t work as one expects. Also, bear in mind that the BSDA doesn’t expect the candidate to be a guru, and won’t be drilling down on the deep nuances of each operating system — just the basics.
The second exam aimed at the more experienced BSD system administrator will have a different layout. While we have yet to create the exam objectives or determine the testing methodology for this second exam, we will be keeping the audience being assessed in mind. As an example, this audience will be expected to know how to choose the best tool for the job at hand, so we are aiming for a testing methodology that allows the testing candidate to choose his own operating system and applications.
NF: Do you think companies will be able to trust the certifications assigned by the BSDCG more since it is run by BSD experts, and not by a company that does this for money?
DL: Trust has different meanings depending upon one’s context. Those that are already within the BSD community are familiar with the BSDCG members and the talents they bring with them as they help define the BSD certification standard. I’m sure that this segment is more comfortable with a certification created by experts from the BSD arena and would be less likely to trust a certification created by “strangers,” regardless of the prominence of the name of the company sponsoring such a certification.
The world outside of the BSD community operates in a different context wherein the BSDCG, and possibly BSD itself, are the “strangers.” Creating trust outside of the BSD community requires the creation of an infrastructure, and this will take time.
Some of that infrastructure can be created by the BSDCG and some of it needs to be created by the BSD projects and the BSD community itself. Much of the work we did in 2005 helped to define which parts of this infrastructure are already in place and which bits need to be worked on.
While the BSDCG is committed to creating superior examinations that provide an accurate assessment of the testing candidate’s skills, we realize that there is more to creating an exam than just creating the exam. We would eventually like to see an infrastructure that includes elements such as:
- Testimonials from companies that have benefitted from BSD certification
- Guidelines for integrating BSD certification within existing college and university degree programs
- Focus groups with IT departments to help keep the exam objectives in line with real world needs
We have many other ideas on the back burner and always appreciate input and assistance.
