Monday, February 25, 2013

Open Certification: The New EUCA3 Project - Item Writing

Eucalyptus to Open Source Certification Item Bank

In this blog post, I will detail the pioneering open certification approach Eucalyptus is taking to the EUCA3 certification, how we formatted our certification questions to make this work, why we believe our process is superior to similar "closed" certifications.

The "Tried and True" (and Broken) Approach

Whenever I have talked to folks in technical certification about open-sourcing our certification questions, I have received responses ranging from simple raised eyebrows to full blown "...thaaaat sounds a little crazy..." replies. Serious certification vendors typically spend a lot of time and money trying to keep the contents of their certification secret, and in turn, cheat sites... er... enhanced study guide sites like Test King spend a much smaller amount to make sure those secrets are exposed within hours of their release. It's a never-ending cycle, and a war that the certification vendors have utterly lost on all fronts. Yet, they still use the same "proven" methodologies across the industry, resulting in numerous item refreshes and vendors being pushed into asking ever more esoteric questions on their exams as they run out of relevant content to test on. In the end, the only people who lose out are the honest folks who maybe aren't quite technically ready for the exam and the employers who rely on those credentials to make hiring decisions. Both cheaters and highly technical folks earn the certification, and an prospective employer has no good way to tell the difference between the two on paper.

The Fix - It Starts with How You Develop an Item

When we set out to build a Eucalyptus exam, we asked ourselves how the process could be improved. While the obvious answer was to build a practical exam or simulation, these turn out to be very expensive to build and maintain, and we frankly didn't have the budget or resources to pull it off. So, then, how could you create a certification exam that would maintain its validity assuming the student had access to the entire item bank prior to the exam?

In the end, we changed certification from the ground up, and started with the way items are developed. In a traditional format, certification vendors get a group of smart people in a room, lock them all together for a few days, and ask them to come up with as many different questions as possible over a given number of areas. Part of the process is coming up with the correct answer, and then a certain number (usually 3 or 4) of reasonable distractors. We went through a similar exercise, only instead asked folks to develop as many items as possible, we asked folks to come up with real-world, complex questions for which the correct answer would change based on changing one of the variables in the question. A good example of the results of this methodology is this question:

Given the following information in the /etc/eucalyptus/eucalyptus.conf file (assume all other network-related values are at their default):  

Network Mode: SYSTEM  

How many security groups are possible? 

By making small changes to individual variables in this question, we could make 20 or more different questions out of this one stem. As a result, instead of having 3 or 4 possible answers, only one of which was ever correct, when you view this question on the EUCA3 exam, you would see 20 possible answers, and depending on which version of the question you got on the exam, any one of them would be right on a given version. It is entirely possible (and probable) that a person would see two or three variations of this question on a single exam.

But Aren't You Still Giving Folks All the Questions and Answers?

Yes, we are, however think about it - for a person to cheat, they would have to learn to identify all 20 variants of this one question, and all the correct answers based on tiny changes in the information. Frankly, by the time a person has been able to do that, we suspect they will likely have actually *learned* the technology and theory behind the question, and as such, will be able to answer the question based on knowing how it works rather than pure memorization. In other words, what we've done is tricked a would-be cheater (or most would-be cheaters, anyway) into actually LEARNING the content, since it's easier than memorizing 20 very similar questions and their answers. Assuming we have done a good job of asking relevant questions, giving folks the item bank actually produces more technically knowledgeable folks, and our certification ends up being *more* valid than most similar exams on the market today.

Also consider the environment today - students who are willing to pay the Test Kings of the world are already going to have access to any certification items they want. In the end, we're no worse off than anyone else, and because we designed our certification items from the start based on the assumption that a student would have access to them beforehand, we believe we are very much ahead of the game.

More Question Examples:

Below are a couple of additional actual questions that would get asked on the EUCA3 exam that would each have multiple possible answers based on small changes to the information:
The virtual machine type m1.large is defined as 2 CPUs, 2.5 GB of RAM, and 20 GB of disk. The available node controller has 4 CPU cores. The available node controller has 80 GB of disk space. The available node controller has 8 GB of RAM. The node controller is running 1 m1.large virtual machine. The MAX_CORES value in /etc/eucalyptus/eucalyptus.conf is set to the default. Can a user launch another m1.large virtual machine? Explain.

Your physical servers each have 4 CPU cores, 6 GB of RAM, and 200 GB of available disk space after all software components are installed.  Your only defined virtual machine type requires 2 virtual CPUs, 2 GB of RAM, and 20 GB of hard drive space.  Your only available image takes up exactly 8 GB of disk space.  What is the minimum number of physical servers you would need to build a cloud that meets the following minimum requirements?  

  • Number of instances to support: 2 
  • Number of clusters: 1  
  • Number of security groups: 1  
  • Volumes: Yes  
  • Snapshots: Yes  
  • Storage as a Service: Yes  
  • Volume storage: internal JBOD  
  • Eucalyptus HA: No  
  • Resource overcommitment allowed: No 
Note: if two requirements conflict, assume the one that requires additional hardware overrides the other. For example, if requirement A implies 4 servers are required, but requirement B conflicts with requirement A and requires 6 servers, the correct answer would be 6.

Future Plans and Modifications to the Test Process

Very soon we will be publishing our entire EUCA3 certification item bank and licensing it under the same Creative Commons license as our documentation and DBM courseware. We still have some work to do to replenish or modify the item bank to address the technological misunderstanding that was cleared up late last year, but we feel pretty good about our chances of re-launching a EUCA3 beta by early Q3. In the meantime, when the questions go public, you are invited - welcomed, even - to look them over, use them for personal study, and even help us come up with even better questions or additional variations on the ones that exist today. We plan to do all of this in the open, so that hopefully the Test Kings of the world won't have a way to make any money off of a EUCA3 exam participant.

As far as the exam experience goes, this changes one small piece of our plans. Rather than making the exam open-book AND open-note, we will no longer allow notes. It would simply be too easy for a cheater to print up small copies of all the items and just look up the one based on the variable information. That said, we will still provide a full digital copy of the Design, Build, and Manage materials that are the source for every question on the test during the exam, and using that resource will not be considered cheating.

I would love to hear what you think about all of this! Questions, comments, and concerns can be posted in the comments, or if you prefer, email them to me at I especially want to hear your concerns, as a test-taker or a prospective employer, and if we haven't thought through all of the potential problems with our process, we'll make adjustments. Thanks for your time and interest!

No comments:

Post a Comment