A few months ago, I was invited to participate in VMware’s VCAP7-CMA Design exam development process. What a great experience!
I’ll keep my comments brief to avoid any material that may be covered by NDA, but I would like to touch on one topic that seems to be a hot subject. In the latest advanced design exams, VMware has announced they are removing the visual drag and drop style whiteboard questions. Some folks (I’m looking at you reddit thread) seem to be under the impression that VMware doesn’t want to bother with a complex testing tool. While I am not privy to the business drivers for this decision, I can certainly relay that the messaging around the change during the workshop I participated in was 100% focused on issues with test taker feedback and grading challenges.
In my technical circles here in Sacramento, this is a fairly common topic of discussion and I imagine elsewhere also. The common complaint is the gap between real world applied skill, and certifications, and closing that gap so certifications carry weight and “mean something.”
The consensus seems to be for exams that test the skill of operating, troubleshooting, and configuring a system, the best test is an extended lab-based exam. In this day of automation and cloud computing, I see no reason labs need to be expensive to administer or grade. Provided the scenarios are complex enough, test dumps become a much smaller issue. You also cut down on nonsense questions and close the gap between what I’ve heard called “book exams” vs. “real world exams,” the former being memorizing every word of documentation to pass an exam, the later being a skilled administrator capable of working with the system in question. At the end of the test, either you have configured the system to operate correctly, or you have not. Simple!
Applying this idea to design exams becomes perhaps more tricky. If I could craft an ideal scenario, it would require every candidate to present a live whiteboard session to a small panel of experts. I suppose in some ways, this describes the later part of the VCDX process. But it comes with high cost, high degree of uniqueness per session, difficult to scale, and so on. Perhaps that is where the drag and drop canvas design idea came from – automate the whiteboard session. Unfortunately, there are clearly problems in executing this approach. One of those problems is the whiteboard session allows the panel to ask interactive questions and get clarification. During a technical interview, this is a great way to see how the candidate thinks as well as how well they grasp the technology.
This is a long way to say the canvas design questions are a great aspiration but have proved challenging in practice. So, how do you write a design exam without canvas questions? There are lots of options. For example, you could present the test-taker with a drawing of a design, perhaps with a flaw, and prompt for clarification regarding the design. Or, you could use the old tried and true scenario based questions. There are other options too. These approaches in my estimation do not demand as much from the test taker as blank whiteboard and marker, but they seem highly repeatable and fair, do not require the test taker to explain their logic in order to see if their drawing is actually valid. They should be sufficient for the degree of knowledge tested in an advanced level exam. Again, we still have the expert level certification to denote true mastery of the subject and products.
All in all, I was very impressed by the test development process, with an enormous amount of metrics, analysis, and process behind the scenes that frankly, I had not really had occasion to think about as a test taker where the exam is a magical black box. I have a high degree of confidence in the folks VMware has running the certification exam development program, and I expect to continue to see improvements in the program based on real-world data collected in the field.