Skip to main content

Modernizing a NetApp Certification

Read on to find out how new versions of NetApp exams are written during an Item Development Workshop at NetApp's RTP office

In mid-October, this message popped up in the NetApp United Slack channel from Petya Stefanova, NetApp United's fearless leader:

Hey @channel there’s a new opportunity to participate in an IDW with NetAppU. This time the workshop will be reviewing the two exams for NetApp Certified Data Administrator ONTAP (NCDA, NS0-192) and NetApp Certified Support Engineer ONTAP (NCSE ONTAP, NS0-590), taking place mid-end January. If you are interested, drop me an email how you quality and can contribute to IDW. I need to submit nominations by Friday. So please let me know ASAP! Partners and customers can participate

I immediately knew that it was something that I would be interested in, so I talked to my employer to get their approval and put in my application. At the time, I didn't have any NetApp certifications so I didn't expect to be selected to take part in the process. 

A couple of weeks after that, I traveled to Las Vegas for Insight 2018 where I decided to take the NCSE and NCSE-OS exams. If I was in the running for participating in the IDW, I figured I might as well take the exam so that I could have some insight into the exam itself. As luck would have it, I passed and now was the proud owner of two new NetApp certifications.

A few weeks after Insight wrapped up, I was greeted one morning by an email from Brandi Einhorn, an Evaluation Manager for NetApp University - I was in for the NCSE-OS item development workshop!

Day 0

I flew in on Sunday night and got to RTP at around 5:30PM or so. I immediately met up with some other NetApp United members for dinner and drinks. Great to finally meet some more NetApp United folks! We called it a night pretty early though since we had work to do in the morning.

Day 1

I arrived at NetApp's RTP office at around 8:30 and got to meet Brandi in person (Brandi, if you're reading this, it was great to meet you!). We were also introduced to a psychometrician from a company called Mensor. What is a psychometrician, you ask? I wondered the same thing, and learned that it is someone who is an expert at the science of the psychological components of exams.

After getting to know the other IDW participants, we learned about the concept of the "minimally acceptable candidate (MAC)" who was the target audience for the exam, and asked to put ourselves in their shoes when designing exam questions. From there, we reviewed the existing exam objectives and then performed a technical review of the current bank of questions for the NCSE-OS, using the following criteria:

Based on this review, some questions were retained as they are still relevant in ONTAP 9.5 and some were retired since they were no longer relevant. We then decided on the importance of each exam objective category, which determined the number of questions per objective. Once this review was complete at the end of the day, we were left with 10-11 questions per IDW participant to write the following day.

Day 2 

We began Day 2 by writing each exam objective up on the board and the number of questions we needed to write for it, and then put our initials next to the ones that we felt we were capable of writing. Immediately, I was struck by the realization that, as a friend put it, "learning to write a good cert test is more difficult than taking a hard cert test". I found it VERY challenging to develop a new question from scratch while maintaining the rules of congruence, relevancy, difficulty and so on. I made a goal for myself to write one good question per hour. Luckily, I was surrounded by some very smart people who gave me some pointers on how to phrase questions/distracters so that they fit into the guidelines.

By some miracle, I was able to finish my questions by the end of the day, and then we were treated to a dinner out with both IDW groups at the Angus Barn. It was a great time and a really good way to blow off some steam after another day of hard work.

I'm starting to sense a pattern here. Maybe my Wisconsin drinking habits are rubbing off on them?

Day 3

Our primary objective was to review all of the new questions that were written following the same set of guidelines that were laid out in the initial technical review of the existing questions. This was the time when your fellow IDW participants had the chance to tear apart your question from a technical/grammar perspective and also share what they thought the answers were. The challenging part was that we were limited to four minutes of discussion per question - if we couldn't come to a consensus within 4 minutes, the question needed to be reworked. I figured if I got even one question in without needing rework, I would consider the day a success. In the end, I think I got two of ten approved without further rework.

Day 4

As we didn't do any revisions of rejected questions during yesterday's session, so now was our chance to polish up the wording on those questions that didn't quite make the cut or correct technical errors/inconsistencies. Once revised, the rejected questions were reviewed again for all of the aforementioned criteria as well as a clearly documented reference for the correct answer to each question.

After the day's activities wrapped up, we were treated to a tour of the Executive Briefing Center, which is designed to accommodate meetings with customer executive teams to talk about their challenges and how NetApp fits into the solutions.

During the tour, there were a couple of statistics provided by the folks there that I found to be really impressive. The close rate of deals that come into the EBC is 89% (!!) and an EBC visit reduces the sales cycle by 43%. Really cool stuff, and I appreciated the tour.

Day 5

As the IDW began to draw to a close, we still had a couple tasks yet to perform. One of them was an exercise where we had 20-30 seconds to decide whether we thought the MAC would be able to get the question right as well as determine the difficulty on a scale of 1-3. This data will be aggregated and used to determine the "cut score", or the minimum percentage a MAC must get correct in order to earn the certification. After some last-minute grammar/syntax corrections, everything was set in stone and my first IDW came to a close.

Final Thoughts

Participating in the IDW was one of the coolest things I've done in my career, honestly. I am so honored that NetApp would consider my contributions to the certification process valuable enough to invite me to RTP for the workshop. It was also a tremendous opportunity to meet new friends (of which there were many) and see old friends. If you're reading this and you are presented with the opportunity to take part in one of these, DO NOT pass it up! It's a solid week of hard work, but there is a lot of fun to be had as well. I can only hope I'll be invited to another one some day.

Special thanks to the following people:

@PetyaStefanova - for your recommendation
@BrandiEinhorn - for organizing the workshop
@WelshMatador - for teaching us Welsh slang - tidy!
@HybridDataCentr - for graciously offering to drive everywhere and talking about smoked meats
@tmacmd - for your experience in writing great questions and the ride to the airport
@pappa24 - for making sure we Americans know all about MetroCluster
@SpindleNinja and @infrachatting - for many good discussions over drinks as always
@NetApp and @NetAppU for the accommodations


Popular posts from this blog

How To: Unjoin NetApp Nodes from a Cluster

Let me paint you a word picture:

You've upgraded to a shiny new AFF - it's all racked, stacked, cabled and ready to rock. You've moved your volumes onto the new storage and your workloads are performing beautifully (of course) and it's time to put your old NetApp gear out to pasture.

We're going to learn how to unjoin nodes from an existing cluster. But wait! There are several prerequisites that must be met before the actual cluster unjoin can be done.

Ensure that you have either moved volumes to your new aggregates or offlined and deleted any unused volumes.Offline and delete aggregates from old nodes.Re-home data LIFs or disable/delete if they are not in use.Disable and delete intercluster LIFs for the old nodes (and remove them from any Cluster Peering relationships)Remove the old node's ports from any Broadcast Domains or Failover Groups that they may be a member of.Move epsilon to one of the new nodes (let's assume nodes 3 and 4 are the new nodes, in th…

ONTAP Configuration Compliance Auditing with PowerShell and Pester

I have been looking for a way to validate NetApp cluster configuration settings (once a configuration setting is set, I want to validate that it was set properly in a programmatic fashion) and prevent configuration drift (if a setting is different than its expected value, I want to know about it). I needed it to be able to scale out to dozens of clusters as well, so it needed to be something that I could run both automatically and on an ad-hoc basis if necessary.

NetApp PowerShell Toolkit

The core of the solution is the NetApp PowerShell Toolkit, without which this would likely not be possible. It contains 2300+ cmdlets for provisioning and managing NetApp storage components. It can be downloaded from the ToolChest on the NetApp MySupport site (with a valid login). You'll find exhaustive documentation there as well for each of the cmdlets along with syntax examples and sample code. It is a fantastic and easy way to automate common storage tasks - we use it in our environment for e…