I have recently wrote a practice exam for the WAA certification, administered as the last eMetrics conference here in Toronto. This ‘Test the Test’ event is a terrific idea which I was very happy to participate in, but the experience did leave me a little deflated in my excitement.
In general, I found the exam covered the topics of organization and administration adequately, but the there was much where little or no attention was given. In particular, the topics I was most surprised to be particularly light and/or under-served were:
- Initial implementation techniques/best practices.
- Deep analytical understanding (e.g. identification of outliers in data, pattern recognition, value proposition of metrics, etc.).
- Strategic planning & action as direct results of analysis (i.e. follow-up & response to analytical results).
Overall, I thought the test was a great start, but not as near completion as I had hoped. When asking around in my own circle of contacts about who was planning to take the exam to become certified an overwhelming majority of people said no. Though polled independently, the explanation of why came back unanimous: they all felt that the WAA certification will set the bar too low and water-down the value of those who hold certification. Looking back on teh exam I wrote, I think I can say with a fair degree of confidence that this will probably be the case.
Case in point: I have taken exams for MySQL and Omniture certifications; the first being very technical, the second being less technical overall, but still requiring a working knowledge of JavaScript. The MySQL exam was 70 multiple choice questions and 90 minutes long. Passing score is 60%. The Omniture Implementation exam was 75 multiple choice, 15 short answer, and was 3 hours in lenght. The passing score for certification is 80%.
In contrast, the WAA exam was 60 minutes and 50 multiple choice questions. I don’t remember what the passing grade is. Either way, I think it is only natural (and fair) to compare the WAA certification with my past certification experience; and the the bottom line is that there is still a long way to go for this exam to measure up to a standard that can be respected as an association certification.
For all my doom and gloom I do still think that this is a very important project. I just think that the value of it (and maybe the purpose) is a bit detached from it’s execution. In general, certifications are supposed to provide a sense of competence and reliability for our clients. If my clients can pass the WAA as part-time analysts just as easily as I can as a full-time analyst, what value does holding this cert add to my offerings as a consultant? Also, if the exam has a 70%,80% or 90% passing rate, what level of confidence does that instill for our clients who see an large number of certified WAA members with such a wide spectrum of qualifications/knowledge?
Hopefully, the WAA continues to pursue the development of the exam and work towards a test that will challenge us as professionals and help distinguish us from the fly-by-night consultants just trying to tap an industry while they coast through their career.
Dreamhost Referral
I think you’ve connected with one of the underlying questions with this certification. Is it for the elite few or the informed minority. Clearly no one wants it to be a wash out. There is however, something to be said of the COST of the exam. I recommend using the Google model- a small fee so there’s buy-in, but it’s approachable by most people. A popular test would do great things in promoting the WAA and it’s leadership in analytics.
Patrick, thanks for helping with Test the Test! Here’s a few items you might be interested in:
You only took a sample of the test, not the whole test. The full test is scheduled for 2 hours.
The average score on Test the Test so far is 58.
The most common score is 60.
Of the 80 or so people who have taken it, only 3 have scored above 70.
Maybe you’re just smarter than the average analyst?
Thanks for the assessment! Considering taking the exam myself. Good to know, presently, where it stands in terms of ones expectations and in particular one of a full-time [web?] analyst. All the best.
Many thanks for your feedback on your experience of the test. However, to reiterate Jim’s comment I wanted to clarify that you only took part of the certification test. Other people in San Jose, Washington etc all helped test different aspects of the test and component questions. We really appreciate your input and the input of all our testers, but until the roll out in San Jose, London and the global test centres, no one has as yet taken the full test version.
Best wishes, Vicky, WAA Board Director
I’m feeling very pessimistic about the test and here’s why:
-It’s too entry level, that’s bad and makes this test worthless to any organization attempting to quantify someone’s WA skill set. (Waste of time and money IMHO – just a revenue raising activity for WAA)
-The answers from WAA people in the comments seem to only condone more of the status quo – (ie: “the test is going to be longer!” Which I read as it is more of the same watered down garbage that anyone can pickup with a little common sense, half a brain and fist of cash.
-In my mind, the certification must mean something or people in the industry will just laugh at it, while people outside the industry will be mislead by it and somewhere puppies will be put to sleep each time someone takes the test. (the last item about puppies probably won’t happen, but I’m seeing the WAA likened to a level of evil if it continues in its current trajectory – who runs this place anyway?)
PS – I obviously obfuscated my name to avoid retribution. My name does not matter in this debate.
Please moderate me to suit your blog comments!
cool anyone for producing that
I agree with everyone’s comments. I was considering taking the exam in San Jose a few months ago. After viewing the sample questions (some of which were very poorly constructed and confusing IMO), seeing the cost of the test, and hearing the average score/passing score, I decided I would wait until the test and qualifications were more refined. I applaud the WAA for starting a program with such great potential, but IMO I think there is a lot of room for improvement here. I honestly don’t see the value of having this cert yet at present, in its current state.