How Are ASPEQ Exam Papers Put Together?
So there you are, sitting in front of an exam that you need to pass to progress to the next part of your Pilot Qualification. So much is dependent on passing the exam that your heart rate is elevated, your mouth is dry and you’re sure you’ve developed a nervous twitch!
Has it ever crossed your mind why certain questions are asked or how the entire exam has been put together?
Today, our old friend, Captain Arthur Gatland, who is undoubtedly one of the great Totaras of aviation, tells us all about the job of putting together the exam you’re looking at. Arthur, alongside being a B777/B787 Flight Simulator Instructor for Air New Zealand, has also been a part time Subject Matter Expert for ASPEQ for more than 10 years.
And just so you can feel that the information you are about to read is accurate, this blog has been approved by both ASPEQ and NZCAA.
Captain Arthur Gatland
For over 10 years now I have been working part-time as a subject matter expert (SME) for Aspeq, writing pilot exam questions for New Zealand Civil Aviation exams and also for some overseas civil aviation authorities. This is an extremely rewarding and also challenging experience.
Subject matter experts are chosen based on their range of experience in various categories of Aviation. I have experience in military Aviation (RAF) for 11 years as well as being a pilot and instructor for Air New Zealand for over 40 years, including 10 years as the Flight Standards and Training Manager. I only have minimal experience flying helicopters – just 10 hours – so I don't feel quite so qualified to write questions for helicopter licensing - so I stick to what I know.
So what do the SMEs do?
Write new exam questions, review existing questions based on candidate comments, or based on routine performance reviews such as pass rate analysis, and also carry out candidate reviews when requested by candidates.
Writing questions:
It is very important that questions comply exactly with the syllabus topic reference.
Even if the SME knows the topic well, they must check the approved reference books to ensure the topic has been covered in the books, and the questions and answers have references that the students can use for their study.
Checking Subject Matter against Syllabus
If the question is multi choice, my philosophy is as follows – if the candidate knows the topic, the answer should be obvious and unambiguous. If they don't know the topic, it should be extremely difficult to guess the correct answer. In other words, the wrong answers (distractors) should be plausible but definitely wrong. Writing the question and the correct answer is relatively straightforward, but thinking of plausible distractors can be quite challenging.
What do we try to avoid?
We strive to ensure there are no unintentional "trick questions"! Questions must be clear, remembering that for some candidates English might be their second language. We try and avoid negative questions, such as "which of the following is not correct". Our collective schooling and human nature wants us to look for correct answers, not to be required to pick the wrong answer!
After we have written a batch of 10 questions, these are checked by an editor for grammar improvements, then peer-reviewed by another SME. This frequently results in fine tuning improvements before being approved for use.
Candidate reviews:
When asked to do a candidate review I check that the question correctly conforms to the syllabus topic specification, that it is written clearly and without ambiguity, that the source reference is correct, and that there is only one correct answer. In most cases, all of these requirements are met, however very occasionally we find there may be two correct answers. In this case we can declare the question void and request a rewrite, and/or instruct that the candidate's exam be marked as correct in this question.
On rare occasions, if I can see how the candidate has arrived at an incorrect answer, I will write notes to be passed to the candidate to explain where they went wrong.
Question reviews:
These are carried out routinely based on performance indications that pass rate is too low or very high on this question, or as a result of valid candidate comments or an SME comment. Often a high pass rate occurs simply because this is essential information and we will expect a high pass rate on that topic. With a low pass rate, it may be a result of the question being ambiguous or overly complicated, or there may be several distractors which are "almost correct". In these cases we will amend the question and some or all answers, to ensure the correct answer is clear and the distractors plausible but definitely wrong. Occasionally we simply ask that the question be completely rewritten, or to advise NZCAA that we think this topic is no longer applicable, mainly due to technology improvements in Aviation.
Syllabus reviews:
Routinely NZCAA carries out reviews of the syllabus topics, to upgrade these with technology improvements such as the introduction of GNSS / RNAV approaches and the use of data link for example. We can also make recommendations to CAA that a particular topic needs updating or review, but this is rarely required. After a syllabus has been reviewed, this requires a significant review of all applicable questions to update the syllabus reference numbers and/or to rewrite some questions, or write entirely new questions.
Aspeq has question writing contracts with various overseas Civil Aviation Agencies, and we follow the same process in writing or reviewing exam questions for these customers. The principles to be applied are identical. The syllabus topics are in general the same, with differences in some detail. Obviously it is critical that we comply exactly with the topic as specified.
As a general observation, there is significant commonality between States in pilot licensing requirements and syllabus topics, as one would expect. ICAO has a large number of documents which successfully specify or recommend operating procedures and licensing requirements for its member States which ensures commonality. This enables a number of countries to recognise licenses held by other States' pilots, and/or in some cases to apply credits in issuing their State's pilot license to someone from overseas.
Other exam writers? Aspeq has a great team of SMEs, a number of whom are my current or ex-Air NZ pilots / instructors (plus other experienced SMEs) and we work together well within the Aspeq processes to peer review each other's exam writing, with very helpful suggestions where applicable.
Checking they’ve got it right before release of the paper
In summary, I hope all future pilots - private or those with airline flying aspirations - feel reassured that the exam writers take all possible precautions to ensure that the questions we write accurately reflect the syllabus topics, are fair and unambiguous, and there are no trick questions! Questions are frequently reviewed for continued relevance and possible improvements.
We all want our pilots to be well-trained, knowledgeable and skilled - and we are proud to do our part towards achieving these goals!
Arthur Gatland
B787/777 Flight Simulator Instructor
(This blog has been approved by Aspeq and NZCAA.)
