FREQUENTLY ASKED QUESTIONS
1. Is WOMBAT a test battery?
WOMBAT is not a test battery. It is unlike any of the more traditional tests presently offered to operators. However, it is a logical development from conventional aptitude batteries that sample several so-called basic abilities individually. Instead, WOMBAT is designed as a complex operational environment that presents multiple sources of information and multiple response alternatives to which attention must be allocated efficiently.
WOMBAT incorporates several tasks that emphasize complex neuropsychological functions. During administration of the test, the candidate is not told which task to perform or in what order to perform them. Instead, the candidate must monitor all information sources vigilantly, exercise good judgment to determine the current priority of each task in a timely manner, and give primary attention to whatever task has the highest momentary urgency in the ever-changing, automatically adaptive scenario.
2. Is WOMBAT a psychomotor test?
WOMBAT is not a psychomotor test in the usual sense of the term.
WOMBAT-CS requires a modest degree of psychomotor ability in the execution of a target tracking task. The left hand operates a single-axis joystick control, and the right hand operates a two-axis joystick control. The task is to position cursors to follow the random movements of two independent targets, one expanding and contracting in size and the other wandering about the face of the display.
WOMBAT-FC has a minor psychomotor component involving the positioning of symbols with the joysticks, but it does not require the degree of skill called for by the WOMBAT-CS test.
In both versions of the test, the joysticks are also used in a figure-rotation task that calls for imprecise control of the orientation of three-dimensional figures to allow them to be viewed from any aspect. The psychomotor skill required is minimal.
Although the WOMBAT tracking tasks do not require unusual psychomotor skill, they are primary in the sense that they must be attended to frequently to maintain a high scoring rate, and for that reason they contribute heavily to the overall score. While there is a premium on timely responses, there is a greater premium on vigilance and giving attention to what is most important at the moment, which in turn depends on keeping an accurate mental picture of a dynamic situation.
3. Do different cultures react differently with WOMBAT?
No. Thousands of WOMBAT scores from five continents in several languages have failed to show significant differences in mean scores for various races, geographic areas, educational or vocational levels, or training in operating specific systems such as airplanes or computers. This tends to support the claim that WOMBAT tests are culture-free.
Some younger testees complain WOMBAT is "boring", "dull", or doesn't display fancy graphics or animations. That's correct. WOMBAT isn't a game and will not motivate people to do more, or to have a good time, nor will it play music or game sounds. WOMBAT is a test.
4. Are testees' scores biased differentially by their levels of computer experience?
On the basis of operational experience and the results of several experiments, the answer is "no." Most testees now are computer literate, and the few who are not have no serious disadvantage because no skills unique to computer use are involved. Typically those with little computer experience spend more time getting up to speed during the instructions phase, but once they are comfortable using the WOMBAT keypad, they have no further problem attributable to their limited computer experience.
5. Is WOMBAT the only test we should use?
No. WOMBAT measures situational awareness in a complex operational environment, but situational awareness is not the only requirement for successful careers in aviation and many other complex vocations. Good eyes and hearts and other attributes of physical and mental fitness are also needed, as are reasonable levels of other basic qualities. However, the lack of an acceptable level of situational awareness could well be a sufficient basis for elimination of a candidate.
Beware of charlatans selling "magic solutions" or "psychological revolutions." Ask questions about the performance criteria used in the development of the tests. Beware of flashy article titles and misleading abstracts. Question the links between writers and test publishers. Question the methodology and look at the numbers. Talk to test users and see how knowledgeable they are. Beware of validation reports based on sanitized data impossible to replicate.
Good tests on the market are not only used in normal operations, they are also used for scientific research in the medical, psychological, psychiatric, and industrial domains. Do your homework or get help from experts. No single test can predict with high accuracy the future operational success of complex-system operators. WOMBAT was designed to measure a candidate's ability to maintain situational awareness (SA) in a complex dynamic environment. Many factors in addition to SA should be involved in a selection program.
Selecting personnel calls for good science. Few validation studies based on objective performance criteria in actual operations have been done. Aero Innovation advocates having an on-going in-house validation program, comparing the objective performance records of your own people with their respective test scores. We also recommend calling on independent experts in the field of psychometrics to make sure you use the best tools available, in the right order, with the correct relative weights, to maximize your predictive validity.
We will be happy to recommend specific names if you contact us. We deal with knowledgeable consultants who will sometimes recommend WOMBAT and sometimes not, depending on the special requirements of the individual operator.
6. What are Type 1 and Type 2 errors?
The success of a selection protocol depends not only on the selection system's efficiency but also on the resources invested in training admitted candidates including those who subsequently fail. Continuing in-house validation makes it possible to come back to the selection phases and adjust/add/remove selection components. With this in mind, there are two types of errors your selection protocol will encounter.
A Type 1 error happens when an inadequate candidate is admitted to training or trial employment despite your screening protocol. This candidate will be costly to train and will represent an increased risk in later operations. Type 1 errors (also called "false positives") are those you want to reduce to a minimum, and you want to find out why they happen. They are the primary target of your in-house validation program.
The more frequent Type 2 error happens when a candidates who would be successful in your operation is rejected by your selection process. Although such rejections represent unfortunate mistakes, Type 2 errors (also called "false negatives") are usually regarded as the price one pays to minimize Type 1 errors. In other words, you accept the loss of a number of good candidates to guarantee a virtually total rejection of the inadequate ones. However, your tolerance for Type 2 errors depends greatly on the supply of new candidates. During periods of shortage, management is often faced with lowering admission standards to produce more graduates, recognizing and accepting the higher attrition rates.
7. What should we do during a recruit shortage?
Dr Diane Damos of Damos Aviation Services near Chicago reports: "If the Type 1 error rate is close to 0, then (as Classical Test Theory predicts) the Type 2 error rate must be extremely high. This is what is expected to happen when an employer opens 700 positions and receives qualified 35,000 applications. In such a case, missing a good applicant doesn't matter; there are plenty of good applicants. So the Type 2 error could exceed 90% with no implications to the employer. To have a selection system work perfectly, what selection people call the Type 1 error must be 0."
For coping with shortages of new recruits, Dr. Damos recommends changing the cutoff point of the selection process and resisting the tendency to be more lenient at the training level. "Decreasing the selection cutoff point has two effects: It increases the Type 1 error rate while decreasing the Type 2 error rate. The end result will be more people admitted into training who should be failed (increasing Type 1 error), and it will find more good people who would normally be eliminated by the selection system. Depending on how the cutoff is moved, the total number of recruits who successfully complete training will increase, and the percentage of recruits failing training will increase."
No doubt Dr. Damos' solution will increase the training costs. However, during a recruit shortage, it may be the only solution to increase the supply of graduates and limit the damage from increased Type 1 errors. Remember that Type 1 errors are accidents waiting to happen. Dr Damos again: "If the training system cannot produce enough graduates to meet the operational demand, then it seems that finding the good people who are usually rejected by the selection system is more important than incurring a little more training cost because some people start training and fail."
A word of wisdom.
8. What if I test the same candidates more than once?
As a general rule, the average scoring rate for the second administration will be very close to the average scoring rate for the final 30 minutes of the first administration. Thus, because there is typically a gradual increase in scoring rate during the first 30 to 40 minutes of the first administration, with a level scoring rate thereafter, the overall final score for the second administration will usually be slightly higher than for the first administration, on the order of five percent.
However, there are many factors that can affect the scoring rates for the two administrations by varying amounts. Some of these are the time interval between administrations and the testing environments on the two occasions (both slight effects), the degrees to which the testee is rested versus fatigued on the two occasions (a potentially large effect), and the relative motivation levels of the individual on the two occasions (potentially an even larger effect; strong effort is required for a valid score).
On the other side of the coin, WOMBAT scores are relatively unaffected by intervening activities such as flight training, practice on the test itself, playing computer games, or other tasks that also place a premium on situational awareness. Data from several experiments and from operational use have repeatedly failed to show significant effects from any of these sorts of intervening or prior activities.
9. What is the minimum acceptable score?
There is no magical "Pass/Fail" criterion score with any WOMBAT test. Administrators of the WOMBAT tests set different minimum scores below which candidates are rejected depending on their operational situations including the availability of candidates, the number of positions to be filled, and the levels of situational awareness deemed necessary for specific operations.
Different versions of WOMBAT products may also require minimum-score adjustments. Aero Innovation will help test administrators set minimum scores during initial implementation and periodic normalization exercises.
10. How long is a typical WOMBAT test?
The recommended test duration is 90 minutes, but this may be reduced to 60 minutes when enough candidates have been tested to determine at what point the predicted final scores correlate almost perfectly with the actual final scores. Depending on the quality of the candidate pool, many candidates may achieve their individual terminal scoring rates in less than 90 minutes, in which case the test duration might be reduced to as little as 60 minutes with little loss in predictive validity.
The recommended instructions period that precedes the test period is 60 minutes, but this can also be shortened with advance distribution of the Candidate Manual with emphasis on its value in achieving the highest possible score. Based on the above considerations, the nominal two and a half hours for a complete WOMBAT test may be reduced by as much as an hour.
After verifying the candidate's identification number, the administrator gives the candidate a short briefing and answers any questions. This takes two minutes, at most. The candidate is then left alone in a quiet room with the door closed. No further intervention by the administrator is required.
11. What is a typical pretest briefing?
Below is an example of the verbal briefing given to each candidate. First have the candidate sit in front of the WOMBAT console, ready to begin. The WOMBAT displays the first page of instructions. The test administrator should question the candidate as necessary during the briefing to make sure everything is understood.
"The instructions time is limited to XX minutes after you leave the present page. You will have plenty of time to read and practice everything. The time remaining before the test begins is displayed at the bottom of every page. As long as you have time remaining, you can go back and review individual exercises as you wish. Practice different ways, make mistakes, try different things. You can't hurt anything, and no scores are recorded during the instructions.
"If you want to interrupt the instructions period to ask me a question or to use the restroom, you may do so provided you are back before the test begins. The test will begin at the end of the instructions time, whether you are ready or not. You may begin the test anytime you are ready.
The last page of instructions will tell you how to start the test. WOMBAT will advise you when the test is complete. Then just leave this room with the door open so I can see you are finished. Do you have any questions? Once the test has started, give your absolute best effort until the end."
12. Will the candidate see the test results?
No. The candidate doesn't see the scores and cannot access them without the administrator's intervention. At the completion of the test, a message is shown on the WOMBAT display congratulating the candidate for having completed the test and asking the candidate to leave the premises.
13. How much training does the administrator need?
WOMBAT is extremely easy to administer. Most of the users have not received any specific training and have simply followed the operating instructions contained in our publications and on the world wide web site. When required, training is always free of charge and supplied directly from Aero Innovation. Test interpretation is equally simple and can be coached by telephone or e-mail using the first few scores gathered after the initial installation.
Most of the test administrators share their test results with Aero Innovation in confidence after having removed candidate identifications. This helps Aero Innovation develop better operating manuals and instructions.
14. Can we change/translate the text displayed on screen?
Yes. The text presented on the screen during the instructions and throughout the test phase is contained in a series of text files easily accessible though any word processor or text editor such as NotePad. The files are unprotected and saved as ANSI text files. Administrators are invited to modify, adapt, or translate the text to suit their respective operational and language requirements. Also, InDesign source files for manuals are available free of charge to official WOMBAT operators for easy customizing.
Remember that this is not a text-comprehension test. Instructions must be clearly understood by all candidates before they begin the test. The 60 minutes recommended as the limit for the instructions period will only be fully used by the slow readers. The world average for time spent reading the instructions in English is 26 minutes.
15. Are there any royalties or user fees to pay?
No! All WOMBAT tests are free of user fees or royalties. Administrators are invited to run as many sessions as desired with no restrictions. This also extends to commercial organizations involved with reselling or leasing testing services based on their WOMBAT systems. When you shop of selection tests, beware of low entry costs based on royalty schemes or user fees as they build no asset equity for your organization and become ongoing operating costs.
16. Are there any software updates?
Yes. Based on a continuing collection of data provided in confidence by most of the test administrators, Aero Innovation occasionally updates the WOMBAT software. Free updates are guaranteed in writing to all users for a period of two years following acquisition. Most of the recent updates were offered free of charge to all our clients to follow the new Windows OS. Updates are automatically posted on the operator's download page, and operators are advised via email. Your WOMBAT software is matched with the serial number of your WOMBAT consoles. Once an update becomes available, we make a distribution archive just for you so it can run with your consoles.
17. Can we try WOMBAT before committing ourselves to it?
Yes. You can try one WOMBAT system for a full year without committing to purchase it. We will charge you about $7,000 USD for that year's use, and that cost will be 100% deductible from the purchase price should you decide to adopt the system. There are no royalties to pay, and there is no limitation on the number of tests you can administer.
This means you can return the WOMBAT system after having used it for 12 months. If during the trial period, you decide to acquire it, we will deduct 100% of the trial cost from the current purchase price. This trial program was created to allow organizations to get acquainted with the test, to convince the management of its efficiency, and to spread the acquisition cost over more than one fiscal period.
18. Can we renew the one-year trial for a second year?
No. At the end of the first year, you must either acquire the system or return it to Aero Innovation in Montreal. We think you will buy it.
19. Who uses WOMBAT?
WOMBAT is being used in 42 countries on five continents by major and regional airlines, aviation training schools, air traffic agencies, nuclear plants, surface and underground transportation organizations, emergency-response teams, dispatchers, etc. It is also used by civilian and military research laboratories in the fields of psychology, medicine, engineering, and psychiatry as well as in industrial manufacturing and product design studies to increase the statistical power of multiple regression and covariance analyses.
20. Can we run WOMBAT on a laptop computer?
All WOMBAT systems are USB-1 and USB-2 compatible and run on the latest Windows platforms. We only support US-English Windows versions.
21. Can we use our own computers?
Yes. Although Aero Innovation can provide you with new ones, we prefer you use locally-serviced computers.
22. How often does the WOMBAT console break?
We have had two hardware failures in more than a decade of operating the WOMBAT equipment. In the early days of the CH Product Flight Sticks, the way the wiring was connected to the potentiometers caused an early wear after a few hundred hours of use. We saw this manifesting at the Montreal Metro in 1992 and immediately raised the issue with CH Products in California. They have since changed their manufacturing process and we retrofitted all existing consoles with new potentiometers. We have never experienced problems again with the joysticks.
Last year, a strong and frustrated Canadian testee broke one joystick from the console, shearing it from just below the hand piece. We shipped the remains to CH Products who confirmed never having seen this kind of failure. It is a rare event, and we replaced the joystick free of cost. To cater for emergencies, Aero Innovation maintains a large inventory of spare parts and consoles, all ready to ship via Federal Express for overnight delivery.
More recently, false contacts on the soft membrane keypad were observed. We adivsed the Montreal-based manufacturer and the defective keypads were replaced. We shipped remplacement keypads to operators for free.
23. Who sells WOMBAT?
All WOMBAT products are exclusively sold directly by Aero Innovation. Katherine Chen represents WOMBAT in Asia for faster and better support.
24. Do WOMBAT consoles come in different colors?
Unfortunately, no.
25. What are the physical specifications of the USB console ?
Width................... 48,0 cm
Depth................... 23,5 cm
Base Height ...........10 cm
Height.................. 26,0 cm
Total Weight............2.1 kg
Surface Finish......... Durable, flat coating.
Control Levers........ CH-Products(R) Flight Sticks(TM).
Connection..............USB-2,3
Side Walls............... Black steel
Keypad................... Soft-Touch Membrane
Certifications........... FCC, CE, CAN ICES-3 NMB-3
The look and feel of WOMBAT may be diffrent from what you expect based on the website's artist impressions. WOMBAT contains no hazardous material or battery, threfore a MSDS form isn't required. Package dimensions are 56 cm X 33 cm X 33 cm. Shipping weight is 3.7 kg (includes cable).