Opinion Testing the test
Unlike its counterparts,the CAT is never scrutinised...
The Common Admission Test,or CAT,is designed and administered jointly by all the Indian Institutes of Management (IIMs),and is the Indian version of the Graduate Management Admission Test (GMAT) designed and administered in the US by the Graduate Management Admission Council. Both the tests have three components; two,Quantitative and Verbal,are the same,while the third is called Logical & Data Interpretation in the CAT and Analytical Writing Assessment in the GMAT. And,of course,both are used for an assessment of the abilities of a candidate for admission to a postgraduate programme in management.
In the US,the GMAT has come to dominate all other similar tests for graduate management admissions. The CAT,too,is used not only by the IIMs but also by more than 100 other business schools in India. The demand for management education being high,a large number of business schools have mushroomed in India; therefore,a fairly large number of candidates appear for CAT every year. There is,however,one big difference between the two tests. Whereas the reliability and validity of GMAT are well-established,nothing is known scientifically of the reliability or validity of CAT. As the website of the GMAC points out,for more than three decades the GMAT has been repeatedly studied and tested. On the basis
of those studies,the average reliability and validity coefficients of GMAT are 0.92 and 0.48 respectively. (The higher the reliability coefficient,the more repeatable or reliable the test scores; the higher the validity coefficient,the more confidence we should have in predictions made from the scores.)
Independent support for these claims comes from the results of hundreds of validity studies carried out by research scholars around the world. A recently published meta-analysis reviewed 402 such studies involving a cumulative total of 64,583 students,and found an average operational validity of the GMAT to be 0.47. The GMAC also runs a Validity Study Service to pinpoint the relationship between GMAT scores and performance of students,available free to participating institutions.
In sharp contrast,we know practically nothing about the reliability and validity of the CAT. A search through the relevant journals reveals no studies on the subject; the CAT-IIM website is also silent on the question. My own current study of a leading business school shows that the CAT Total Score has no statistically significant relationship with the first-year grade-point-average of students in the postgraduate diploma course in mana- gement. The observed correlation was found to be 0.083 for General Management Course and -0.194 for the HR Management course both being non-significant.
While it is not appropriate to generalise from the findings of a single study,the absence of any significant positive relationship between CAT score and student performance does raise serious doubts about the tests validity. It is,therefore,the responsibility an obligation,even on the part of those who have developed this test to establish and demonstrate the reliability and validity of this all-important selection device which after all decides the fate of lakhs of candidates every year. Such studies,moreover,need to be carried out on a continuing basis as the GMAC does in the US. This is necessary not only because questions keep changing but also because the mode of administration changes.
In case CAT is not really valid,its use may result in two types of errors: the type-1 error,rejecting good candidates who,if admitted,would have performed well; and the type-2 error,admitting not-so-good candidates who may be unable to cope with the required standard of performance in business school. Unaware and/ or unmindful of these potential hazards,the widespread use of CAT seems to be based entirely on an untested belief that so-called objective tests (true or false; multiple choice),especially those scored by machine,must be reliable and valid!
Science has no place for untested beliefs or myths. Before you accept any principle,you ask for the evidence on which it is based. Why is evidence on the reliability and validity of the CAT not available to its various stakeholders? Parents,candidates,business schools,government) have not cared to ask. And so the tests developers and administrators have not cared to devote a part of their substantial revenue from it to research on this vital subject. The HRD ministry is currently involved in improving the process of IIT admissions. It should also look seriously at the CAT and its impact on the selection of our future managers.
The writer is professor emeritus,International Management
Institute,Ghaziabad
express@expressindia.com
Share
