CASE NO.:
Writ Petition (civil) 165 of 2005
PETITIONER:
Sanjay Singh & Anr. \005 Petitioners
RESPONDENT:
U.P. Public Service Commission,Allahabad & Anr. \005 Respondents
DATE OF JUDGMENT: 09/01/2007
BENCH:
Y.K. SABHARWAL, C. K. Thakker & R. V. Raveendran
JUDGMENT:
J U D G M E N T
[With W.P. (C) Nos.172, 409, 466 and 467 of 2005]
RAVEENDRAN, J.
These petitions under Article 32 of the Constitution of India have
been filed by the unsuccessful candidates who appeared in the examinations
conducted by the Uttar Pradesh Public Service Commission (’Commission
for short) for recruitment to the posts of Civil Judge (Junior Division).
2. On the request of the Allahabad High Court, to conduct the
examination for filling 347 posts of Civil Judge (Junior Division), the
Commission issued an advertisement in the Employment News dated
28.11.2003. As many as 51524 candidates appeared for the "U.P. Judicial
Service Civil Judge, (Junior Division) Preliminary Examination, 2003"
conducted by the Commission on 21.3.2004. The preliminary examination
was of ’objective’ type consisting of two papers \026 General Knowledge and
Law. The result was declared on 30.6.2004 and 6046 candidates were
declared qualified to appear for the "U.P. Civil Judge (Junior Division)
Examination (Main), 2003" which was of ’descriptive’ (conventional) type.
The Main examination consisted of five papers (each carrying 200 marks) -
General Knowledge, Language, Law I, II and III - and was held between 5th
and 7th October, 2004. The number of candidates who took the said
examination was 5748.
3. The answer scripts relating to each subject were distributed to several
examiners for valuation, as it was not possible to get the large number
evaluated by a single examiner. The number of examiners, to whom the
answer-scripts were distributed for valuation, were as follows : General
Knowledge \026 18, Language \026 14, Law-I \026 11, Law-II \026 10, and Law-III \026 14.
The marks assigned by the examiners were subjected to ’statistical scaling’
and the results of written examination based on such scaled marks, were
declared on 7.3.2005. Thereafter, 1290 candidates were interviewed between
14.4.2005 and 26.4.2005. After such interview, the Commission declared the
final results of the examination on 1.5.2005 based on the aggregate of
’scaled marks’ in the written (Main) examination and the marks awarded in
the interview. On the recommendations made by Commission, appointments
were made to 347 posts of Civil Judge, Junior Division.
4. The petitioners, who were unsuccessful, are aggrieved. They contend
that the statistical scaling system adopted by the Commission is illegal as it
is contrary to the Uttar Pradesh Judicial Service Rules, 2001. They also
contend that conversion of their raw marks into scaled marks, is illegal as it
was done by applying an arbitrary, irrational and inappropriate scaling
formula. It is submitted that the Commission’s exercise of subjecting the http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 2 of 24
marks secured by the candidates to scaling, has resulted in meritorious
students being ignored, and less meritorious students being awarded higher
marks and selected, thereby violating the fundamental rights of the
candidates.
4.1) W.P. [C] No.165/2005 was filed on 5.4.2005 even before the final
results were declared, praying (i) for a direction to the Commission not to
adopt the system of scaling and to declare the results of the Main
Examination on the basis of actual marks obtained by the candidates; and (ii)
for a direction that the petition be heard by a Bench of three or more Judges
as the decision of a Bench of two Judges of this Court in U.P. Public Service
Commission v. Subhash Chandra Dixit [2003 (12) SCC 701] upholding the
system of scaling adopted by the Commission does not lay down the correct
law.
4.2) The other petitions were filed after declaration of the final results, in
effect, for the following reliefs : (a) for quashing the results of the U.P. Civil
Judge (Junior Division) Main Examination-2003 declared on 7.3.2005 and
the final results declared on 1.5.2005 on the basis of scaled marks and direct
the Commission to declare the results on the basis of actual marks secured
by the candidates; (b) to direct an inquiry by an independent agency into the
irregularities committed by the Commission in the said examination; (c) for
a declaration that the use of ’statistical scaling’ in regard to the examinations
for the subordinate judiciary is unconstitutional; and (d) to reconsider the
law laid down in Subhash Chandra Dixit (supra).
5. The respondents raised the threshold bar of maintainability. It is
submitted that this Court in S. C. Dixit (supra), has rejected identical grounds
of attack and upheld the statistical scaling method adopted by the
Commission in the examination conducted in 2000. It is contended that the
prayers in these petitions under Article 32, in effect, seek setting aside or
review of the decision in S. C. Dixit, and that is impermissible. Reliance is
placed on the Constitution Bench decision of this Court in Rupa Ashok
Hurra v. Ashok Hurra [2002 (4) SCC 388], to contend that a writ petition
under Article 32 would not lie to challenge any judgment of this Court or
that of a High Court, as superior courts are not ’State’ within the meaning of
Article 12 and their judgments cannot be termed as violative of fundamental
rights. It is also pointed out that Review Petition (Civil) No. 162/2004 and
Curative Petition No.43/2004 filed in respect of S. C. Dixit (supra) were
rejected on 04.2.2004 and 6.10.2004 respectively.
6. In regard to merits, the Commission contended that the ’statistical
scaling’ method adopted in regard to Civil Judge (Junior Division)
Examination is legal, scientific and sound and its policy to apply statistical
scaling to marks of written examination, was based on experts’ opinion as
also the experience gained in conducting several examinations. It is
submitted that under the proviso to Rule 50 of the U.P.Public Service
Commission (Procedure and Conduct of Business) Rules, 1976, it is entitled
to adopt any formula or method or device to eliminate variation in marks;
that it found variation in the marks awarded by different examiners on
account of a phenomenon known as ’examiner variability’ and to eliminate
it, statistical scaling was introduced. It is further submitted that matters
relating to the conduct of Examination, evaluation of answer-scripts,
application of methods to bring in uniformity in evaluation are matters of
policy involving technical and scientific decisions based on expert opinion;
that courts are not equipped to pronounce upon such matters and, therefore,
should not interfere in the absence of manifest arbitrariness or mala fides;
and that, at all events, in the absence of an opinion by a body of experts in
the field of statistics certifying that the system of scaling adopted by the
Commission is unsound and irrational, there should be no interference.
Lastly, it is submitted that if the court, for any reason, should hold that the
existing scaling system should be substituted, that should be done
prospectively.http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 3 of 24
7. On the contentions urged, the following questions arise for our
consideration :
(i) Whether the writ petitions are not maintainable ?
(ii) Whether ’scaling’ of marks is contrary to or prohibited by the
relevant rules ?
(iii) Whether the ’scaling system’ adopted by the Commission is
arbitrary and irrational, and whether the decision in S. C. Dixit
(supra) approving the ’scaling system’ requires reconsideration
?
(iv) If the statistical scaling system is found to be illegal or
irrational or unsound, whether the selections already made,
which are the subject-matter of these petitions, should be
interfered with?
Re : Question (i) :
8. It is true that a judgment of this Court cannot be challenged in a
petition under Article 32. It can, however, be reviewed under Article 137 or
in exceptional circumstances reconsidered in exercise of inherent power, on
a curative petition (See Rupa Ashok Hurra). It is equally true that a final
judgment of a High Court can be challenged only by an appeal under
Articles 132 to 134 or by obtaining ’special leave’ under Article 136 and not
by a petition under Article 32. But that is not the issue here.
9. In regard to decisions of civil courts in suits governed by Civil
Procedure Code or appeals therefrom, the term ’judgment’ refers to the
grounds of a decree or order, ’decree’ refers to the formal expression of an
adjudication in a suit and ’order’ refers to formal expression of any decision
of a civil court which is not a decree. In regard to the decisions of High
Court and Supreme Court in writ jurisdiction, the term ’judgment’ is
normally used to refer to the ’judgment and order’, that is the grounds for the
decision and the formal expression of the decision. The petitioners do not
seek to upset the ’order’ part of the judgment in S. C. Dixit (supra) which
decided the validity of UP Civil Judge (Junior Division), Examination, 2000,
held under the UP Nyayik Sewa Niyamawali 1951. The grievance of the
petitioners is in regard to the UP Civil Judge (Junior Division) Examination,
2003, held under the UP Judicial Service Rules 2001. They, however,
contend that the ratio decidendi of the decision in S.C. Dixit upholding the
Commission’s system of scaling of marks in written examination, requires
reconsideration. Therefore, these petitions are neither for ’review’ nor for
’setting aside’ or ’questioning’ the decision in S.C. Dixit. Therefore, the bar,
referred to in Rupa Ashok Hurra, will not apply.
10. The contention of Commission also overlooks the fundamental
difference between challenge to the final order forming part of the judgment
and challenge to the ratio decidendi of the judgment. Broadly speaking,
every judgment of superior courts has three segments, namely, (i) the facts
and the point at issue; (ii) the reasons for the decision; and (iii) the final
order containing the decision. The reasons for the decision or the ratio
decidendi is not the final order containing the decision. In fact, in a judgment
of this Court, though the ratio decidendi may point to a particular result, the
decision (final order relating to relief) may be different and not a natural
consequence of the ratio decidendi of the judgment. This may happen either
on account of any subsequent event or the need to mould the relief to do
complete justice in the matter. It is the ratio decidendi of a judgment and not
the final order in the judgment, which forms a precedent. The term
’judgment’ and ’decision’ are used, rather loosely, to refer to the entire
judgment or the final order or the ratio decidendi of a judgment. Rupa Ashok
Hurra (supra) is of course, an authority for the proposition that a petition
under Article 32 would not be maintainable to challenge or set aside or
quash the final order contained in a judgment of this Court. It does not lay http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 4 of 24
down a proposition that the ratio decidendi of any earlier decision cannot be
examined or differed in another case. Where violation of a fundamental right
of a citizen is alleged in a petition under Article 32, it cannot be dismissed,
as not maintainable, merely because it seeks to distinguish or challenge the
ratio decidendi of an earlier judgment, except where it is between the same
parties and in respect of the same cause of action. Where a legal issue raised
in a petition under Article 32 is covered by a decision of this Court, the
Court may dismiss the petition following the ratio decidendi of the earlier
decision. Such dismissal is not on the ground of ’maintainability’ but on the
ground that the issue raised is not tenable, in view of the law laid down in
the earlier decision. But if the court is satisfied that the issue raised in the
later petition requires consideration and in that context the earlier decision
requires re-examination, the court can certainly proceed to examine the
matter (or refer the matter to a larger Bench, if the earlier decision is not of a
smaller Bench). When the issue is re-examined and a view is taken different
from the one taken earlier, a new ratio is laid down. When the ratio
decidendi of the earlier decision undergoes such change, the final order of
the earlier decision as applicable to the parties to the earlier decision, is in no
way altered or disturbed. Therefore, the contention that a writ petition under
Article 32 is barred or not maintainable with reference to an issue which is
the subject-matter of an earlier decision, is rejected.
Re : Question (ii) :
11. Article 234 of the Constitution requires appointments to the Judicial
Service of a State (other than District Judges) to be made by the Governor of
the State in accordance with the Rules made by him in that behalf, after
consultation with the State Public Service Commission and with the High
Court exercising jurisdiction in relation to such State. The UP Judicial
Service Rules, 2001 (for short ’Judicial Service Rules’) were made by the
Governor of Uttar Pradesh in exercise of powers conferred by Article 234
and Article 309 of the Constitution, in consultation with the Commission
and the Allahabad High Court, to regulate the recruitment and appointment
to Uttar Pradesh Judicial Service. The Judicial Service Rules replaced the
’Uttar Pradesh Nyayik Sewa Niyamawali, 1951’ which was in force earlier.
The Judicial Service Rules were amended by the Uttar Pradesh Judicial
Service (Amendment) Rules, 2003.
11.1) Rule 7 of the Judicial Service Rules provides that recruitment to the
post of Civil Judge (Junior Division) shall be by direct recruitment on the
basis of a competitive examination conducted by Commission. Part V of the
said rules lays down the procedure for recruitment to Judicial Service. Rule
16 provides for competitive examination and Rule 19 deals with the
syllabus. The said rules are extracted below :
"16. Competitive Examination \026 The examination may be conducted at
such time and on such dates as may be notified by the Commission and
shall consist of \026
(a) a written examination in such legal and allied subject including
procedure, as may be included in the Syllabus prescribed under rule 19,
unless the same is otherwise modified by the Governor in consultation
with the court and the Commission;
(b) an examination to test the knowledge of the candidates in Hindi,
English and Urdu;
(c] an interview for assessing merit of the candidate giving due regard to
his ability, character, personality, physique and genera suitability for
appointment to the service.
19. Syllabus \026 The syllabus and the rules relating to the competitive
examination shall be such as given in the Appendix II, provided that the
syllabus and rules may be amended by the Governor in consultation with
the Commission and Court."http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 5 of 24
Appendix II to the Rules contains the syllabus for the competitive
examination. It enumerates the details of the five subjects for the written
examination and the number of marks carried by each subject (200 each). It
also provides for a Personality Test (interview) to find out the suitability of
the candidates (carrying 100 marks). Note (i) to Appendix-II provides that
"the marks obtained in the interview will be added to the marks obtained in
the written papers and the candidate’s place will depend on the aggregate
of both".
12. Sub-Rule (1) of Rule 20 of the Judicial Service Rules requires the
Commission to prepare the result of the written examination and thereafter,
invite such number of candidates, who in the opinion of the commission
have secured minimum marks as may be fixed. Sub-Rule (2) provides for
participation of a sitting Judge in the interview of candidates.
Sub-rule (3) provides that the Commission shall prepare a final list of
selected candidates in order of their proficiency as disclosed by aggregates
of marks finally awarded to each candidate in the written examination and
the interview. The proviso thereto provides that if two or more candidates
obtain equal marks in the aggregate, the name of the candidate who is elder
in age shall be placed higher and where two or more candidates of equal age
obtain equal marks in the aggregate, the name of the candidate who has
obtained higher marks in the written examination shall be placed higher.
Rule 21 provides that the Governor shall on receipt of the list of candidates
submitted by the Commission under Rule 20(3) make appointment on the
posts of Civil Judge (Junior Division) in the order in which their names are
given in the list provided. Thus the Judicial Service Rules constitute a
complete code in itself in regard to recruitment to Judicial Service. It is also
evident that the marks finally awarded to each candidate in the written
examination and interview are crucial both for appointment as also for
purposes of inter se seniority.
13. The petitioners point out that the Judicial Service Rules do not
provide for substituting the actual marks obtained by a candidate by scaled
marks. It is contended that the words "marks obtained in the written papers"
in Note (i) of Appendix II clearly indicate that the actual marks obtained in
the written examination alone should be taken into account and not any
moderated or scaled marks; that in the absence of any provision for scaling
in the Judicial Service Rules, the Commission had no authority to substitute
the actual marks by ’scaled marks’; and that the places/ranks of the
candidates should be determined strictly on the basis of the aggregate of the
actual marks obtained in the main written examination plus the marks
obtained in interview.
14. The Commission contends that the manner of conducting examination
by the Commission, even in regard to recruitment to Judicial Service, is
governed by the Uttar Pradesh Public Service Commission (Procedure and
Conduct of Business) Rules, 1976 (for short ’PSC Procedure Rules’) made
by the Commission in exercise of the power conferred by the UP State
Public Service Commission (Regulation of Procedure and Conduct of
Business) Act, 1974. Rule 26 provides for preparation of a panel of
Examiners or constitution of a Committee for the purpose of holding
examination in each subject. Rule 28 provides that the question papers set by
the examiners shall be placed before the Commission to ensure conformity
with the required standard of examination and the Commission may
moderate the question papers or constitute a Committee to perform the work
of moderation. Rule 30 provides for advertisement of vacancies for which
selections are to be made and scrutiny of applications received. Rule 33
provides for the determination of place, dates and time of examination and
the centres for examination. Rule 34 provides for the list of persons suitable
to be appointed as invigilators and appointment of invigilators. Rule 37
provides for fictitious roll numbers (code numbers) to be allotted to each
candidate before the answer books are dispatched to the examiners for
assessment. Rule 38 provides that the number of answer books to be sent to
each examiner shall be fixed by the Commission. Rule 44 requires the http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 6 of 24
Secretary of the Commission to take steps for tabulation of marks obtained
by each candidate as soon as the answer-scripts are received after valuation,
after scrutiny of scripts, removal of discrepancies and corrections. Rule 45
provides for random checking of the tabulation to ensure correctness and
accuracy of tabulation. Rule 47 provides that the original roll numbers of
candidates shall thereafter be restored to the answer-scripts and for issue of
interview letters. Rule 49 authorizes the Commission to decide the number
of candidates to be called for interview to appear before a Board on any day.
Rule 50 provides that the interview marks awarded shall be kept in safe
custody. Rule 51 provides that mark-sheets shall be opened on the last day
of interview and immediately thereafter the marks of interview/personality
test shall be added to the marks obtained by the candidates in the written
examination, and thereafter on the basis of the total so obtained, the merit
list shall be prepared and placed before the Commission for final declaration
of the result. The proviso to Rule 51 provides that the Commission with a
view to eliminate variations in the ranks awarded to candidates at any time at
any examination or interview, adopt any method, device or formula which
they consider proper for the purpose. The Commission contends that having
regard to the proviso to Rule 51 which specifically enables them to adopt
any method, device or formula to eliminate variations in the marks awarded
to any at any examination, they are entitled to adopt the scaling system to
eliminate variations in marks.
15. The petitioners point out that the PSC Procedure Rules were not made
in consultation with the High Court. On the other hand, the Judicial Service
Rules, 2001 which came into effect from 1.7.2000, were made in
consultation with both Commission and the High Court. It is, therefore,
submitted that the Judicial Service Rules alone will regulate and govern the
recruitment of Civil Judges (Junior Division) including examinations and
interviews and the proviso to Rule 51 of PSC Procedure Rules will not apply
to recruitment of Civil Judges. Reliance is placed on the decisions of this
Court in State of Bihar v. Bal Mukund Sah [2000 (4) SCC 640], Union of
India v. Hansoli Devi [2002 (7) SCC 273] and Union of India v. Deoki
Nandan Aggarwal [1992 Supp. 1 SCC 323] in regard to interpretation of the
Rules.
16. This question was considered briefly by this Court in S. C. Dixit
wherein it was held that the PSC Procedure Rules made in exercise of power
under the U.P. State Public Service Commission (Regulation of Procedure
and Conduct of Business) Act, 1974 give the guidelines for any examination
to be held by the Commission and therefore, all the provisions of the said
Rules will be applicable to an examination for recruitment to judicial
service also.
17. It is no doubt true that Judicial Service Rules govern the recruitment
to Judicial Service, having been made in exercise of power under Article
234, in consultation with both the commission and the High Court. It also
provides what examinations should be conducted and the maximum marks
for each subject in the examination. But the Judicial Service Rules entrust
the function of conducting examinations to the Commission. The Judicial
Service Rules do not prescribe the manner and procedure for holding the
examination and valuation of answer-scripts and award of the final marks
and declaration of the results. Therefore, it is for the Commission to regulate
the manner in which it will conduct the examination and value the answer
scripts, subject, however, to the provisions of the Judicial Service Rules. If
the Commission has made Rules to regulate the procedure and conduct of
the examination, they will naturally apply to any examination conducted by
it for recruitment to any service, including the judicial service. But where the
Judicial Service Rules make a specific provision in regard to any aspect of
examination, such provision will prevail, and the provision of PSC
Procedure Rules, to the extent it is inconsistent with the Judicial Service
Rules, will be inapplicable. Further, if both the Rules have made provision in
regard to a particular matter, the PSC Procedure Rules will yield to the
Judicial Service Rules.
18. The manner in which the list of candidates as per merit should be
prepared is provided both in the Judicial Service Rules and the PSC http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 7 of 24
Procedure Rules. Relevant portion of Rule 20(3) and Note (i) of Appendix-II
of the Judicial Service Rules and Rule 51 of the PSC Procedure Rules
providing for the aggregation of marks and preparation of the merit list, are
extracted below :-
Judicial Service Rules
PSC Procedure Rules.
Rule 20(3). The Commission then
shall prepare a final list of selected
candidates in order of their proficiency
as disclosed by aggregate of marks
finally awarded to each candidate in
the written examination and the
interview.
Note (i) of Appendix-II. - Marks
obtained in the interview will be added
to the marks obtained in the written
papers and the candidates’ place will
depend on the aggregate of the both.
Rule 51. The marks-sheets so
obtained shall be opened on the
last day of interview and
immediately there after the marks
of interview/ personality test shall
be added to the marks obtained by
the candidates in the written
examination. Thereafter, on the
basis of the totals so obtained the
merit list shall be prepared and
place before the Commission for
final declaration of the result.
Provided that the Commission
may, with a view to eliminating
variation in the marks awarded
to candidates at any examination
or interview, adopt and method,
device or formula which they
consider proper for the purpose.
(different emphasis supplied)
As the field is occupied by Rule 20(3) and Note (i) of Appendix-II of
Judicial Service Rules, they will prevail over the general provision in Rule
51 of PSC Procedure Rules.
19. Rule 20(3) provides that the final list of selected candidates in order of
their proficiency as disclosed by the aggregate of ’marks finally awarded
to each candidate in the written examination and the interview". Note (i)
to Appendix II of the Judicial Service Rules provides that the "marks
obtained in the interview" will be added to "the marks obtained in the
written papers" and that the candidate’s place will depend on the aggregate
of both. Though Judicial Service Rules refers to ’marks finally awarded’, the
said Rules do not contain a provision similar to the proviso to Rule 51 of
PSC Procedure Rules, enabling the Commission to adopt any method,
device or formula to eliminate variation in the marks. It is not possible to
read the proviso to Rule 51 or words to that effect into Rule 20(3) or Note (i)
of Appendix-II of Judicial Service Rules. It is well settled that courts will
not add words to a statute or read into the statute words not in it. Even if the
courts come to the conclusion that there is any omission in the words used, it
cannot make up the deficiency, where the wording as it exists is clear and
unambiguous. While the courts can adopt a construction which will carry out
the obvious intention of the legislative or rule making authority, it cannot set
at naught the legislative intent clearly expressed in a statute or the rules.
Therefore, Rule 20(3) and Note (i) of Appendix-II has to be read as they are http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 8 of 24
without the addition of the proviso to Rule 51 of PSC Procedure Rules. If so,
what can be taken into account for preparing final list of selected candidates,
are ’marks finally awarded to a candidate’ in the written examination and the
interview. The marks assigned by the examiner are not necessarily the marks
finally awarded to a candidate. If there is any error in the marks awarded by
the examiner it can always be corrected by the Commission and the
corrected marks will be ’the final marks awarded to the candidate’. Where
the Commission is of the view that there is ’examiner variability’ in the
marks (due to strict or liberal assessment of answer scripts) or improper
assessment on account of erratic or careless marking by an examiner, they
can be corrected appropriately by moderation. The moderation is either by
adding (in the case of strict examiners) or deducting (in the case of liberal
examiners) a particular number of marks which has been decided with
reference to principles of moderation applied. If there is erratic or careless
marking, then moderation is by fresh valuation by another examiner.
Therefore, the marks assigned by the examiner as moderated will be the
marks finally awarded to the candidates or marks obtained by the candidates.
Moderation, it has to be held, is inherent in the evaluation of answer scripts
in any large scale examination, where there are more than one examiner.
20. We cannot accept the contention of the petitioner that the words
"marks awarded" or "marks obtained in the written papers" refers only to the
actual marks awarded by the examiner. ’Valuation’ is a process which does
not end on marks being awarded by an Examiner. Award of marks by the
Examiner is only one stage of the process of valuation. Moderation when
employed by the examining authority, becomes part of the process of
valuation and the marks awarded on moderation become the final marks of
the candidate. In fact Rule 20(3) specifically refers to the ’marks finally
awarded to each candidate in the written examination’, thereby implying that
the marks awarded by the examiner can be altered by moderation.
21. But the question is whether the raw marks which are converted into
scaled scores on an artificial scale which assumed variables (assumed mean
marks and assumed standard deviation) can be considered as ’marks finally
awarded’ or ’marks obtained’. Scaled scores are not marks awarded to a
candidate in a written examination, but a figure arrived at for the purpose of
being placed on a common scale. It can vary with reference to two arbitrarily
fixed variables, namely ’Assumed Mean’ and ’Assumed Standard Mean’.
We have dealt with this aspect in greater detail while dealing with question
(iii). For the reasons given while considering question (iii), we hold that
’scaled scores’ or ’scaled marks’ cannot be considered to be ’marks awarded
to a candidate in the written examination’. Therefore, scaling violates Rule
20(3) and Note (i) of Appendix-II of Judicial Service Rules.
22. Rule 20 of Judicial Service Rules requires the Commission to call for
interview such number of candidates, who in its opinion have secured the
minimum marks fixed by it. Because of application of scaling system by the
Commission, it has not been possible for the Commission to fix such
minimum marks either for individual subjects or for the aggregate. In the
absence of minimum marks, several candidates who secured less than 30%
in a subject have been selected. We note below by way of illustration, the
particulars of some candidates who have been selected in spite of securing
less than 20% in a subject :
S.
No.
Roll No.
Subject
Actual
Marks
(in %)
Scaled
Marks
Rank in
Selection
1.http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 9 of 24
2.
3.
4.
5.
6.
7.
8.
9.
012610
032373
002454
008097
017808
010139
012721
002831
004998
Language
Language
Language
Language
Law-I
Language
Law-I
Language
Language
8%
8%
11%
13%
13%
14%
15%
16%
17%
79
79
79
89
76
85
100
89
91
225
290
196
85
317
333
172
263
161
Thus scaling system adopted by the Commission, contravenes Rule 20(1)
also.
Re : Question (iii) :
23. When a large number of candidates appear for an examination, it is
necessary to have uniformity and consistency in valuation of the answerscripts. Where the number of candidates taking the examination are limited
and only one examiner (preferably the paper-setter himself) evaluates the
answer-scripts, it is to be assumed that there will be uniformity in the http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 10 of 24
valuation. But where a large number of candidates take the examination, it
will not be possible to get all the answer-scripts evaluated by the same
examiner. It, therefore, becomes necessary to distribute the answer-scripts
among several examiners for valuation with the paper-setter (or other senior
person) acting as the Head Examiner. When more than one examiner
evaluate the answer-scripts relating to a subject, the subjectivity of the
respective examiner will creep into the marks awarded by him to the answerscripts allotted to him for valuation. Each examiner will apply his own
yardstick to assess the answer-scripts. Inevitably therefore, even when
experienced examiners receive equal batches of answer scripts, there is
difference in average marks and the range of marks awarded, thereby
affecting the merit of individual candidates. This apart, there is ’HawkDove’ effect. Some examiners are liberal in valuation and tend to award
more marks. Some examiners are strict and tend to give less marks. Some
may be moderate and balanced in awarding marks. Even among those who
are liberal or those who are strict, there may be variance in the degree of
strictness or liberality. This means that if the same answer-script is given to
different examiners, there is all likelihood of different marks being assigned.
If a very well written answer-script goes to a strict examiner and a mediocre
answer-script goes to a liberal examiner, the mediocre answer-script may be
awarded more marks than the excellent answer-script. In other words, there
is ’reduced valuation’ by a strict examiner and ’enhanced valuation’ by a
liberal examiner. This is known as ’examiner variability’ or ’Hawk-Dove
effect’. Therefore, there is a need to evolve a procedure to ensure uniformity
inter se the Examiners so that the effect of ’examiner subjectivity’ or
’examiner variability’ is minimised. The procedure adopted to reduce
examiner subjectivity or variability is known as moderation. The classic
method of moderation is as follows :
(i) The paper-setter of the subject normally acts as the Head Examiner
for the subject. He is selected from amongst senior
academicians/scholars/senior civil servants/Judges. Where the case of
a large number of candidates, more than one examiner is appointed
and each of them is allotted around 300 answer-scripts for valuation.
(ii) To achieve uniformity in valuation, where more than one examiner is
involved, a meeting of the Head Examiner with all the examiners is
held soon after the examination. They discuss thoroughly the question
paper, the possible answers and the weightage to be given to various
aspects of the answers. They also carry out a sample valuation in the
light of their discussions. The sample valuation of scripts by each of
them is reviewed by the Head Examiner and variations in assigning
marks are further discussed. After such discussions, a consensus is
arrived at in regard to the norms of valuation to be adopted. On that
basis, the examiners are required to complete the valuation of answer
scripts. But this by itself, does not bring about uniformity of
assessment inter se the examiners. In spite of the norms agreed, many
examiners tend to deviate from the expected or agreed norms, as their
caution is overtaken by their propensity for strictness or liberality or
erraticism or carelessness during the course of valuation. Therefore,
certain further corrective steps become necessary.
(iii) After the valuation is completed by the examiners, the Head Examiner
conducts a random sample survey of the corrected answer scripts to
verify whether the norms evolved in the meetings of examiner have
actually been followed by the examiners. The process of random
sampling usually consists of scrutiny of some top level answer scripts
and some answer books selected at random from the batches of
answer scripts valued by each examiner. The top level answer books
of each examiner are revalued by the Head Examiner who carries out
such corrections or alterations in the award of marks as he, in his
judgment, considers best, to achieve uniformity. (For this purpose, if
necessary certain statistics like distribution of candidates in various
marks ranges, the average percentage of marks, the highest and lowest
award of marks etc. may also be prepared in respect of the valuation http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 11 of 24
of each examiner.)
(iv) After ascertaining or assessing the standards adopted by each
examiner, the Head Examiner may confirm the award of marks
without any change if the examiner has followed the agreed norms, or
suggest upward or downward moderation, the quantum of moderation
varying according to the degree of liberality or strictness in marking.
In regard to the top level answer books revalued by the Head
Examiner, his award of marks is accepted as final. As regards the
other answer books below the top level, to achieve maximum measure
of uniformity inter se the examiners, the awards are moderated as per
the recommendations made by the Head Examiner.
(v) If in the opinion of the Head Examiner there has been erratic or
careless marking by any examiner, for which it is not feasible to have
any standard moderation, the answer scripts valued by such examiner
are revalued either by the Head Examiner or any other Examiner who
is found to have followed the agreed norms.
(vi) Where the number of candidates is very large and the examiners are
numerous, it may be difficult for one Head Examiner to assess the
work of all the Examiners. In such a situation, one more level of
Examiners is introduced. For every ten or twenty examiners, there will
be a Head Examiner who checks the random samples as above. The
work of the Head Examiners, in turn, is checked by a Chief Examiner
to ensure proper results.
The above procedure of ’moderation’ would bring in considerable
uniformity and consistency. It should be noted that absolute uniformity or
consistency in valuation is impossible to achieve where there are several
examiners and the effort is only to achieve maximum uniformity.
24. In the Judicial Service Examination, the candidates were required to
take the examination in respect of the all five subjects and the candidates did
not have any option in regard to the subjects. In such a situation, moderation
appears to be an ideal solution. But there are examinations which have a
competitive situation where candidates have the option of selecting one or
few among a variety of heterogenous subjects and the number of students
taking different options also vary and it becomes necessary to prepare a
common merit list in respect of such candidates. Let us assume that some
candidates take Mathematics as an optional subject and some take English as
the optional subject. It is well-recognised that a mark of 70 out of 100 in
mathematics does not mean the same thing as 70 out of 100 in English. In
English 70 out of 100 may indicate to an outstanding student whereas in
Mathematics, 70 out of 100 may merely indicate an average student. Some
optional subjects may be very easy, when compared to others, resulting in
wide disparity in the marks secured by equally capable students. In such a
situation, candidates who have opted for the easier subjects may steal an
advantage over those who opted for difficult subjects. There is another
possibility. The paper setters in regard to some optional subjects may set
questions which are comparatively easier to answer when compared some
paper setters in other subjects who set tougher questions difficult to answer.
This may happens when for example, in a Civil Service examination, where
Physics and Chemistry are optional papers, examiner ’A’ sets a paper in
Physics appropriate to a degree level and examiner ’B’ sets a paper in
Chemistry appropriate for matriculate level. In view of these peculiarities,
there is a need to bring the assessment or valuation to a common scale so
that the inter se merit of candidates who have opted for different subjects,
can be ascertained. The moderation procedure referred to in the earlier para
will solve only the problem of examiner variability, where the examiners are
many, but valuation of answer scripts is in respect of a single subject.
Moderation is no answer where the problem is to find inter se merit across
several subjects, that is, where candidates take examination in different
subjects. To solve the problem of inter se merit across different subjects,
statistical experts have evolved a method known as scaling, that is creation http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 12 of 24
of scaled score. Scaling places the scores from different tests or test forms
on to a common scale. There are different methods of statistical scoring.
Standard score method, linear standard score method, normalized equipercentile method are some of the recognized methods for scaling.
25. A. Edwin Harper Jr. & V Vidya Sagar Misra in their publication
"Research on Examinations in India" have tried to explain and define
scaling. We may usefully borrow the same. A degree ’Fahrenheit’ is
different from a degree ’Centigrade’. Though both express temperature in
degrees, the ’degree’ is different for the two scales. What is 40 Degrees in
Centigrade scale is 104 Degrees in Fahrenheit scale. Similarly, when marks
are assigned to answer-scripts in different papers, say by Examiner ’A’ in
Geometry and Examiner ’B’ in History, the meaning or value of the ’mark’
is different. Scaling is the process which brings the mark awarded by
Examiner ’A’ in regard to Geometry scale and the mark awarded by
Examiner ’B’ in regard to History scale, to a common scale. Scaling is the
exercise of putting the marks which are the results of different scales
adopted in different subjects by different examiners into a common scale so
as to permit comparison of inter se merit. By this exercise, the raw marks
awarded by the examiner in different subjects is converted to a ’score’ on a
common scale by applying a statistical formula. The ’raw marks’ when
converted to a common scale are known as the ’scaled marks’. Scaling
process, whereby raw marks in different subjects are adjusted to a common
scale, is a recognized method of ensuring uniformity inter se among the
candidates who have taken examinations in different subjects, as, for
example, the Civil Services Examination.
26. The Union Public Service Commission (’UPSC’ for short) conducts
the largest number of examinations providing choice of subjects. When
assessing inter se merit, it takes recourse to scaling only in civil service
preliminary examination where candidates have the choice to opt for any one
paper out of 23 optional papers and where the question papers are of
objective type and the answer scripts are evaluated by computerized/
scanners. In regard to compulsory papers which are of descriptive
(conventional) type, valuation is done manually and scaling is not resorted
to. Like UPSC, most examining authorities appear to take the view that
moderation is the appropriate method to bring about uniformity in valuation
where several examiners manually evaluate answer-scripts of descriptive/
conventional type question papers in regard to same subject; and that scaling
should be resorted only where a common merit list has to be prepared in
regard to candidates who have taken examination of different subjects, in
pursuance of an option given to them.
27. But some Examining Authorities, like the Commission are of the view
that scaling can be used, not only where there is a need to find a common
base across different subjects (that is bringing the performance in different
subjects to a common scale), but also as an alternative to moderation, to
reduce examiner variability (that is where different examiners evaluate
answer scripts relating to the same subject).
28. Let us now examine the reasons as to why the Commission adopted
’scaling’ instead of moderation. The Committee states that the anomalies
caused on account of ’examiner variability’ was engaging its attention. It
found that a candidate’s score may depend upon the "chance’ factor of
whether his answers script is assessed by a lenient or a strict examiner; and
that in an extreme case, while a candidate of a given merit may get a First
Class/Division, another student of equal merit may be declared to have
failed. Therefore, the Commission constituted a Committee to carry out an
indepth study into the matter and suggest appropriate means to ensure that
the evaluation was on more equitable basis. The Committee by its Report
dated 2.9.1996 suggested statistical scaling system as the remedy and
recommended the linear standard score method which operates on the
following formula :
Z= Assumed mean + [ (X-M) x Assumed S.D.]http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 13 of 24
SD
Z= is the Scaled Score.
X = is the Raw mark.
M = is the mean of Raw Marks of the group/subject.
S.D. is the Standard Deviation of Raw Marks of the
group/subject.
The Committee suggested the following ’assumptions’ or ’parameters’ for
applying the formula :
(i) Assumed Mean will be taken as Half of the
maximum marks of the group/subject.
(ii) Assumed S.D. will be taken as one-fifth of the
assumed mean.
(iii) If scaled score is less than zero after scaling, then
candidates will be allotted zero marks in the said
group/subject.
(iv) If scaled score after scaling is more than maximum
marks, then candidate will be allotted maximum marks in
the said group/subject.
29. Eversince then, the Commission has been following the statistical
scaling. According to the Commission, the scaling method is rational,
scientific and reasonable and would lead to assessment of inter se merit of
the candidates in a just and proper manner. The use of the said method was
reviewed by an Expert Committee on 31.7.2000 and it was reiterated that the
formula and method presently used for scaling can be continued to be used
in future also and there was no need to change the same. Thus the scaling is
continued.
30. We may at this stage refer to the condition to be fulfilled, for scaling
to be effective. For this purpose, we are referring to passages from the
Authors/Experts relied on by the Commission itself.
30.1) A. Edwin Harper & Vidya Sagar Misra (in ’Research on
Examinations in India) make it clear that scaling will be useful and effective
only if the distribution of marks in the batch of answer scripts sent to each
examiner is approximately the same as the distribution of marks in the batch
of answer scripts sent to every other examiner.
30.2) A similar view is expressed by J.P. Guilford & Benjamin Fruchter (in
their treatise ’Fundamental Statistics in Psychology and Education’ page
476-477). They say that two conditions are to be satisfied to apply scaling :
(i) The population of students from which the distributions of scores arose
must be assumed to have equal means and dispersions in all the abilities
measured by the different tests; and (ii) the form of distribution, in terms of
skewness and kurtosis, must be very similar from one ability to another. He
proceeds to refer to the disadvantages of scaling thus :
"Unfortunately, we have no ideal scales common to all these tests, with
measurements which would tell us about these population parameters.
Certain selective features might have brought about a higher mean, a
narrower dispersion, and a negatively skewed distribution on the actual
continuum of ability measured by one test, and a lower mean, a wider
dispersion, and a symmetrical distribution on the continuum of another
ability represented by another test. Since we can never know definitely
about these features for any given population, in common scaling we often http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 14 of 24
have to proceed on the assumption that actual means, standard deviations,
and form of distribution are uniform for all abilities measured. In spite of
these limitations, it is almost certain that derived scales provide more
nearly comparable scales than do raw scores."
30.3) V. Natarajan & K. Gunasekaran in their treatise ’Scaling Techniques
\026 what, why and how’, have warned :
"If one studies the literature in this field, he can find that there are a
number of methods available ranging from simple to complex. Each has
its own merits and demerits and can be adopted only under certain
conditions or making certain assumptions."
The Authors describe the Linear Standard Score method (which is used by
the Commission) thus :
"Unlike Z-score (Standard score) which has a mean of ’zero’ and standard
deviation ’one’, the linear standard score has some pre-determined mean
and standard deviations.
\005..the choice of the mean and standard deviations is purely arbitrary.
Each has its own advantages and disadvantages and useful for specific
purpose only. It may be emphasized here that both the standard scores and
linear standard scores retain the shape of the original distribution of raw
marks. Therefore, if the original distribution is ’normally’ distributed, then
any type of Linear Standard Scores will also be ’normally’ distributed.
Taking the Normal Curve as the model, various points in other scales are
plotted. It should be, however, noted that the kind of relationship shown in
Figure -2 between normal curve vis-‘-vis the other scores are valid only if
the raw score distribution can be assumed to approximately normally
distributed.
(emphasis supplied)
30.4) The Kothari Report, 1976 (’Policy & Selection Methods’ published
by UPSC) while referring to scaling in regard to papers in different subjects,
by using appropriate statistical techniques as a recognized procedure for
improving the reliability of examination as a tool for selection, however
cautions that the method should be under continuous review and evaluation,
that continuing improvement in the light of experience and new
developments, taking into account advancement of knowledge, is essential.
31. The entire basis for applying scaling in regard to marks awarded by
different examiners in the same subject is the assumption that all answer
scripts have been thoroughly mixed, and that equal number of answer scripts
drawn at random and sent to each examiner for valuation will contain
answer scripts of candidates with equal distribution of abilities. When the
distribution of abilities in each batch is approximately equal, the mean
marks and standard deviation of the scaled marks of each batch will be
identical. To put it differently, if each examiner is sent 300 answer scripts
and each batch of 300 candidates have almost equal number of good,
average and poor standard students, they can all be brought to a common
scale for comparing their merit inter se. But we find that there is no such
broad equal distribution in the examination with which we are concerned.
We find from the Tables furnished that the range of marks awarded and the
range of deviation have varied enormously from examiner to examiner in the
same subject. We extract below these ranges, which demonstrate the wide
diversity, in turn indicating that scaling method was inappropriate for http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 15 of 24
bringing uniformity in valuation :
Subject
No. of
Examiner
No. of Scripts
Examined
(range)
Mean Marks of
the examiner
(range)
Standard
Deviation of
marks allotted
(range)
Minimum
Marks
(awarded by
the
Examiner)
Maximum
Marks (awarded
by the
Examiner)
1. General
Knowledge
2. Language
3. Law-I
4. Law-II
5. Law-III
18
14
11
10
14
50 to 800
231 to 800
300 to 900
200 to 1402
150 to 1000
47.4 to 83.91
37.51 to 82.43
30.83 to 56.90
70.57 to 94.40http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 16 of 24
63.14 to 86.74
12.24 to 20.49
14.16 to 31.75
12.45 to 17.85
11.48 to 20.05
13.16 to 19.54
10 to 43
0 to 30
0 to 10
0 to 40
0 to 31
84 to 126
105 to 145
83 to 113
113 to 132
99 to 134
32. The formula heavily relies upon the standard deviation among the
candidates in a given pool or batch. The standard deviation is a measure of
the range and distribution of marks awarded by an examiner. It depends on
the set of students in any given pool. If an examiner has a set of extremely
good or poor standard candidates and another examiner has a more even set
of average candidates, the standard deviation would be high for the first
examiner and low for the second examiner, having regard to the range of
distribution of marks. Consequently the scaled marks of a candidate
calculated on a formula heavily relying on standard deviation, would be
based on the cumulative standard deviation of all the candidates in his pool
rather than the strictness or liberality of the examiner. Therefore, standard
deviation has only a bearing on ascertaining the range of capabilities of the
candidates in a given examination and in no way eliminates the anomalies
arising out of the strictness or liberality of the examiner. We may
demonstrate the fact that the scaled marks vary with reference to the extent
of standard deviation (and has nothing to do with the issue of strictness or
liberality of the examiner), from the following examples :
Actual
Marks
Average
(Mean) Marks
Strict Examiner No. I
Strict Examiner No. II
Standard
Deviation
Scaled Marks
Standard
Deviation
Scaled Marks
0http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 17 of 24
5
20
50
50
50
15
15
15
33
40
60
25
25
25
60
64
76
Actual
Marks
Average
(Mean) Marks
Liberal Examiner No. I
Liberal Examiner No. II
Standard
Deviation
Scaled Marks
Standard
Deviation
Scaled Marks
50
120
150
90
90
90
15
15
15
47
140
180
25
25
25
68
124
148
The reason given for introducing scaling is to cure the disparity on account
of strictness or liberality of the examiners. But the effect of the scaling
formula adopted by Commission is to average the marks of a batch of
candidates and convert the raw marks of each candidate in the batch into
scaled marks with reference to the average marks of the batch and the
standard deviation. The scaling formula therefore, does not address or rectify
the effect of strictness or liberality of the examiner. The scaling formula is
more suited and appropriate to find a common base and inter se merit, where
candidates take examinations in different subjects. As the scaling formula
has no nexus or relevance to give a solution to the problem of eliminating
the variation or deviation in the standard of valuation of answer scripts by
different examiners either on account of strictness or liberality, it has to be
concluded that scaling is based on irrelevant considerations and ignores
relevant considerations. http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 18 of 24
33. We will next refer to apparent anomalies which show scaling of marks
is arbitrary. The Commission has furnished five Tables relating to the five
subjects showing the following particulars : (i) The number of examiners,
(ii) Number of answer scripts allotted to each examiner; (iii) Mean marks of
each examiner; (iv) Standard deviation of the marks allotted by each
examiner; (v) Minimum raw marks secured by a candidate in the batch of
answer-scripts corrected by each examiner; (vi) Maximum raw marks
secured by a candidate in the batch of answer-scripts corrected by each
examiner. The Commission has also furnished the tabulation of scaled and
actual marks of all the candidates. An examination of the particulars
furnished discloses several glaring anomalies.
I. Award of high scaled marks to those who secured zero marks :
We find from Table-II (furnished by the Commission) that the answer scripts
relating to Language Paper were distributed among 14 examiners. Several
candidates whose papers were evaluated by examiners 2, 3, 4, 5, 6, 8, 13, &
14 have secured zero marks. Evidently only those who did not attempt any
answer or had absolutely no knowledge of either Hindi or English would
have got zero marks. But such candidates who actually secured zero marks
have strangely been assigned scaled marks ranging from 36 to 67, depending
upon the examiner, in whose pool, they fell. We give below scaled marks
obtained by different candidates who secured zero marks with reference to
the examiners.
Subject : Language
Examiner
No.
Raw Marks of the
candidate
Scaled Marks
2
0
(100)+(0-66.58 x20) = 44
23.73
3
0
100+(0-55.29 x20) = 47
20.91
4
0
100+(0-74.88 x20) = 0 (-5 to be taken as zero)
14.20
5
0
100+(0-44.48 x20) = 58
20.06
6
0
100+(0-61.52 x20) = 50
24.8
8
0
100+(0-52.86 x20) = 67
31.75
13
0
100+(0-43.11 x 20) = 66
25.50
14
0http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 19 of 24
100+(0-54.77 x20) = 36
17.02
But unfortunately in the same subject, candidates who secured 32 to 30
marks, assessed by Examiner No.10, got their marks reduced to 31 to 28 on
scaling. (Mean being 80.93 and SD being 14.16). The devastating effect of
awarding such high scaled marks, that too ranging from 36 to 67, to those
who have secured ’0’ need not be stressed. In fact UPSC has clarified that
whenever they follow scaling procedure, no scaling is applied to ’0’ marks.
But the Commission had not applied its mind to this aspect when applying
’scaling’.
II. Equalization of marks of persons who secured very high marks.
The scaling has equalized the different high end marks of candidates, where
the mean marks is low. To give a hypothetical example if the mean marks is
70 and the standard deviation is 15, all candidates securing raw marks 145 to
200 will be assigned the equal scaled marks of 200. If the mean marks are 60
and the standard deviation is 15, all candidates securing 135 to 200 will be
awarded the scaled marks of 200. Similarly, if the mean marks are 80 and
the standard deviation is 20, all candidates securing raw marks between 180
to 200 will be awarded equal scaled marks of 200. In addition to the above
hypothetical examples, we may give a concrete example. In regard to
Examiner No. 14 in Language Paper, Table-II shows that the highest marks
secured is 145. In regard to that examiner, the mean marks is 54.77 and
standard deviation is 17.02. By applying the scaling formula, the marks of
145 secured by that candidate becomes 206 which is taken as 200 as per the
formula. All candidates who were awarded raw marks of 140 to 145 by
Examiner No. 14 in Language paper will be assigned the equal scaled marks
of 200. This leads to unequals being treated as equals. In case of candidates
securing marks in higher ranges on scaling, there is likelihood of their marks
being equalised with those who secured lesser marks thereby losing the
benefit of their higher marks and inter se merit.
III. Equalization of marks of persons who secured low marks.
The scaling has also equalized the different low end marks of candidates,
where the mean marks is high. To give a hypothetical example, if the mean
marks is 95 and the standard deviation is 11, then all candidates securing 40
and below will be awarded only ’0’. To give a concrete example, in regard
to Examiner No. 7 in Law Paper-II, one candidate has secured 32. In respect
of that examiner, the mean marks is 94.4 and standard deviation is 11.48. By
applying the scaling formula, the scaled marks of the said candidate who
secured 32 becomes ’0’. Not only that. Scaled marks of all candidates who
were given raw marks of 37 and less by that examiner, becomes ’0’. This
leads to unequals being treated as equals and candidates who secured marks
in the lower ranges (from that examiner) losing out to candidates who
performed much worse but were in the pool of other examiners.
IV. Inadequate mixing of answer scripts and improper distribution of
answer scripts :
The basic requirement for scaling is that all answer scripts will be mixed
thoroughly and that approximately equal number of answer scripts drawn at
random will be allotted to each examiner so as to infer equal distribution of
ability of candidates in each batch of answer scripts. But that was
apparently not done by the Commission. We give below the details of
distribution of answer scripts which demonstrate that they were nowhere
equal :
General Knowledge Paper (18 Examiners) \026 The distribution of answer
scripts is : 50 papers (2 examiners), 100 (3 examiners), 150 (1 examiner),
200 (2 examiners), 250 (2 examiners), 300 (1 examiner), 350 (1 examiner), http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 20 of 24
400 (1 examiner), 500 (2 examiners), 648 (1 examiners) and 800 (2
examiners).
Language Paper (14 Examiners) \026 The distribution of answer scripts is :
231 papers (1 examiner), 300 (5 examiners), 350 (1 examiner), 400 (2
examiners), 450 (3 examiners), 700 (1 examiner), 800 (1 examiner).
Law Paper-I (11 Examiners) - The distribution of answer scripts is : 100
papers (1 examiner), 300 (2 examiners), 400 (2 examiners), 450 (1
examiner), 600 (1 examiner), 700 (1 examiner), 775 (1 examiner), 800 (1
examiner), 900 (1 examiner).
Law paper-II (10 examiners) - The distribution of answer scripts is : 200
papers (1 examiner), 300 (1 examiner), 350 (1 examiner), 450 (1 examiner),
500 (2 examiners), 650 (2 examiners), 700 (1 examiner), 1402 (1 examiner).
Law paper-III (14 examiners) \026 The distribution of answer scripts is : 150
papers (3 examiners), 200 (1 examiner), 250 (1 examiner), 300 (1 examiner),
350 (2 examiners), 400 (1 examiner), 444 (1 examiner), 500 (1 examiner),
550 (1 examiner), 900 (1 examiner), 1000 (1 examiner).
Very large variation in the number of answer scripts allotted to each
examiner has a bearing on the mean marks and the standard deviation. The
fact that there was no proper randomization and distribution is also evident
from the fact that though approximately equal number appeared in each
segment of 10000 from among the roll nos. 1 to 51524, selection is
inexplicably high in the first segment of roll nos. 1 to 10000. The particulars
of roll number segments and the number of persons who appeared for the
main examination from each segment are as follows :
Roll Numbers No. of Persons
1. 1-10000 1072
2. 10001 to 20000 1115
3. 20001 to 30000 1124
4. 30001 to 40000 1031
5. 40001 to 50000 1112
6. 50001 to 51524 170
If there was proper randomization and distribution leading to equal
distribution of the candidate capacity, it would have been expected that the
number of selected candidates also would have been proportionate to each
segment. But we find that out of 347 candidates selected, as many as 139
candidates fall in first segment alone (within Roll nos. 1 to 10000) and 208
fall in the next five segments put together. Significantly out of the top 150
selected candidates, as many as 68 candidates also fall within Roll nos. 1 to
10000. Be that as it may.
V. Low raw marks were further lowered (or made into ’0’) and
higher raw marks were further increased due to scaling
Example : Law Paper-II.
Examiner No. 5 : 33 became 9; and 120 became 146
Examiner No. 6 : All marks between 9 and 1 became 0; and 119
became 139
Examiner No. 7 : All marks between 37 and 1 became 0; and 132
became 165
Examiner No. 9 : 4 became 0; and 122 became 156
In contrast, in some cases all raw marks whether low or high, became
higher.
Example : Law Paper-I.
Examiner No. 4 : 1 became 56; and 102 became 177.
Examiner No. 6 : 9 became 66; and 85 became 184.
Examiner No. 9 : 1 became 60; and 107 became 184.http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 21 of 24
Examiner No. 10 : 9 became 49; and 83 became 156.
The petitioners have referred to certain other absurdities arising from the
application of scaling, with reference to the results of 2000 examination
which was the subject matter of S.C. Dixit. (For example, it was
demonstrated that in some cases, the low marks awarded by liberal
examiners had increased and high marks awarded by strict examiners had
reduced, thereby achieving the opposite of the goal sought to be achieved --
that marks given by liberal examiners should be reduced and marks given by
strict examiners should be increased). We however consider it appropriate to
rely only on the anomalies/absurdities demonstrable with reference to the
2003 examination which is the subject matter of these petitions, and do not
propose to rely on the anomalies noticed in regard to the 2000 examination.
34. When selections are made on the basis of the marks awarded, and the
inter se ranking depends on the marks awarded, treating unequals equally, or
giving huge marks to candidates who have secured zero marks in some
subjects make the process wholly irrational, virtually bordering on
arbitrariness. It is no doubt true that such irrationality may adversely affect
only those cases which are at either end of the spectrum, and if they are
excluded, by and large the scaling system may be functional. But if the
extreme cases are even 20 out of 5000 for each of the subjects, it becomes
100 for 5 subjects, which means that the results of as many as 100 are likely
to be affected. It may be more also. In that process, at least 5% to 10% of the
vacancies are likely to be filled up by less meritorious candidates. This will
lead to considerable heart-burn and dissatisfaction. When the object of the
selection process is to try to select the best, and even one mark may make
the difference between selection or non-selection, the system of scaling
which has the effect of either reducing or increasing the marks in an
arbitrary manner will lead to unjust results. This is in addition to the main
disadvantage that scaling does not remedy the ill-effects of examiner
variability arising out of strictness or liberality in valuation.
35. The illustrations given above with reference to the 2003 examinations
clearly demonstrate the arbitrariness and irrationality of scaling, particularly
in cases falling at the two ends of the spectrum. We, therefore, hold that
scaling system as adopted by the Commission is unsuited for the Civil Judge
(Junior Division) Examination.
36. We may now summarize the position regarding scaling thus :
(i) Only certain situations warrant adoption of scaling techniques.
(ii) There are number of methods of statistical scaling, some simple
and some complex. Each method or system has its merits and
demerits and can be adopted only under certain conditions or
making certain assumptions.
(iii) Scaling will be useful and effective only if the distribution of
marks in the batch of answer scripts sent to each examiner is
approximately the same as the distribution of marks in the batch
of answer scripts sent to every other examiner.
(iv) In the Linear Standard Method, there is no guarantee that the
range of scores at various levels will yield candidates of
comparative ability.
(v) Any scaling method should be under continuous review and
evaluation and improvement, if it is to be a reliable tool in the
selection process.
(vi) Scaling may, to a limited extent, be successful in eliminating
the general variation which exists from examiner to examiner,
but not a solution to solve examiner variability arising from the
’hawk-dove’ effect (strict/liberal valuation). http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 22 of 24
The material placed does not disclose that the Commission or its expert
committee have kept these factors in view in determining the system of
scaling. We have already demonstrated the anomalies/absurdities arising
from the scaling system used. The Commission will have to identify a
suitable system of evaluation, if necessary by appointing another Committee
of Experts. Till such new system is in place, the Commission may follow the
moderation system set out in Para 23 above with appropriate modifications.
37. We may now refer to the decision of this Court in S. C. Dixit. The
validity of scaling was considered in paras 31 to 33 of the judgment
extracted below :
"31. There is a vast percentage difference in awarding of marks between
each set of examiners and this was sought to be minimized by applying the
scaling formula. If scaling method had not been used, only those
candidates whose answer-sheets were examined by liberal examiners
alone would get selected and the candidates whose answer-sheets were
examined by strict examiners would be completely excluded, though the
standard of their answers may be to some extent similar. The scaling
system was adopted with a view to eliminate the inconsistency in the
marking standards of the examiners. The counsel for the respondents
could not demonstrate that the adoption of scaling system has in any way
caused injustice to any meritorious candidate. If any candidate had secured
higher marks in the written examination, even by applying scaling
formula, he would still be benefited.
32. The Division Bench of the High Court observed that the process of
scaling was done examiner-wise only and the scaling formula did not take
into consideration the average of mean of all the candidates in one
particular paper but took the mean of only that group of candidates which
has been examined by one single examiner. The counsel for U.P. PSC
submitted that the observation made by the High Court is incorrect. The
scaling formula was adopted to remove the disparity in the evaluation of
14 examiners who participated in the evaluation of answer-sheets and the
details have also been furnished as to how the scaling formula was
adopted and applied. Therefore, we do not think that the observation of the
Division Bench that the Commission did not take care of varying
standards which may have been applied by different examiners but has
sought to reduce the variation of the marks awarded by the same examiner
to different candidates whose answer-sheets had been examined, is
correct. The Division Bench was of the view that as a result of scaling, the
marks of the candidates who had secured zero marks were enhanced to 18
and this was illegal and thus affected the selection process. The finding is
to be understood to mean as to how the scaling system was applied. 18
marks were given notionally to a candidate who secured zero marks so as
to indicate the variation in marks secured by the candidates and to fix the
mean marks.
33. In that view of the matter, we do not think that the application of
scaling formula to the examinations in question was either arbitrary or
illegal. The selection of the candidates was done in a better way.
Moreover, this formula was adopted by U.P. PSC after an expert study and
in such matters, the court cannot sit in judgment and interfere with the
same unless it is proved that it was an arbitrary and unreasonable exercise
of power and the selection itself was done contrary to the Rules.
Ultimately, the agency conducting the examination has to consider as to
which method should be preferred and adopted having regard to the
myriad situations that may arise before them."
S. C. Dixit, therefore, upheld scaling on two conclusions, namely (i) that the
scaling formula was adopted by the Commission after an expert study and in
such matters, court will not interfere unless it is proved to be arbitrary and
unreasonable; and (ii) the scaling system adopted by the Commission http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 23 of 24
eliminated the inconsistency arising on account of examiner variability
(differences due to evaluation by strict examiners and liberal examiners). As
scaling was a recognized method to bring raw marks in different subjects to
a common scale and as the Commission submitted that they introduced
scaling after a scientific study by experts, this Court apparently did not want
to interfere. This Court was also being conscious that any new method, when
introduced, required corrections and adjustments from time to time and
should not be rejected at the threshold as unworkable. But we have found
after an examination of the manner in which scaling system has been
introduced and the effect thereof on the present examination, that the system
is not suitable. We have also concluded that there was no proper or adequate
study before introduction of scaling and the scaling system which is
primarily intended for preparing a common merit list in regard to candidates
who take examinations in different optional subjects, has been
inappropriately and mechanically applied to a situation where the need is to
eliminate examiner variability on account of strict/liberal valuation. We
have found that the scaling system adopted by the Commission leads to
irrational results, and does not offer a solution for examiner variability
arising from strict/liberal examiners. Therefore, it can be said that neither of
the two assumptions made in S.C. Dixit can validly continue to apply to the
type of examination with which we are concerned. We are therefore of the
view that the approval of the scaling system in S.C. Dixit is no longer valid.
38. Learned counsel for the Commission contended that scaling has been
accepted as a standard method of evaluation in the following decisions and
therefore it should be approved :-
(i) Kamlesh Haribhai Goradia vs. Union of India [1987 (1) Guj.LR
157], upheld by this Court by order dated 11.3.1987 in SLP (C)
No. 14000/1986.
(ii) Muhesh Kumar Khandelwal vs. State of Rajasthan [1994 (1)
Raj.LR 533] upheld by this Court by order dated 22.1.1996 in
SLP(c) No. 15682-15684 of 1994.
(iii) K. Channegowda vs. Karnataka Public Service Commission
[2005(12) SCC 688).
All the three cases related to moderation and not scaling. There are,
however, passing references to scaling as one of the methods to achieve
common standard of assessment. The fact that scaling is a standard method
of assessment, when a common base has to be found for comparative
assessment of candidates taking examinations in different optional subjects,
is not in dispute. In fact the Commission may continue to adopt the said
system of scaling, where a comparative assessment is to be made of
candidates having option to take different subjects. The question is whether
scaling, in particular, linear standard scaling system as adopted by the
Commission, is a suitable process to eliminate ’examiner variability’ when
different examiners assess the answer scripts relating to the same subject.
None of the three decisions is of any assistance to approve the use of method
of ’scaling’ used by the Commission.
39. Learned counsel for the Commission also referred to several decisions
in support of its contention that courts will be slow to interfere with matters
affecting policy requiring technical expertise and leave them for decision of
experts. (State of U.P. v. Renusagar Power Co. Ltd. - 1988 (4) SCC 59, Tata
Iron & Steel Co. Ltd. v. Union of India \026 1996 (9) SCC 709, Federation of
Railway Officers Association v. Union of India \026 2003 (4) SCC 289). There
can be no doubt about the said principle. But manifest arbitrariness and
irrationality is an exception to the said principle. Therefore, the said
decisions are of no avail.
40. We should, however, record the fair submission on behalf of the
Commission that it is not irrevocably committed to any particular system
and will adopt a different or better system if the present system is found to
be defective.
Re : Point No. (iv). http://JUDIS.NIC.IN SUPREME COURT OF INDIA Page 24 of 24
41. The petitioners have requested that their petitions should be treated as
being in public interest and the entire selection process in regard to Civil
Judge (Junior Division) Examination, 2003 should be set aside. We are
unable to accept the said contention. What has been made out is certain
inherent defects of a particular scaling system when applied to the selection
process of the Civil Judges (Junior Division) where the problem is one of
examiner variability (strict/liberal examiners). Neither mala fides nor any
other irregularities in the process of selection is made out. The Commission
has acted bona fide in proceeding with the selection and neither the High
Court nor the State Government had any grievance in regard to selections. In
fact, the scaling system applied had the seal of approval of this Court in
regard to the previous selection in S.C. Dixit (supra). The selected candidates
have also been appointed and functioning as Judicial Officers. Further as
noticed above, the scaling system adopted by the Commission has led to
irrational and arbitrary results only in cases falling at the ends of the
spectrum, and by and large did not affect the major portion of the selection.
We, therefore, direct that our decision holding that the scaling system
adopted by the Commission is unsuited in regard to Civil Judge (Junior
Division) Examination and directing moderation, will be prospective in its
application and will not affect the selections and appointments already made
in pursuance of the 2003 Examination.
42.. However, in so far as the petitioners are concerned, we deem it proper
to issue the following directions to do complete justice on the facts of the
case :
a) If the aggregate of raw marks in the written examination and the
marks in the interview of any petitioner is less than that of the last
selected candidate in the respective category, he will not be entitled to
any relief (for example, the petitioners in WP(C) No. 165/2005
belonging to the Category ’BC’ have secured raw marks of 361 and
377 respectively in the written examinations, whereas the last five of
the selected candidates in that category have secured raw marks of
390, 391, 397, 438 and 428 respectively. Even after adding the
interview marks, the marks of the petitioners in W.P. [C] No.165/2005
is less than the marks of the selected candidates).
b) Where the aggregate of raw marks in the written examination and the
interview marks of any petitioner, is more than the aggregate of the
raw marks in the written examination and interview marks of the last
selected candidate in his category, he shall be considered for
appointment in the respective category by counting his appointment
against future vacancies. (For example, we find that petitioner Archna
Rani, one of the petitioners in WP (C) No. 467/2005 has secured 384
raw marks which is more than the raw marks secured by the last five
selected candidates [347, 337, 336, 383 and 335] under the SC
category and even after adding the interview marks, her marks are
more than the five selected candidates. Hence, she should be
considered for appointment). This relief will be available only to such
of the petitioners who have approached this Court and the High Court
before 31st August, 2005.
43. The petitions are allowed in part accordingly.
No comments:
Post a Comment