| 1. |
- Alger, Susanne, 1967-
(författare)
-
Is This Reliable Enough? : Examining Classification Consistency and Accuracy in a Criterion-Referenced Test
- 2016
-
Ingår i: International journal of assessment tools in education. - : International Journal of Assessment Tools in Education (IJATE). - 2148-7456. ; 3:2, s. 137-150
-
Tidskriftsartikel (refereegranskat)abstract
- One important step for assessing the quality of a test is to examine the reliability of test score interpretation. Which aspect of reliability is the most relevant depends on what type of test it is and how the scores are to be used. For criterion-referenced tests, and in particular certification tests, where students are classified into performance categories, primary focus need not be on the size of error but on the impact of this error on classification. This impact can be described in terms of classification consistency and classification accuracy. In this article selected methods from classical test theory for estimating classification consistency and classification accuracy were applied to the theory part of the Swedish driving licence test, a high-stakes criterion-referenced test which is rarely studied in terms of reliability of classification. The results for this particular test indicated a level of classification consistency that falls slightly short of the recommended level which is why lengthening the test should be considered. More evidence should also be gathered as to whether the placement of the cut-off score is appropriate since this has implications for the validity of classifications.
|
|
| 2. |
- Alger, Susanne, 1967-, et al.
(författare)
-
Jämförelser mellan provorter : en studie av körprov för behörighet B
- 2016
-
Rapport (övrigt vetenskapligt/konstnärligt)abstract
- Man kan köra upp på fler än 130 orter i Sverige, alla med olika trafikmiljöer, förarprövare och provtagare. Provtagarnas ålder och anmälningssätt skiljer sig åt på olika orter. På några orter genomförs få prov, på andra många. En del förarprövare genomför väldigt få prov, andra väldigt många. Vissa orter saknar vissa inslag i trafikmiljön, som exempelvis motorväg, järnvägskorsning eller cirkulationsplats. Trafikintensiteten på orterna skiljer sig åt. Kort sagt, det finns väldigt många aspekter som kan variera mellan enskilda prov. I rapporten presenteras olika sätt att åskådliggöra hur dessa skillnader mellan orter ser ut och några sätt att kategorisera orter.Får detta konsekvenser för resultatet? Det är svårt att utreda eftersom flera variabler sannolikt samverkar och det är svårt att särskilja dem från varandra, och eftersom data ser ut som det gör. Dels är utfallet binärt – godkänt/underkänt – vilket begränsar vilka analyser som kan göras. Dels är det många kategorier inom variablerna ort och förarprövare. För att minska antalet kategorier kan man slå ihop dem till färre och större kategorier. Frågan är i så fall utifrån vilka kriterier och till hur många kategorier. Av analyser framgår att provinnehållet skiljer sig åt mellan prov, såväl inom som mellan orter. Det är dock oklart om man mäter samma eller olika kvaliteter på olika orter, men det är nog inte så enkelt att säga att det är "lättare" att ta körkort på vissa orter än andra baserat på enkla jämförelser av godkännandegrad. Provtagarnas ålder och anmälningssätt, aspekter som vi vet påverkar godkännandegraden, skiljer sig också åt mellan orter.
|
|
| 3. |
- Alger, Susanne, 1967-
(författare)
-
Licence to drive : the importance of reliability for the validity of the Swedish driving licence test
- 2019
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Background: The Swedish driving licence test is a criterion-referenced test resulting in a pass or fail. It currently consists of two parts - a theory test with 65 multiple-choice items and a practical driving test where at least 25 minutes are spent driving in traffic. It is a high-stakes test in the sense that the results are used to determine whether the test-taker should be allowed to drive a car without supervision. As the only other requirements for obtaining a licence is a few hours of hazard education (and a short introduction if you intend to drive with a lay instructor) it is important that the test result, in terms of pass or fail, is reliable and valid. If this is not the case it could have detrimental effects on traffic safety. Examining all relevant aspects is beyond the scope of this licentiate thesis so I have focused on reliability.Methods Reliability for both the theoretical and practical test results was examined. As these are very different types of tests the types of reliability examined also differed. In order to examine inter-rater reliability of the driving test 83 examiners were accompanied by one of five selected supervising examiners for a day of tests. All in all 535 tests were conducted with two examiners assessing the same performance. At the end of the day the examiners compared notes and tried to determine the reason for any inconsistencies. Both examiners and students also filled in questionnaires with questions about background and preparation. As for studying decision consistency and decision accuracy of the theory test, three test versions (a total of around 12,000 tests) were examined with the help of methods devised by Subkoviak (Subkoviak, 1976, 1988) and Hanson & Brennan (Brennan, 2004; Hanson & Brennan, 1990).Results The results from two research studies concerning reliability were presented. Study I focused on inter-rater reliability in the driving test and in 93 per cent of cases the examiners made the same assessment. For the tests where their opinions differed there was no correlation to any of the background variables or other variables examined except for three, which had logical explanations and did not constitute a problem. Although there were cases where the differences were due to different stances on matters of interpretation the most common suggested cause was the placement in the car (back seat vs. front seat). Although the supervising examiners gave both praise and criticism as to how the test was carried out the study does not answer the question whether the tests were equal in terms of composition and difficulty.In Study II the focus was on decision consistency and decision accuracy in the theory test. Three versions of the theory tests were examined and, on the whole, found to be fairly similar in terms of item difficulty and score distribution, but the mean was so close to the cut-score (i.e. the score required to pass) that the pass rate differed somewhat between versions. Agreement coefficients were around .80 for all test versions (between .79 and .82 depending on method). Classification accuracy indicated an .87 probability of a correct classification.Conclusion It is important to examine the reliability and validity of the driving licence test since a misclassification can have serious consequences in terms of traffic safety. In the studies included here the rate of agreement between examiners is deemed as satisfactory. It would be preferable if the classification consistency and classification accuracy, as estimated by the methods used, were higher for the theory test, given its importance.While reliability in terms of agreement between raters/examiners or consistency and accuracy of classification are routinely examined in other contexts, such as large-scale educational testing, this is not often done for the driving licence tests. At the same time, the methods used here can be transferred to contexts where such properties are generally not examined. Collecting information about test-takers and examiners, like in Study I, can provide evidence concerning possible bias.Examining to what extent decisions are consistent is one important aspect of collecting evidence that shows that test results can be used to draw conclusions about driver competence. Still, regardless of outcome, validation is a process that never ends. There is always reason to examine various aspects and make further improvements. There are also many other relevant aspects to examine. A prerequisite for the validity of the score interpretation of a criterion-referenced test like this one is that the cut-score is appropriate and the content relevant. This should therefore be the subject of further research as the validation process continues.
|
|
| 4. |
- Alger, Susanne, 1967-, et al.
(författare)
-
Samma prov - andra provtagare : en studie av förändringar i körprovsresultat och provtagargruppens sammansättning
- 2017
-
Rapport (övrigt vetenskapligt/konstnärligt)abstract
- The purpose of this study is to describe the trend regarding the pass rate for the driving test and examine to what degree changes of the pass rate can be linked to changes in the composition of the group of test-takers. Previously there have been no trend studies exploring changes with regard to the language of the theory test, i.e. whether the test-takers take the test in Swedish or another language so special attention will be given to this aspect.The composition of the group of test-takers has changed quite a lot over the last 20 years. Test-takers can either register for the test via a driving school or on their own. The percentage of tests registered via a driving school has decreased, and in later years the number of driving tests not registered via a driving school following theory tests in other languages than Swedish has increased. The pass rate among the tests that were not registered via a driving school has decreased more than for those registered via a driving school. If the results are weighted to reflect the distribution of groups as it was in 1998 or 2010 one may conclude that if the total group of test-takers had had the same composition in later years the pass rate would probably not have decreased as much.If theory tests are categorised according to whether they were carried out in Swedish or another language (translated or interpreted) results show that the percentage of tests in other languages has increased significantly since 2013. The percentage of driving tests following theory tests in other languages has also increased. Most of the driving tests after theory tests in other languages are carried out by test-takers who are 25 years or older and who have not registered for the test via a driving school. Such tests are also failed to a greater degree.Many more tests are administered today than 10 or 20 years ago. The number of test-takers increased by 87 per cent between 1998 and 2016 while the number of tests increased by 137 per cent. A contributing factor is the change in 2008 that all were allowed to take the driving test regardless of result on the theory test (previously only those who passed could). If results on the driving test are divided by result on the preceding theory test (pass/fail) it turns out that the pass rate has decreased at roughly the same rate since 2012 for both those who failed and those who passed the theory test. However, a larger proportion of those who have passed the theory test also pass the driving test.The changes in conditions for taking the driving test does not constitute the whole explanation. If comparing the number of driving tests in 2010 (when all were part of the new system) and 2016 the increase is 22 and 33 per cent respectively.When performing analyses based on test-takers rather than tests it is also clear that the number of test-takers who are 25 or older and not registered via a driving school is increasing. At least from 2013 this increase is almost entirely due to test-takers who have taken the theory test in other languages than Swedish. In 1998 24 per cent of the test-takers only took tests for which they had not registered via a driving school. In 2010 that percentage was 42 per cent an in 2016 53 per cent. Students who do not register for the test via a driving school also carry out more tests on average within the year than those who do.Test-takers who carry out many driving tests have, to a greater extent than others, had the examiner intervene (e.g. by braking) during their first test attempt, which perhaps is a consequence of not being properly prepared.To summarize: the trend with decreasing pass rate, an increasing proportion of older test-takers and of tests not registered via a driving school that has been visible over the last 20 years or so seems to be continuing. When looking at trends in a shorter perspective, the most notable change is the increase in theory tests that are taken in languages other than Swedish. These tests are, to a great extent, taken by older test-takers, not registered via a driving school, factors that are associated with a lower pass rate. Findings may inform policy makers and other stake-holders when it comes to understanding changes in the composition of test-takers and how possible efforts to increase pass rates could be targeted.
|
|
| 5. |
- Alger, Susanne, 1967-, et al.
(författare)
-
Trender 1998-2015 avseende förändringen av andelen godkända på körprov för körprovsbehörighet B
- 2016
-
Rapport (övrigt vetenskapligt/konstnärligt)abstract
- Data har samlats i samband med körproven under många år. Ett av de mått som presenterats är andel godkända prov och denna andel har minskat sedan 1998. I en tidigare rapport diskuterades faktorer som kan påverka resultatet i termer av förändringar i provtagargruppens sammansättning. Denna rapport redovisar utvecklingen under senare år och beskriver dessutom sammansättningen utifrån såväl personer som prov.Flera av resultaten presenteras utifrån sexton delgrupper. Indelningsgrunderna har varit ålder, kön och anmälningssätt. Inom respektive grupp har resultaten varit tämligen stabila över tid. Det är bland de privatanmälda provtagarna man kan se en nedåtgående trend. Samtidigt har andelen privatanmälda prov ökat.I och med att sammanhållet prov infördes 2008 fick även de som underkänts på kunskapsprovet genomföra körprov, vilket tidigare inte varit fallet. En sådan omställning får även konsekvenser för godkännandegraden som blev lite högre för de prov som föregåtts av godkända kunskapsprov och lite lägre för samtliga prov.Om man viktar resultaten för att återspegla den fördelning mellan grupper som rådde 1998 kan man konstatera att om provtagargruppen hade haft samma sammansättning under de följande åren skulle andelen godkända totalt ha varit högre under dessa år än vad som nu är fallet.Data har samlats i samband med körproven under många år. Ett av de mått som presenterats är andel godkända prov och denna andel har minskat sedan 1998. I en tidigare rapport diskuterades faktorer som kan påverka resultatet i termer av förändringar i provtagargruppens sammansättning. Denna rapport redovisar utvecklingen under senare år och beskriver dessutom sammansättningen utifrån såväl personer som prov.Flera av resultaten presenteras utifrån sexton delgrupper. Indelningsgrunderna har varit ålder, kön och anmälningssätt. Inom respektive grupp har resultaten varit tämligen stabila över tid. Det är bland de privatanmälda provtagarna man kan se en nedåtgående trend. Samtidigt har andelen privatanmälda prov ökat.I och med att sammanhållet prov infördes 2008 fick även de som underkänts på kunskapsprovet genomföra körprov, vilket tidigare inte varit fallet. En sådan omställning får även konsekvenser för godkännandegraden som blev lite högre för de prov som föregåtts av godkända kunskapsprov och lite lägre för samtliga prov.Om man viktar resultaten för att återspegla den fördelning mellan grupper som rådde 1998 kan man konstatera att om provtagargruppen hade haft samma sammansättning under de följande åren skulle andelen godkända totalt ha varit högre under dessa år än vad som nu är fallet.
|
|
