Guest post: Deveral Capps, Dean of Leeds Law School, and his colleagues Donna Brown, Jenni Taylor and Safina Zarif share their experiences with piloting eAssessment.    

On 19th September 2019 (19.9.19) Leeds Law School signed a contract with Inspera to pilot their eAssessment tools; by the end of November, a mere two months later, the Law School had delivered 5 formative assessments and 3 summative assessments using their system, amounting to almost 2000 students in total.

Before looking at the results of the pilot, it might be helpful to understand why Leeds Law School decided to invest in eAssessment.  A little while ago we inserted summative assessment points early in the academic year to check on student engagement and these fell outside of ‘normal’ University assessment periods.

As a result, finding a room (or rooms) to host exams for around 350 students during teaching time was difficult.  We faced stark choices, either pay commercial rates for rooms outside of the University, cancel the exams or assess our students in a different way; we chose the last.  As a large University, and one with excellent teaching accommodation, we are fortunate to have a considerable number of computer labs.  It was, therefore, a short journey from identifying free and available labs to running large-scale eAssessments within them.

There were some other subsidiary reasons that shored up our decision to embark on our eAssessment journey:

  • Overseas examinations would be easier to run as the security issues surrounding the export of hard copy assessment instruments and the return of scripts would disappear.
  • The use of multiple-choice questions, “MCQs”, present opportunities for a variety of questioning techniques.  For example: true/false questions; one correct answer and three (or more) incorrect distractors; single best answer questions; best and worst answer; and extended matching questions.
  • eAssessment can switch the assessment burden.  Whilst creating an assessment instrument might be more arduous, marking can be a lot easier and results made available more quickly.
  • Law student exposure to MCQ type assessments will be helpful given the planned introduction of the Solicitors Qualifying Examination that will use this method.
  • eAssessment can be more inclusive and more easily accommodate students who require reasonable adjustments.
  • When moving to eAssessment, a torch is shone on assessment scrutiny processes, thereby producing high levels of quality assurance.
  • eAssessment can produce immediate feedback, thereby closing the assessment loop swiftly.
  • There is growing desire to respond and implement technological advances in the learning and teaching environment and incorporate these into assessment processes – QAA benchmark statements.

Disappointment with our current eAssessment solution, where previous experiences had completely removed the staff appetite for eAssessment, meant we needed a new technological solution. Through conversations, testing and finally, and perhaps most importantly, a personal recommendation, we settled on a pilot with Inspera.

How did the staff get on?

Moving from hard copy to eAssessment is no mean feat; it takes time, effort and energy.  Following our decision to move to Inspera the Law School created a small project team; met and planned for all eventualities, created topic question/item banks, and checked, proofed, moderated, scrutinised and re-scrutinised each question.

At Leeds, we are fortunate to have a keen, innovative and conscientious group of both academic and support staff.  As such, calling in the extra effort, at short notice, was possible.  However, if we learned anything from the swift roll-out of eAssessment, we’d have preferred a little more time to plan and most importantly create the various MCT question banks.  Whilst you know it will take time, it will probably take a longer time than you think.

Significant worries revolved around assessment integrity and the protection of the question bank.  When you consider how long it takes to create a multiple-choice question, multiplied by the totality of the bank, losing these through either student impropriety or carelessness can be expensive.  Using the Safe Exam Browser, which locks down the student computer, helped allay these fears.

Informal feedback from staff was overwhelmingly positive. Our conclusion was that the system was excellent and where mistakes happened, these were because of human error rather than systemic ones. 

Our ‘lessons learned’ included:

  • It took longer for some staff to get to grips with the ‘system’ than others.
  • Some internet connectivity (Wi-Fi) issues impacted on our formative assessments using BYOD (bring your own device).  Inspera’s system of continuous ‘background’ saving worked a treat.
  • We didn’t use single sign-on, i.e where students can use their University login details to access the eAssessment system.  In retrospect this would have removed a significant administrative burden.
  • The comfort and cleanliness of a couple of the IT labs sometimes left a little to be desired – they probably would not have passed the mother-in-law test!

What did the students think?

We decided to take the eAssessment temperature and surveyed our students. We carefully considered who to survey and concluded the best cohort were our level 5 students studying property law.  This cohort had prior experience of paper-based MCQ assessment and had also been exposed to testing with our previous eAssessment provider.  We ran the survey immediately following their Inspera summative MCQ assessment and received feedback from 191 students.

The survey revealed that students found the Inspera eAssessment platform to be user friendly, that they were confident with using eAssessment and believed the system recorded their answers accurately. Students were satisfied that eAssessment was as rigorous as paper-based assessment and over 60% of students stated they would prefer more of their written exams to be available in an eAssessment format.

In fairness, some free text answers revealed a small minority of students who preferred ‘traditional’ hard copy assessments, with one writing ‘nothing beats paper and pen’!  However, overall the feedback received undeniably positive and the majority of students were delighted by their eAssessment experience.

Conclusion

With the benefit of hindsight, there are often things in life we might wish we had done differently, however, our decision to swiftly move to eAssessment was absolutely the right thing to do.  For those yet to embark on their eAssessment journey, integrating single sign-on is highly recommended and having a little longer to create MCQ question banks would have been helpful.

Our success with Inspera would not have been possible without an excellent team and so thanks must be duly paid to Adam Gulley, Malcolm Hirst, Dr Victoria Hamlyn and Dr Louisa Ashley.

Leeds Law School looks forward to more eAssessment developments in 2020 and beyond.

About the author

Deveral is the Dean of Leeds Law School. He joined Leeds Beckett in 2015 and has been involved in Higher Education for over 20 years, 18 of which were spent at Northumbria. As an expert in Quality Assurance, Deveral has a broad understanding of academic quality and assurance processes. He is an experienced external examiner, including lead external examiner, for all types of academic award and also solicitors’ Higher Rights.

You are welcome to visit Deveral Capps’ profile on LinkedIn or view his research profile.