Meeting with VU University’s student panel

On Thursday April 12th, Silvester Draaijer and Robin van der Wiel presented the OP4RE project and the concept of online proctored exams to the VU University’s student panel. As an introduction to the topic, the students watched a video made by ProctorExam in order to understand how the system works. Students quickly started thinking critically about online proctoring and asked questions such as “is someone going to watch all of the video footage?”, “how is my video footage processed?” and “why are toilet breaks not allowed during the online proctored exam?”. After Silvester had explained some more about the system, the floor was opened for a discussion of privacy matters concerning the system.

Most of the students noted that they would not feel comfortable with someone watching their every move. At a university exam location, there is one proctor for every 20 or so students; being watched one on one feels extreme. Students also recognized the positive aspects of the system. It allows students that are abroad or not able to come to a university location to make exams safely from a remote location. It can be concluded from the student panel that it is very important to inform students. They want to know how the system works and how their video footage is processed. The remarks of the student panel will be taken into consideration during the remainder of the OP4RE project.

Preparing students for an online proctored exam

As the project progresses and more assessments take place via online proctoring, we -the project team- have gained more experience. Communication with our students appears to be a critical issue in the process of organising and setting up the online proctoring of exams. As a result Fontys have developed a short video in which they explain in ‘student language’ what online proctoring is, how such assessments work, which videos are recorded and the overall process for online assessment  at Fontys. Tips and tricks, and do’s and don’ts are given. In addition, they have also developed a video to guide students through the start-up procedure of ProctorExam.

Watch the videos here (Dutch version):

OP4RE – ProctorExam and the Mathematics online Eligibility test 2018

Managing a diverse stream of incoming domestic or international students can be a real challenge for universities. That is why, in partnership with several Dutch universities, SOWISO will start offering upcoming first-year students an online mathematics test to show that their mastering of mathematics is sufficient to be eligible for enrollment.

To make sure that there’s no cheating, the test will be proctored with the help of ProctorExam and OP4RE; this means students will be monitored during the test.

The content of the test was constructed by a small group of university mathematics teaching experts to converge to a test that has broad acceptance with respect to validity and reliability in the Netherlands. The offering of this test as pilot has started at the beginning of may 2018. More details: take a look at our website.

Currently, the first candidates have taken this online test. Expressed in numbers we now see the following:

  • About 80 students of the Vrije Universiteit were offered the option to take either a mathematics eligibility test in the Netherlands or online.
  • About 35 students expressed their interest to take this test on the online platform of PassYourMath
  • About 20 students registered to take the online preparation modules for the test to become accustomed to the platform (content and level of difficulty of questions, method of entering answers).
  • About 10 students registered to take the online proctored test so far. About half of them had some kind of difficulty in setting up the test correctly. It turned out that because of some internet restriction imposed by the country in which the candidate resided, it was not always possible to use the webcam.
  • Not a single case of suspected fraud was identified. All students were issued a Certificate of Achievement by PassYourMath.
  • About 7 of these students achieved a sufficient score set as threshold for eligibility.
  • The first Certificate has now been verified as means to grant a student access to the Vrije Universiteit.

The development of this initiative shows that online proctoring serves well the goal of increasing access to higher education. This is exactly the main goal of the Erasmus funding. No processes have yet been automated with respect to registering students and have student sign-on the PassYourMath and ProctorExam with one account. This pilot however shows the demand by prospective students for online proctoring. The development also shows the feasibility of the applied procedures and technologies. In the upcoming year the pilot will be scaled up.

Read the blogpost 

Jury prize @ the FIED International colloquium

At the FIED international colloquium held in Paris on 13 and 14 December 2017 the OP4RE project was presented at a round table discussion attended by Silvester Draaijer, Daniel Haven, Sabine Bottin-Rousseau and Pierre Beust.
It should also be noted that a video competition entitled “My innovation in distance learning” was organised prior to the colloquium, the prizes of which were awarded during the colloquium.
The experimentation of remote monitoring of home examinations carried out in Caen within the framework of OP4RE’s IO3 won the jury prize of this competition.
Watch the video here.

Update on IO 5: Privacy

Outline of IO5 – Privacy

In order to be compliant with respect to privacy in international online proctored remote examinations, it is needed to have ‘lawyer proofed’, EU compliant contract templates, all kind of agreement forms and guidelines for examinees and tester. These templates are intended to be part of the Intellectual outputs of the OP4RE project, but also to convince all future relevant stakeholders, such as accreditation agencies and data protection officers of the value and legitimacy of online proctoring.

Current situation

The first step of the team responsible for IO5, which info (inside which documents) we need in order to fulfill our goals. Therefore two privacy impact assessments have been performed. The identified documents are risk acceptance forms, data classification and processing agreements, privacy policies. Furthermore guidelines and templates for legal notices, consent texts, and privacy statements are required.

The next step was to collect information from different sources, such as data protection experts, EU regulations, and already existing material from all OP4RE partners as best practice templates. Now, we are in the middle of the info consolidation phase.

Planning of year 2

The finalization of documents for review is followed by a fine tuning by all OP4RE partners. In parallel, the experiments with online proctoring at the various partners of the OP4RE project will be expanded.

An update on IO 4

Last Wednesday 6 December 2017, the first large-scale experiment for an online proctored exam was conducted at the Vrije Universiteit Amsterdam. The goal of the study was to investigate to what extent students would successfully go through the stages of setting up the student computers at home and take a trial exam in light of optimizing the student experience.

The target group consisted of about 600 first year Bachelor students for a course in Statistics. The group was divided at random in two groups. Group one took the trial exam at the central exam hall of the university. Group two took the trial exam at home. The preliminary results of this experiment show that students prepare their hardware (computer, webcam, screen recording, microphone, mobile phone) at very late stages before the actual exam. For a number of students, this leads to problems during the log-in process of the proctoring session. As this number was quite high, it turned out that assisting these students by the VU proctors using the integrated chat-possibilities of the proctoring platform was difficult. Especially because in the tense and short period in which the exam could be started, communicating clearly, understanding the problem and provide solutions was not satisfactory for both students and proctors. The experiment showed that students need to be guided and urged in earlier stages to check all their equipment. Maybe guidelines should be developed in which the responsibility for a technical successful exam is transferred to the students at a certain stage before an actual exam. This experiment provided the OP4RE project already with important learning experiences  that will enable the design of fit-for-purpose guidelines and rulebooks.

Update on IO 1

Imagine Bachelor and Master degrees being completed online. Envision students enrolled in online degree programs, taking validated assignments and exams from home or any other location of choice. Increasingly mobile and flexible online offered education asks for more flexible and off-campus examination procedures in the near future.

In the past months, Risbo has taken  steps towards this goal, by starting with collecting of information on current laws and regulations about (inter)national privacy, fraud and security. Meetings with experts in the field of education accreditation like the Dutch-Flemish Accreditation Organisation (NVAO), and the Privacy Officer at the Erasmus University Rotterdam- who is an expert on privacy, fraud and security – have well informed our research.

Early 2018 will start off with the actual testing of online assessment. Students will take proctored assignments and exams within MOOCs and within Educational Programmes. For these pilots, Risbo is excited to collaborate with the IHS and RSM who have both developed  MOOCs. Other departments at de Erasmus University have shown interest to collaborate as well. Eventually, these pilots contribute to the collective development of a guideline for educational institutions who are interested in online proctoring in the near future.

Risbo/Erasmus University Rotterdam is looking forward to the first outcomes and will keep you posted in the upcoming year!

IO2: An update on security and anti-fraud legislation protocols

The work at the University of Hertfordshire has continued from the activities outlined in the annual report written for 2016-17 for IO2, with a focus on developing and refining a risk assessment document for online proctoring.

Currently the team is researching with a group of campus-based and trained proctors employed by our university with the aim of developing a set of guidelines for training online proctors. The team has designed a variety of methods for investigating proctoring approaches for invigilating online examinations. These are:

  1. Showing a ‘Record and Review’ scenario to a group of current proctors, followed by a guided discussion in a focus group to discuss the points they would raise as potentially suspicious behaviour from students. This was a largely reflective and qualitative study undertaken in August/September 2017
  2. A blended mixed methods approach using a campus-taught module with an online assessment is planned for early 2018. This will use a live student authentication session followed by a Record and Review mode for the assessment, with the same group of proctors retrospectively reviewing the process and identifying any potential ‘pinch points’.
  3. Thirdly, a complete ‘Live proctored’ session will be reviewed by our group of campus-trained proctors, this is planned for later in 2018.

Using recorded ‘live scenarios’ helps us to further understand what the trained and experienced invigilators/proctors in a campus-based situation are looking for when invigilating an assessment and thus where the threats to secure online assessment which is free of cheating might be.