Newsletter 5

On May 9th 2017 we organized our first online reach-out session. It was a rich learning experience, both on content level as on organizational level. The technology let us down, but we’ll take this as a working point to our next reach-out session. We hope to welcome more of our associated partners the next time.

The reach-out started with an explanation of the design of the OP4RE project and of the content of the Start Report. After this explanation of the OP4RE by project coordinator Silvester Draaijer, Daniel Haven, from ProctorExam, showed their proctoring tool. After this introduction, 3 partners explained about their first experiences with online proctoring. First Michiel van Diesen (Fontys, the Netherlands) shared their experience with online proctoring. Then Pierre Beust (FIED, France) highlighted the experiments of home proctor exams at the University of Caen. From Belgium we invited VIVES University College, department of Distance Education. They have a long tradition of distance learning and they work with ProctorExam to deliver some 1000 of online exams annually. In order to support this amount of online tests to be taken, VIVES has dedicated staff to plan, administer and proctor the exams. For more information and presentations, we refer to the OP4RE website.

In the coming months we will continue to work on intellectual outputs and more experiments will be set up. As we go on, we experience that we differ from our planning, due to the fact that the IOs regarding Accreditation and Privacy do not lend themselves very well for strict linear project planning. Maybe some shifts will be made during the second Transnational Meeting on October in Darmstadt. Concerning the previously mentioned issues with privacy and data protection, we continue our quest. The exact rules and practices to comply with high sensitive personal digital data are far from clear yet. Further, ProctorExam has made a new partnership with the “Associatie voor Examinering”, which was already an Associate partner of the OP4RE project. We start more intensive discussions with them for collaboration.

 

Newsletter 4

The anticipated complexity of privacy and data protection is surfacing as is shown in the flow chart below. For any institution wanting to start to use online proctoring one should consider the concept of Privacy by Design. This flow chart can be of assistance in designing processes and agreements with that goal. So, after identifying opportunities for online proctoring, each institution will have to develop and implement privacy and data protection policies, regardless of any proctoring system being used. The relevant officers need to be identified and the relevant procedures and agreements should be drawn up and agreed. This Privacy by Design approach needs to be conducted along other aspects of online proctoring that are of importance such as Security, Fraud detection, Fraud regulations etc. So, multiple streams of policies and technical studies need to be executed when an institution want to start using online proctoring.

Yet again, some general – and relative easy and obvious – guidelines can be identified already when doing any kind of online proctoring. We will provide a few examples:

  • When performing an online exam, candidates need to be informed in advance about the nature of the exam and their consent to use the data is needed. Furthermore, candidates need to be made explicitely aware of what is going to happen with the data (ownership, privacy,…). In some institutions this kind of experiments (with students) needs to be submitted to an ethical commission.
  • When conducting online remote examination, some Privacy Policy arrangements need to be provided, including protocols, procedures and rules of conduct. Both for the data owner (institution of higher education) as well as the data processor party (the proctoring solution provider). Privacy and security impact assessments need to be performed by institutional Privacy and Security Officers. In addition processor agreements should be drawn up and agreed upon between the proctor service provider and the institution as data owner. Additionally, it may be that also processor agreements are needed between the test supplier and the data owner.
  • Issues concerning Privacy Regulations when multiple and/or foreign countries are involved should be cleared out as well. For example: which regulations need to be complied with when an institution for higher education in the Netherlands is organizing online proctoring for remote examination of Russian students, and the video data is stored in Germany. How do international regulations (i.e. foreign laws, local laws) and institutional procedures match? Which specific regulations are applicable? Not at least because institutions for HE as responsible parties can face large fines if the act illegally or do not comply fully.

Newsletter 3

Dear Associate Partner of the Erasmus+ project OP4RE,

Following the first transnational meeting of October 25 and 26, the OP4RE Project Management Group is progressing in the development of the project. For each of the 5 Intellectual Outputs (IO), more detailed plans are under construction.

The project members that are concerned with privacy and security aspects of remote examination are working on setting up a framework for institutional impact assessment procedures and agreements regarding security, risk acceptance and privacy. More information on this IO will follow in later Newsletters.

Identifying and getting all conditions in place in order to conduct experiments takes quite some leeway and members of the PMG has started to initiate this already. Below we present three examples of experiments in preparation.

Example 1: Selection test for entrance to Bachelor programs with numerus fixus

Experiments for remote proctoring are now set-up for Bachelor study programs that require selection of students (Numerus Fixus programs). In normal conditions Dutch candidates needed to take a digital selection test in the Digital Exam Hall of the Vrije Universiteit Amsterdam for that purpose. Dutch institutions for higher education however are obliged to offer students living in the so-called special overseas municipalities (Bonaire, Saba and Sint Eustatius) the possibility to take these selection tests under the same conditions as Dutch students. For these selection tests, the OP4RE project will provide the services to take the test for these overseas candidates remote at their home. The selection test that will take place in February 2017 to April 2017 will be used as a pilot for future selection test (2018). In the pilot, the procedures, the survey questions, the technical set-up of the online tests etc. are being detailed and tested. Extensive discussions are conducted with regards to these procedures and the possible problems that could arise and their mitigation.

  • The problem of preventing fraud or the illegal recording of the test itself by candidates;
  • The procedures in which students are made aware of their consent for privacy and security of online remote testing;
  • The equivalence of the different forms of the test. For example, in the Digital Exam Hall, students get a case-based questions handed out on paper, while overseas students need to read the case information from the screen;
  • How to deal with the issue of going to the toilet for a 3-hour test. In the face-to-face setting, candidates are allowed to go to the toilet after finishing part of the test and before entering the successive part: how to deal with this in a remote situation.

Example 2: Experimental study with a large cohort of students

The second pilot study is concerned with an experiment in which the equivalence and comparability of exam results for on-campus proctoring and remote proctoring of digital tests is studied in a scientific experimental setting for the domain of statistics for a large cohort (300+) of students for December 2017. For this experiment, teachers and the board of examiners need to be involved heavily in the decision making process. In particular for this experiment, the set-up is chosen to split the final grade in two parts. The first part will be the online test, the second part will be a regular final exam. In order to enable this experiment, changes need to be approved for the official course as laid down in the previously established regulations of the course.

Example 3: Mathematics proficiency tests for international students

The third example is concerned with offering remote examinations for international students who want to enter an International bachelor study program in the Netherlands. A number of students need to provide evidence of sufficient mathematical skills. The current methods of providing this evidence is in the form of supplying known diplomas or certificates based on:

  • International Baccalaureate, Mathematics Higher level
  • UK: Mathematics A levels
  • Germany: Zeugnis der allgemeinen Hochschulreife, including Mathematics LK/eA.
  • Romania: Diploma de Bacalaureat, including a Mathematics and Computer Science Profile, or Natural Sciences profile.
  • Courses and exams at international locations associated with the International Baccalaureate Mathematics Higher Level/Cambridge International A-levels or in the Netherlands itself at a few specialized institutes.

The leeway up to a possible experiment with remote testing involves discussions with a lot of stakeholders in the University concerned with international admissions, subject matter experts, as well as setting up a business case. Conducting an experiment could provide more empirical evidence to support such a business case. In such a business case, the possible outcomes for remote examination (in terms of increase of student enrollments for specific countries and chances of successful study careers) are weighted against the costs of designing, maintaining and administering high quality home-made mathematics tests.

Reach-out session 1 SAVE THE DATE: April 4 2017

The OP4RE project will produce a Start Report in which the current state-of-the-art with respect to online remote examination will be described. Given the progress of our project, the concept of this report will be shared with the Associate Partners on March 24. Discussions of the Start Report with the Associate Partners will be conducted on April 4 in an online session.

Newsletter 2

Last October 25 and 26, the OP4RE Project Management Group, additional staff from the participating institutions and a number of invited speakers gathered at the Vrije Universiteit Amsterdam for their first transnational meeting. The agenda for the meeting can be found on the project’s website https://onlineproctoring.eu under PROJECT PAGES -> TRANSNATIONAL MEETINGS

The meeting was lively and a number of issues were discussed relating to the set-up, the intended outcomes, and anticipated difficulties of the project.

We learned a lot from the presentation of Lex Sietses of SURFnet of the Netherlands. Lex his presentation can be found here:  https://cloud.swivl.com/v/97092479f18197382e04545765f38efb and https://cloud.swivl.com/v/ee5048f14859f5f145cc26fb99ade7ee. Lex has written a White Paper on Online Remote Examination in the context of higher education with an emphasis on privacy issues. Lex made very clear that privacy is of utmost importance as remote video capture and storage of individuals and installing software deals with high sensitive personal data. Deploying online remote proctoring is definitely possible from a privacy point of view, but a number of conditions must be taken into account which may limit initial deployment. His presentation and work provides the project with a head-start for our Intellectual Output regarding privacy regulations.

From discussions concerning security issues in remote examination, we learned that we need to pursue several lines of inquiry. First, the project needs to dive into describing how institutional privacy, security, information safety policies, risk assessment and risk acceptance procedures of an institution’s top-level board of directors is organized. Then can be described how online remote examination fits into these procedures and guidelines. Second, security issues can be identified at a number of levels, actors and configurations in the technical proctoring systems. Which issues are those and what is the best way to test and describe security risks in view of institutional security policies. Third, security issues and fraud can also be identified in the test-development and test-taking, proctoring and test-analysis process. Fourth, given the results of such impact assessments, a comparison needs to be made with the current paper-and-pencil examination procedures to establish a baseline with which to compare online remote proctoring. These lines of inquiry all form output of the OP4RE project.

A discussion led by Marinke Sussenbach of Delft University of Technology showed that it is difficult to assign and use Massive Open Online Courses’ (MOOC) acquired credits in accredited education systems. She presented the case of Delft University of Technology that tries to set-up a ‘trust’-system with a limited number of top-tier Universities for credit recognition. Her presentation can be found here: https://cloud.swivl.com/v/394d80f17ac3dbbd9a4d20e54465583e (unfortunately, the sound recording was bad). Her presentation gave food for thought for Intellectual Output 1 in which experiments will be set up to recognize MOOC credit in a higher education context.

The project members agreed that an initial report will be developed in which the current state-of-affairs regarding online remote examination in higher education will be described as a base for the further activity plan within the project.

The discussions and information gathered during the first transnational meeting will be used to further detail the working plan for the project. We already want to inform you that we will select a number of the associate partners to be involved in the quality assurance process of the experiments and outputs that the project will deliver.

Newsletter 1

The Erasmus+ (E+) project Online Proctoring for Remote Examination (OP4RE) was approved by the Dutch E+ Agency during the summer break. The selection committee of E+ was impressed by the number of Associate Partners for the project, which emphasized the importance of the project. Therefore, we would like to thank the Associate Partners for their support.

As a very first step, the team put up a website to post the public preliminary information regarding the project, the activities and the intended outcomes: https://onlineproctoring.eu. We invite you to visit the website and leave any comments.

The Project Management Group (PMG) of OP4RE is now working on detailing their plans. As an important step, the PGM will gather for their first transnational Kick-off meeting on October 25 and 26 in Amsterdam, at the Vrije Universiteit Amsterdam. If you have any input for the PGM to discuss, please let us know.

Following that meeting, we will inform you further about the project and will involve you to ensure that the project delivers it’s intended outputs and ensures wide dissemination of the project’s results.