Investigating the Impact on Learning Outcomes Through the Use of EdTech During Covid-19: Evidence from an RCT in the Punjab province of Pakistan

Investigating the Impact on Learning Outcomes Through the Use of EdTech During Covid-19: Evidence from an RCT in the Punjab province of Pakistan

Donor: EdTech Hub Consortium, World Bank & UK-AID

Duration period: From Feb 25, 2021 to Dec, 2021

Introduction: Globally, the Covid-19 pandemic has deepened the learning disparities between students from disadvantaged backgrounds compared to those from more advantaged groups. The closure of schools due to the pandemic has perpetuated inequalities, given that students from more disadvantaged backgrounds have limited access to the technological devices needed for learning continuity. Surveys carried out during the school closures in Pakistan have found that the situation has created severe gender and wealth gaps in learning (⇡Crawfurd et al., 2021).

Objectives: The objective of this study was to test the impact of using low-tech devices on the literacy and numeracy outcomes of secondary school students during school closures in Pakistan. Our target population was Grade 8 students attending private schools in Bahawalnagar District. This is an economically deprived area situated in the Southern part of Punjab province. We tested the impact of three interventions, which were administered online on 258 students in 12 schools. These 03 interventions were:

1. Computer-assisted teaching at the right level (TaRL) interventions,

2. Fortnightly assessments (FAS)

3. Digital teacher training sessions (DTS)

The primary research questions which this study was interested in probing were:

RQ1: What were the key issues regarding access, barriers, and usage of EdTech by students during school closures?

RQ2: What were the separate and combined effects of delivering TaRL, fortnightly assessments, and digital training sessions online on student learning outcomes in Urdu, English, and maths? online learning

RQ3: What did stakeholders identify as the key barriers relating to access to technology for online learning during the school closures?

Activities/Findings: Our results show a significant and positive ‘Intention to Treat’ (ITT) impact on the Urdu and English scores of the students who were part of the computer-assisted TaRL treatment group. These students increased their English and Urdu scores by 0.56 SD. However, we found no significant impact on maths scores in the TaRL treatment group. We also found a positive impact of the fortnightly assessment and digital teacher training session interventions on students’ English scores. However, we found no ITT impact on the maths and Urdu scores for these treatment groups. The Local Average Treatment Effects on Treated (LATE) analysis revealed positive and significant improvement in the Urdu and English scores of the students in the TaRL treatment group.

The study found a positive and significant impact of attending online classes due to school closures on students’ maths and English scores. However, we didn't find any significant impact of online classes on the Urdu scores of students. When differentiating for income, 1 we found that students from relatively higher income brackets scored 0.18 SD and 0.14 SD points higher in English and maths tests but 0.2 SD lower in Urdu tests. Students from high-income households also scored significantly higher scores in maths and English than students from low-income households.

Outcomes: Key stakeholders whom we interviewed for this study recommended redesigning the curriculum to incorporate approaches such as TaRL to help alleviate the current learning crisis defining the education system in Pakistan. Moreover, parental involvement with students’ education and parental cooperation with teachers and school administrations is also vital.

Impact/ policy level intervention: Our findings offer some useful lessons for policymakers to help improve the learning outcomes for the most disadvantaged groups. These include:

1. Actively involving school actors in the design of a technology-assisted TaRL is important for the long-term buy-in of teachers and students.

2. Understanding the complexities involved when it comes to implementing technology-assisted TaRL interventions and how this can impact their level of success is essential.

3. Considering factors relating to access and use of devices that go beyond mere device ownership is key.

4. Community perceptions of technology must be factored into the planning of any technology-supported education programmes.