Making sense of the business of edtech has always been a challenge. It’s why I set up the Assignment Report in 2005 and why I read the DfE’s most recent edtech report, published in June, with interest.
Business Intelligence (BI) is weak in most education and edtech companies and is too rarely integrated effectively into a business’s decision-making process. Getting access to decent data for BI can cost £’000s, so when the DfE commissions research and publishes it for free, it gets widely read.
Sadly, their most recent publication, Future opportunities for education technology in England, is not one you should waste your time on because I’ve done it for you and it’s bad from start to finish. Here are some of my concerns:
- Who wrote the brief for this at the DfE? It’s a waste of £60k of taxpayers’ money, but worse it presents a misleading and out-of-date picture about the complexities of the UK edtech market
- Who selected the researchers? Ecorys may claim to be ‘answering tomorrow’s challenges today’, but their experience in edtech and the business of education seems minimal. Far better to have a framework edtech research contract to include experienced researchers such as Cooper Gibson or C3 Education who have done good previous work for the DfE and BESA’s external market research respectively
- Who checked sources before publication? Here are a few examples:
‘EdTech in the UK, accelerated by the pandemic, with the sector growing by 72% in 2020 and anticipated to be worth £3.4 billion by the end of 2021’ Education Technology & FE News
The original source for both stories is a report EdTech: The Hyper-Accelerator produced by generalist headhunters Robert Walters and Vacancysoft. The FE News story was actually written by Robert Walters and is little more than a business development/marketing tool that cites only one actual source, a story from Tech Nation based on their own 2019 report that doesn’t even mention edtech! In terms of relevant industry expertise the single source comes from Korean company Kidsloop (and their CEO), a business with an almost invisible footprint in the UK. Ironically after featuring in the GSV Edtech 150 in Jan. Kidsloop tanked and went into administration in July.
Brighteye Ventures 2021 European Edtech Funding Report is quoted twice, but their 2022 report was published in February this year and shows EU edtech investment jumped from sub-$1bn (2019/2020) to $2.5bn in 2021. So the figure of $1.69bn used in the DfE report is an underestimate of 47%!
Of the data sources in the report several are from 2018 and these use research that goes back, in some instances to 2012,. It fails to even acknowledge the data from Oak National Academy, arguably the largest K12 success story of the entire pandemic which was presented publicly at the researchEd National conference in Oct 2021 (TES story about this).
There are some more credible sources like Holon IQ, but what really shows the methodological weakness of the entire document is that some of the best data comes from the 2021 DfE Edtech Survey done by Cooper Gibson.
Why a firm with no edtech experience (Ecorys) was chosen to compile the report when the DfE have an existing relationship with an experienced research firm (Copper Gibson) says a lot about the weakness in policy teams and procurement at the DfE
- Who did a reality check? The report seems to see edtech as some sort of homogenous market when it is anything but. Many of the ideas, data and assumptions come from way outside K12 in England. For example, the claim that in terms of global expenditure on technology, AI will be the second largest spend followed by AR/VR, Robotics and Blockchain are taken from two reports from Holon IQ. Their conclusions cover the entirety of edtech – from B2B, B2C, consumer, professional learning, apprenticeships, tutoring, hardware, software services, etc. This is not the reality in England’s K12 edtech/education market, where most schools will struggle to pay for the transition from Windows10, let alone implement ideas like social robots or to effectively use products that claim to be based on AI
- Why didn’t the report include the ongoing failure of AI in edtech? This is well explained in a recent video series from EDuGrowth, featuring Professor Dr Wayne Holmes, which is far more up-to-date the book Holmes co-authored in 2019.
In terms of assessment Ecorys could have used a story from the founder of Janison (the listed Australian assessment company) showing that the AI tools most effective in edtech are almost exclusively based on integrations of two specific AI tools – BERT from Facebook and XLM-R from Google. (I’d add a third, the AI/machine learning tools from Amazon Web Services.) However, for any AI-derived edtech you need access to a large amount of high-quality data, something most UK edtech companies claiming to use AI, simply don’t have
- Why didn’t the report include more digitally-advanced edtech countries? Comparing Denmark (popn. 5.8m) with China (1.4bn), France (65m) and the USA (332m) is unhelpful as they have very different education structures (China and France are centralist vs the USA’s devolved state control). It’s missing more relevant digitally-mature countries, like Australia, New Zealand, Canada, Singapore and the Philippines
- Why is there a lack of basic educational knowledge? On p27 the report states that AI, ‘which employs machine learning, can detect patterns in written work, speech, and other actions, which then can be individually adapted to students’ learning styles and needs’. This cites an NMC Horizon report, but shows the Ecory’s staff lack of pedagogical knowledge as there is 40+ years of educational research that clearly shows the concept of learning styles is an edu myth. (If you don’t believe me, ask Tom Bennett, founder of researchEd). The report focuses on concepts like Social Robots but misses far more evidence-backed pedagogical concepts like Spaced Repetition, something that underpins one of the most interesting UK edtech startups Carousel Learning
- Why the lack of transparency? None of the ‘13 subject experts, representing UK and international leaders in EdTech policy, academia, and industry’ are identified, arguably a breach of the Nolan Principles and the DfE’s own statutory requirements. Experts often have an agenda which may be linked to their employer or other ties to tech/edtech providers, etc. All this should be declared upfront.
- Why the lack of balance on the expert panel? In looking for digitally-mature edtech nations, Ecorys recruited twice as many experts from China (a substantially smaller and completely differently regulated edtech market) than the USA (arguably the most regulated), missed the entire APAC and LATAM regions and didn’t disclose which sub-sectors the experts had actual experience in
- How were the workshops run? Again there should be full disclosure of the ‘67 respondents’
I could go on but let’s make the point – this report is a waste of time and money.
Footnote
The most recently published DfE ‘advice’ about choosing an MIS is another example of poor research by the DfE. I spoke to one major MIS provider on the list who commented that they had no contact from the DfE who had simply copied some of the data from their website.
