Built on Evidence: What AI Readiness Data Reveals Worldwide  transcript

Welcome to Built on Evidence, What AI readiness Data reveals Worldwide.

I'm Joe Kimmelman, Research strategy principal in Pearson's Chief Strategy Office.

In today's session, we'll hear from an esteemed panel who recently contributed to the research study that you heard referenced in that video.

Pearson conducted this study in partnership with AWS and we focused on the impact AI is having on the transition between higher education and the AI enabled workforce.

Now for this research, we we surveyed and interviewed 2700 learners, education leaders and employers in six countries.

And we found that AI is compounding many of the challenges that were already faced, but in the transition between higher education and work.

But we also found bright spots where we found employers and education institutions innovating and evolving their approach to better prepare graduates for the AI enabled workforce.

So today what we'll do is we'll draw on the research and the experiences of our expert panel to explore three things.

The first is what AI readiness looks like.

The second are the set of common frictions that make it challenging for institutions to develop AI ready graduates.

And the third is a set of concrete recommendations and tools that can help educators and education leaders overcome those frictions.

So I'm going to start off by handing over to our panellists to introduce themselves and we'll start with Tanzila.

Hello, Joe, good, good morning again.

Thank you so much for this introduction.

I am Professor Tanzila Saba.

I'm working in Prince Sultan University at the College of Computer and Information Sciences.

I am the Associate Director of Research and Initiative Centre and leading artificial intelligence and data analytics lab at Prince Sultan University.

And today, I am truly honored to be here as a one of the speaker among the distinguished speakers.

And thanks for this.

Fantastic.

Thank you, Tanzila.

And now can I hand to Mike just to introduce yourself?

Thanks, Joe.

And hi everyone.

I'm Mike Perkins, I'm the head of the Centre for Research and Innovation and an associate professor at British University of Vietnam, Hanoi.

Fantastic.

And lastly, Grasa.

Hello everyone.

Grasa Carvalho.

I am the Director of the UCL Centre for Digital Innovation, UCL just decoding for people who are not from UK University College London, and I'm also the director of Strategic Alliances in the Department of Computer Science and part of the Office of the Vice Provost of Research, Innovation and Global Engagement.

Lovely to be here.

Thank you.

Fantastic.

So we're going to get started.

We've got lots to get through.

So our research highlights the enduring value of university, but it also shows that what employers are looking for in graduates is evolving as AI disrupts work.

So we refer to this idea of the AI ready graduate and AI readiness.

You've heard it a few times already today.

But before we dive into talking about institutions and what educators can do, I want to make sure that we're aligned on what we're talking about there.

So I'm going to ask each of our panellists to share.

What does it mean for you to think of a graduate who is ready for AI enabled work?

So what is an AI ready graduate?

And what can a genuinely AI ready graduate do that that another graduate couldn't who who might just be familiar with everyday LLMS and tools like ChatGPT?

So I'm going to start here with Grasa as you were the first on my screen.

Like what does AI read and this been to you?

I think that an AI ready graduate actually combines a set of two principal skills.

1 is a willingness to experiment and having the tools and the basic understandings of the technology that allow allow him her to not be afraid of experimenting and testing new ideas while at the other on the other side.

It's the grounding basic principles that are essentially built upon a series of education activities and building on the knowledge base that they have as a person and as as someone who accumulates knowledge through their life.

So it's the unique combination of grounding principles and as well technology witness and and experimentation.

The domination of a tool specifically will not be sufficient for a last longing success in the marketplace.

So still a basic education, grounding in research and education principles and then experimenting and trying and not being afraid to fail and retry again, learning the hypothesis, testing, evaluating, starting again.

It's what I think will be the key enablement.

Fantastic.

I love that.

And I, I really like this idea of it's, it's not about just technical proficiency with one tool that's not going to endure.

So Tanzila, how about you?

What, what, what does it mean to you for a graduate to be AI ready?

So it's not about only the graduates are ready for the AI.

The most important thing is are they just familiar with the ChatGPT and some of the generative AI tools are actually they know what is the AI responsible use, how to use it, for what purposes they use and most importantly where they shouldn't use it.

And if they use it, they are using it just as a thinking partner, as a facilitator or actually they are replacing it with their own critical thinking capabilities or these things.

So these are very critical points which we have to consider when we think after just delivering a couple of workshops or training or providing some subscription to the AI tools and we think our graduates are ready.

This is not actually the scenario.

So if we really want to make them ready, we have to go with the most important nowadays is the responsible AI, the ethical concerns and how to use it.

This is the most important thing.

That's great, Tantila, that's really helpful.

I'm, I'm going to come to Mike here with the same question, but I know we've covered a lot of ground already.

We've we've talked about using with purpose, responsible use, familiarity with the actual tools themselves and experimentation and some of the underlying principles.

Mike, is there anything you'd add there in terms of what it means to be an AIAI already graduate?

Yeah, I think I'd go beyond what Tanzila said in terms of the, the responsible AI and about for me an AI already graduate as someone who can take responsibility for what they've actually produced with AI.

Now, I, I disagree in terms of saying that an AI should only just be a thinking partner.

I think an AI tool can go certainly beyond that in terms of when we're talking about frontier models, what we can produce with AI tools is fantastic.

And if you're just using it as a thinking partner, I would that's a little bit of a level below what I would look for for an AI ready graduate.

I want somebody who would be able to really explain the decisions behind the, the specific model choices that they've used.

Why did you use that model for this task?

What was the decisions that led you to to choose this model and to ask AI to support in this past somebody else who can then go beyond and actually check the evidence?

You know there?

We've all seen that AI tools are very good at being confident, but being confident in AI and being competent in AI are two really different things.

And just getting that fluent answer is really simple, but recognizing and then AI is giving you something that's really plausible but not actually right.

This is where we need to blend that AI readiness with the subject readiness because without the subject readiness and the core understanding, this is where a graduate isn't really going to be able to do something appropriate with an AI tool.

So that's where I think that an AI ready graduate is for me.

That's excellent.

OK, so we've got some really interesting points in here.

We've got this idea of we're looking for a graduate who combines some technical proficiency with the different tools.

They've got an ethical approach to the decision making with AI, sort of understanding responsible AI use.

They're understanding where AI convert add value in their work and where it can't.

So the purpose and then why they're using it.

They're understanding the value of different tools as they develop and different models for different use cases.

And we've got something here coming through as well on, on some of the more softer maybe human skills alongside that as well.

So we have a target in terms of what we're aiming for here as an in terms of what an AI graduate may look like.

Now I want to move us on to explore some of the key frictions that universities might face in developing these kind of AI ready graduates.

And so we're going to work our way through some of the frictions that came through in this research, but we're going to start with the one that underpins all of the others.

And I'm going to come to you, Grasser, with this idea of pace friction.

So the research highlights that AI driven workplace change is moving faster than traditional curriculum review cycles in many institutions.

And in the UK, if we focus just on the UK for a second, only 13% of learners told us that they think their institution is keeping pace.

Now, Gracie, what do you think is going on here?

Are universities adapting quickly enough?

And if not, how can they increase the pace while maintaining high standards?

OK, so I, I think that the first thing that I need to do is to contextualise UCL in the UK landscape for higher education, because AI machine learning at ECL started actually 20 years ago.

And UCL is one of the cornerstones, developers and creators of AI technology.

The the other thing that is also important to understand is OK this, most of the higher education institutions in UK are 100 and hundreds of years old.

UCL is celebrating 200 years anniversary this year.

And therefore the processes and the the, the frameworks for change and for developing curricula.

Indeed, sometimes is not fast enough and adaptive enough to understand and to survive a world where every three months we have a frontier model that completely outdates everything that we have taught in the previous three months before.

So indeed, the normal management organizational frameworks of UK universities do have a challenge and the change management is not something that particularly easy in the context of UK higher education.

However, there are ways to solve that.

And specifically at UCL, having in mind that we're probably 2 times bigger than the next big university in UK.

So we're time 2 times bigger than Oxford and Harvard.

So Harvard and Oxford have the same side with around 2026 thousand 25,000 students.

We have 51,800 students and next week we'll be welcoming 13,700 new students, right?

So the infrastructure that you have to develop curriculum for this number of a student has to be evolving, adapt in some way.

What we have done is we have inserted in all years of all degrees additional courses that their content is adaptive every year.

So it's seminar activities, it's project led by challenges that come from the external industry partners.

And typically with these two modules, one that is seminar that is built every year with a new content and delivers sometimes in partnership with external speakers that come from different backgrounds and a practical project that is defined in terms of problem statements that come from the outside world.

These two artefacts have allowed us to build a flexible, a flexible model.

The other component that we are now increasing is while we have half of the degrees that have mandatory compulsory courses, we do have much more elective courses available as well.

So what that means is that a student can indeed build their curricula in a more much more fluid way.

And we are now allowing students to tap into modules.

That's something that is quite frequent in the US, but less so in UK modules from different faculties and departments in a much more creative way.

So we have students from arts come into computer science teams from computer science growing to humanities and and that is creating a very interesting experience and the set of skills that become unique for each student.

So it's essentially trying to put together the basic principles of how you organise a very large delivery program for a significant amount of students, but building artefacts that include agility and flexibility to provide update of content and as well a combination of topics that can be interesting for the students.

Fantastic with some great points.

Aggressor.

So we're, we're talking here about more flexibility, more fluidity in terms of how the course is put together, but also this regular cadence of updates and this engagement with employers as well, which we can come back to in a moment.

But I just, I want to, I want to build on that and, and think about.

So if that's the pace friction that's kind of underpinning everything and you've, you've surfaced a few ways in which UCL is thinking about and, and, and tackling that.

Obviously that's, this has implications for the educator.

And, and if we, we think about sort of like when we build on this pace friction, we, we also highlighted through the research this idea of faculty capability being critical in developing AI ready graduates.

And obviously that's intuitive.

We'd all expect that, but this is more than simply knowing about the the latest LLM.

We've, we've touched on this a number of times that developments are coming quickly, but I'm going to turn to Tanzila here.

And, and I want to ask you, what does meaningful faculty capability look like in this context?

And when we think about the idea that educators will need to be updating their course content and curriculum faster and faster in response to things that are happening outside the institution, like what, what kinds of support do educators need to develop the capabilities that are required to develop AI ready graduates?

Thank you, Joe, for this interesting question.

And I think it is the demand of the time as AI is booming now.

And Greg has just mentioned, we have a couple of ways where we can accommodate our curriculum because the technology is evolving and each single day something new is emerging technologies are there and how to manage all these.

So I think the first thing we need to clarify is what is actually mean by facility capability.

For me, facility capability is not simply knowing how to use jet duty or another generative tool.

That is only the starting point for each and every single faculty member.

A capable faculty member in AI enables universities, which I know personally believe most of the universities either in Europe or USUK, America or Asia, and they are trying their best to come on this way.

They need to understand, as I mentioned earlier, when to use AI, how to use it effectively, how to evaluate its output, and perhaps then in the most important thing is when not to use it.

Actually, there are several dimensions to this capability.

First of all, AI literacy, understanding what AI can do and what it can't, its limitations, its biasness.

Just as Mike mentioned and myself, any responsible AI, which is very, very important, the reliability issues and how different AI tools work.

It's very important to know about each and every tool.

2nd is which is the most important.

We are considering the graduate, the student, the curriculum, the pedagogical capability.

So facility needs to know how AI can improve the learning rather than simply making the task very easier for designing the assignment, for designing some of the assessments.

So for example, if we redesign assignment so that student can use AI as a thinking partner where Mike was not agree.

And I totally agree with this point.

It is beyond the thinking partner, but while still demonstrating their own reasoning, creativity and subject knowledge.

So it is very important.

And the third thing for faculty capability is the assessment capability.

So this is becoming, I believe my all colleagues who are here now, they agree with this.

The most important challenge which we are facing is the assessment.

You assign the assignment to the student and then in minutes are in second, the assignment is ready, the results are ready.

So how can we assess all these output?

It's very important.

If AI can produce an FA code presentation or even a research proposal just in seconds, then we have to rethink what we assess.

We should increasingly assess the student ability to question validation, applying, explain and approving AI generated output rather than simply they produce an output.

So I totally agree with the MIC point of view and we have to train them and who we are as a facility.

This is our capability where we can say.

So for me, the meaningful facility capability is to integrate AI into teaching, learning and assessment in a way that strengthen human capabilities rather than just replacing them.

So university need to move.

This is very important and I think Gresa has just mentioned it.

We are not just considering the computing field or engineering field where they have the basic knowledge and the basic skill of all these AI tools and some of the computing in the university.

We have the diversity.

So the discipline specific faculty development is very important.

We have the business domain, we have the law, social sciences.

So how we prepare these faculties to get ready for the AI illiteracy and how to train their graduates for this.

So it's a very important connection here to the graduate student will not become AI ready simply because we give them just access to the AI tools.

They will get ready when the facility is ready, when the facility is capable to train them to guide them how to use these AI tools.

And then personally, I believe AI ready faculty first come and then the graduates where we are hoping we will have the best result.

So this is any even our life would just like to share our Prince Sultan University the best practices.

We have the Sadaya which is our national organization and they have their governance system for leading the AI.

So we are seeing the conversation is moving beyond whether faculty use AI.

The most important question is how to integrate it into their disciplines and into their earning experience.

Because diversity is there different majors are there different skill sets are there?

So it means bringing together AI expertise, teaching expertise, and discipline expertise, not expecting every faculty member to become AAI specialist.

That makes sense.

That makes sense.

Have a capable faculty to deal or to proceed with this AI era where every, every single day something new is emerging.

No, that, that makes a lot of sense.

And, and I want to, I want to pick up on something you said there where you were talking about some of the challenges around assessment and this, this sort of governance piece.

And I'm going to bring Mike in here because one of the other key frictions that came up in this work, it's like it's not just about the capabilities of the educator.

There's another key factor here that we need to account for is the, the broader context within the institution in terms of how we are integrating AI into an institution.

And governance is 1 factor in, in that.

So our research found that there was this persistent gap between having AI rules and governance and policies and actually creating trusted AI practice among students.

So Mike, I, the, the challenge here is sort of, it's essentially like how should educators move from policing AI in terms of the way it's used towards designing learning experiences where students can be transparent about their use of AI.

So we're doing something more supportive, supportive and still protecting the learning and academic integrity.

Thanks.

I, I think that's a really tricky issue.

And what we're looking to try and get students to recognize is that they need to know that being transparent about the permitted AI use or using AI in an appropriate way is a normal part of learning or should be a normal part of learning.

But if, when they disclose their AI use, if that feels like, you know, they're inviting suspicion in terms of their work, why should we be surprised when students are staying quiet?

You know, there's this, there's this huge gap.

I think the report had only 35% of UK learners being comfortable with their staff knowing all their AI usage, which is a really, really small amount.

And, you know, this is, I think this comes down to an, you know, an AI disclosure penalty.

And you know, there's a really great piece that that, you know, did some experimentation about that.

I've just dropped a link on this in the in the chat where we see that if people believe that work is written by an AI tool or supported by an AI tool, even if it's not, they will rate that work, that creative work as lower than if they thought it was a human piece of work.

And this is something that the authors describe as hugely sticky.

You know, it's something that if we just say, oh, by the way, AI use AI to help support me with this, immediately people go, Oh, well, it's not as good then.

But we know that using an AI tool can make for even better work than we'd be able to produce by ourselves.

So I think it's about how do we make something authentic?

How do we make things specific to each individual assessment in terms of what we expect from students using this?

And, and just saying, you know, you can use AI, but use AI in a responsible way, that leaves way too much interpretation open to students.

If we've got a task that is asking students to think about using AI in an appropriate way, well, we need to explain what that task assesses.

What, why am I asking you to do this task?

So what's the point of it?

First of all, how does AI fit into that, if it does at all?

What students need to demonstrate themselves and what's the evidence that they need to submit in terms of their authentic learning?

I still see a lot of people saying, oh, well, you know, I'm just going to ban students from using AI and you can't use AI for this now go away and do this piece of work.

And what, what do we expect when students are, you know, under pressure, they're, you know, they're struggling, they've got time constraints.

Of course they're going to lean on AI if there's if there's an opportunity to do this.

So I think we really need to consider how we actually design assessments in an appropriate way.

And in the report, we we talk about our work with the AI assessment scale, which is a tool that myself and some of the colleagues developed, which is a way to actually explain to students what are our expectations for using AI in an appropriate way.

But more importantly, how do we as educators redesign assessments?

Because to reduce this friction, I think we need to be clearer in ourselves and leaning on what Tanzila was saying, it's about the educator's ability to work with an AI tool if or or be competent in some form.

If our educators aren't competent, how can we possibly expect our students to be competent?

So they've got to have the ability to understand where a piece of work can be supported using an AI tool, develop an assessment in an appropriate way, and then encourage and show students how this is going to help support their work.

You know, so we think about the learning outcomes first.

What is it we want students to be able to demonstrate?

Then we've got to align the task and the marking criteria in an appropriate way.

Just saying to students, oh, here's the same piece of work that I've done for the last 10 years.

And your students love this.

This is a great assessment.

It's fantastic.

It just doesn't work anymore.

You know, we've got to really be able to get creative with our assessment practices, but also we have to recognise when we've got to tighten things up.

Some tasks really do need independent demonstration under much more controlled conditions.

Whereas and that's where we talk about the subject specific elements because if we without the subject specific elements being good at AI doesn't mean anything in the future.

So we've got to have sometimes where we're really being able to specifically assess what students fundamentally know and then other assessments which we design in an appropriate way to say, OK, those are the tasks were about what you know, now shows what you can do.

What can you do differently?

How can you get that human edge into an AI tool to do something that's really cool, but other that somebody else couldn't do with either your subject specific knowledge or your AI knowledge.

And that's a great way to start to connect and start to try and reduce those frictions.

And, you know, linked into this, we could also be a bit clearer in terms of asking students how to be more transparent in their approaches to AI tools.

You know, asking them, you know, evidence of what they've done, an account of the decisions that they've made, as I mentioned, where AI helped, what the student checked, what have they changed, and be able to demonstrate this in a clear way.

So I think all of this, all of this is good governance and that should make that participation much more straightforward.

Yeah.

So it's it's a great point.

And and I think it's this kind of supportive AI governance that you're describing there that sits around the educator activity that we've already heard heard about.

And in order to foster the right kind of AI habits and productive use in students that essentially, as we talked about in the report, it like it ends up carrying through into the workplace.

So if you are fostering these, this, this approach to AI among your students in higher education, it, it's mitigating a risk that would otherwise be carried into the workforce in terms of people using AI in, in problematic ways.

And so I, we've focused quite a bit on the institution and on the educators and a little bit on the graduate themselves so far, but we've not really got into the employer at all and the role of the employer.

So I'm going to come to Grasser here.

I want to talk about the, the connection friction, which was another connection, another friction that came up in the research.

And this was really about highlighting the need for universities and employers to form stronger connections to maintain their alignment as employer needs evolve faster and faster, as we, we heard earlier on.

So Grasso, like what does a productive education employer feedback loop actually look like?

So when you've got this feedback loop between employers and and institutions, and what impact do you think it should have on, on teaching and assessment and the student experience?

OK, so I'm, I'm actually speaking today from the AI Centre where we are doing a showcase event with a company that has hosted 22 master students doing their final year projects and dissertations.

And this has been such a mind blowing event where we could see what happens when instead of fabricating a final year project or thesis titles that are artificial and academic only, you actually do a call for problems and a call for projects and open that to the community.

And that's what we have somehow created a a process to do in the computer science department.

We have 730 undergraduate students, approximately 800 master students, and what we do every year is 3 * a year.

We do a call for problems and a call for projects to be industry.

And we have created a format where we have project coordinators that help our industry partners distill and elaborate the projects in a way that is appropriate to the level of expertise or to the level of complexity that is adequate for a first year student or for a final year student or for a specialised master student.

And the during that project, the student works with that problem owner, with the industry partner to deliver something as a, as an output, as an asset, but as well as a research project and the research dissertation or a master thesis.

And that has proven to be quite an effective way to get have a feedback loop when we receive the challenges and the problems from industry, our program directors actually understand if the students they have been training for the past year are ready to solve the questions and the problems that we are, we are receiving at the same.

At the same time that feeds into understanding the capabilities that students can deliver during the project as well.

And at the end, what the way we measure if this loop is actually working or not is monitoring the number of students that then are employed by the companies who have proposed the project.

And typically we have a ratio of, you know, 3040% of them are employed by the, the companies that have hosted them for the projects, which given that they have to a lot of other options as well.

We think that the loop is working and somehow the alignment of what we teach, knowing that every year we, we have a consultation with industry works.

This, this program is called in the industry Exchange Network.

And it's based on the pedagogical methodology of first year, you copy what you have done from other years.

Second year, you actually elaborate and the, the finally your master's student, you are creating your new asset to developing your own idea so that there's a complete pedagogic process, but developed in partnership with with the external environment and with the employers.

Fantastic.

I love this really tight connection and and regular feedback loop that you're talking about there.

I think when, when we, we're going to move on now we've we've heard a little bit about some of the common frictions that are highlighted by the research, But I want to make sure that we've got time to to talk about sort of how we move from the frictions into sort of concrete recommendations for people who are listening on the call today.

So I want the three of you on the panel to to imagine that there's an educator or a leader in the audience who agrees with the diagnosis in terms of some of these frictions in this transition, But there may be unsure about where to start.

So maybe daunted by the scale of the challenge or, and, and would welcome some recommendations here.

So for each of you, we're in the rapid fire and I'm going to challenge you to to, to recommend one action they could take in the next 12 months that would help to prepare their graduates better for AI enabled work.

So I'm going to start with Grassa as you were.

OK.

So I I think that one action that is accessible to most higher education organization is to create formats of engagement that are not utterly complex and not expensive where you can think about hackathons or you can think about Saint Pete's events or challenge LED opportunities.

These are not highly complex to organise, but we'll and and do that you following the four Helix principle where you have the higher education organization, you have the students, you might have an industry partner providing technologies.

What technology partner and then a problem owner and these four agents together working on small formats of challenges, competitions, hackathons are it's really something that is quite accessible and proved to transform slowly the the the temperature of collaboration and the temperature of innovation within the organisations.

Brilliant.

I love it.

That's a great suggestion and Mike, I'm going to come to you now.

What would your one recommendation be?

OK, I think it's about choosing 1 assessment in a core module that somebody runs and redesigning it so that students have got to really be able to demonstrate responsible critical work with AI.

So not just, oh, you can use Chachi PT if you want, if you're careful with a like really redesign and think about it.

Bring the teaching team together with the students and, and maybe even an employer to identify what that meaningful task might be.

Agree with the students what they're going to learn, how they're going to demonstrate it.

And then make the role of AI clear and, and demonstrate what students are actually going to be able to learn if they actually do that.

So then within a year you could have a tested assessment, some great examples of student works and cases to move forward in your teaching team.

Fantastic.

And Tanzila, how about you?

Yes, I totally agree with them.

But however, the key factor if if we are talking globally, not in a specific top ranked universities are the leading universities.

Definitely the maturity, the awareness and the skills level is totally different when you go to the developing countries or least developed countries are even the universities which are totally well new established.

So there the most important thing is to create the environment by arranging some type of exchange program, by creating the awareness in the student because it's really, really important to make them ready psychologically, behaviour wise and then knowledge wise.

And then you have to see how to implement it and in one year the most, if I look at my university at Prince Sultan University, the best practices as Grace has mentioned before, the senior project, we have the seven months Co-op.

We have the senior projects or we have the senior elective courses.

So when the students are mature enough, they know then we can integrate these type of AI things.

And again, not only the very typical tools, how to create assignments and how to create a research proposal from the chair deputy, but actually how this AI help them facilitate them for their critical thinkings and come up with the best output which can be truly assessed and which can be come up with the impactful output.

So this is the thing I would like to say in specifically for our environment.

That's brilliant.

Thank you very much.

And and thank you for for all humouring me with that rapid fire round.

So I'm, we're going to close now.

So I, I just want to say to our audience, firstly, thank you for, for listening today.

And I hope you come away from this session with a clearer view of what an AI ready graduate might look like and the frictions that institutions might face in developing AI ready graduates, as well as some concrete recommendations for how to overcome them.

We've only skimmed the surface of this research, so I would really encourage you to download the AI readiness report where you can get more detail on the frictions, the some detail on the case studies of how other institutions are overcoming them.

And crucially, a diagnostic tool which will help you to identify which of these frictions are most acute in your own setting and your own context.

So thank you very much to our panellists for answering our questions and sharing their expertise and their experiences.

And thank you all for for joining the session today.