Daniel Emmerson 00:02

Welcome to Foundational Impact, a podcast series that focuses on education and artificial intelligence from a nonprofit perspective. My name is Daniel Emerson and I'm the Executive Director of Good Future Foundation, a nonprofit whose mission is to equip educators to confidently prepare all students, regardless of their background, to benefit from and succeed in an AI infused world. This podcast series sets out to explore the trials and tribulations of building a nonprofit from the ground up, while also investigating the changing world of technology in life, learning and work. Today, I have the privilege of speaking with Rohan Light, who began his data governance career leading a technology project in 2010 for New Zealand Inland Revenue before working with the Data Futures Partnership on re-identification risk in 2017. In 2018, he worked with the New Zealand Ministry of Social Development on responsible information use and the New Zealand Ministry of Education on openness and transparency in 2018. Before working with the labour regulator Employment New Zealand looking at the gig economy in 2019. Over 2020 to 2022, he engaged heavily with overseas colleagues in governance, risk and compliance, AI assurance and AI ethics, as well as speaking at events and guesting on podcasts. His main published work is the Self Synchronisation of AI Ethical Principles. His current area of responsibility is working on health data governance for Health New Zealand. Rohan, it's wonderful to have you with us. If we could just start out, Rohan, by giving our listeners a little bit of context with regards to your work, particularly For Humanity, and how that feeds into the conversation around artificial intelligence and ethics.

Rohan Light 02:04

That's a good intro. Artificial intelligence and ethics, it's a big combination of two very large elements, the technology space and the human space. The For Humanity is a not for profit charity. It's looking at ways of bridging the gap between the ability and non ability to assure and provide some audit oversight as to AI broadly understood. The field, looking at this connection between technology and society is old, that's for certain. However, the from about 2017 onwards, more care and concern was raised around the current evolution of technology driven by social media, etc., and that led to searching around for different frameworks, different bodies of work, different professional bodies of work that could be bootstrapped from one domain to another. So many professions, for instance, will derive their rigour from the financial side of the world simply because they have to audit a lot. So it was a big deal before OpenAI, before ChatGPT landed that did change the game considerably, and I would say for many in the wider AI assurance and AI ethics field, it was the expected moment, not a great moment to wait for, but recognizing that when the awareness hits of the large amounts of digital intermediation, then many professions will suddenly notice it. And that's what happened.

Daniel Emmerson 04:12

So when you're conducting your audit and when you're looking at that, interplay the humanity and technology with a focus on artificial intelligence. What does that look like? What are you looking for and why is it important?

Rohan Light 04:26

That's interesting. In my current work it is around, assurance is about being able to provide an account to someone about something. The governor, I'm not answering the question, but I'm circling around it. The basic approach or the basic function that I think data governance, AI governance needs to do is what I would call the agent based approach. Help someone explain system behaviour. That's it. So it doesn't matter if you're the chief executive or you've just arrived. If something happens around a data system now that could be a data, a simple novel data system, we're surrounded by them. Or some high grade, I don't like to use the term, something very, very sophisticated foundation model, something on the frontier of machine learning. Or actually it could, to be honest, still happen, just an unsecured server or someone spilled something they shouldn't. We want to be able to provide any of those people the means to describe system behaviour. Okay, so it's like a precursor for accountability and transparency. But transparency needs to show agency. Could the individual have reasonably intervened? It's the basic question. So the assurance based approach to data governance looks at that. And we have on the one hand, as I said, the agent based approach. But you also have very large programmatic approaches for humanity. As an example of a non government approach, you also have the entire body of work of Luciano Floridi. Jessica Morley is a favoured person to follow, if you are following. She's doing health at the moment, but education is never far away from the use cases that we look at. And I'll crawling to your point, because we basically want our technology to help us grow. We have had instances in our industrial history of turning that around and people were subservient to the machine, so to speak. And what we have is that same, same old challenge. It's just that artificial intelligence, broadly understood, has suddenly appeared everywhere.

Daniel Emmerson 07:08

And do you think that's going to increase that level of subservience because of what this technology can do?

Rohan Light 07:15

Great question. That's my third question. You've just gone straight to the big one. I'm an optimist. So one thing I'm watching at the moment is the generational response, the current generation. How are they responding to technology? One marker is the growth of dumb phones. People just want the basic phone again, it's a sign that the technological apparatus around us has packed too much stuff into the simple device. It's burdened a person out of control and actually all they really want to do is send a text, take a photo, and that's kind of it. The, in that sense, the, also I'm watching. I quite like the fashion industry to watch. And the growth in dupe culture and long term fashion, long term is in, you know, very low footprint. You make something once and wear it a million times. That's all growing too. So I would suggest that actually we may be. I don't think we're circling, heading towards a desperate future. We are heading towards a future where we need to really think about the extent that we devolve our thinking to a device. That's the bit to worry about.

Daniel Emmerson 08:51

I guess there isn't, well, there certainly isn't as much of a public discourse as it feels like there needs to be around this conversation when you look at how big tech companies are incorporating artificial intelligence already into existing products that people can buy. So, whereas last year, for example, when we were speaking to teachers and students about artificial intelligence and feeling comfortable and confident in being able to utilise that in best practice, it almost feels as though that's been eroded because of how artificial intelligence is being incorporated or brought into existing products and programs. It doesn't feel like as much of a separate thing anymore. I'd love to get your thoughts on that and whether or not that's a risk in itself.

Rohan Light 09:42

We are at the back end of a long digital supply chain now. That supply chain can be updated far faster than we can adapt to it. What happened was from when GPT-3 dropped, we started thinking, this is going to be different. It suddenly was. And yet its development time was decades. Decades, and it will still be decades, though more and more people are starting to realise, I think, that large language models have a fundamental limitation. So it feels like it hit us fast. But actually you're in the UK. Cambridge Analytica was the big data governance event that woke us up in the 2010s. And from then to the UK GDPR to several additional changes, I would say certainly the UK population is, well, moving through its adjustment phase. What has has happened is large language model is really easy to market. It can produce pretty words and great pictures, it can be bolted onto virtually any workflow. It can suddenly appear just with your overnight updates. All of a sudden, people could find themselves. Hang on, if I just go to Copilot I can skip all of the other stuff. It tells us that the entry points for the digital experience is everywhere. And note, I am being very careful not to use the term AI. It is as vague as the term mammal. It covers so many things, and the use cases are all different. And that's my number one prepared point for this conversation, is whenever we're talking about AI, make sure we know what we're actually talking about. And if the other person can't actually nail it down, and you can't nail it down, that's a warning sign. It means it's indeterminate or ambiguous. More than one possibility, the stakes are too high to just go from soundbite to soundbite.

Daniel Emmerson 12:10

I'd agree with that. And I'd say we're trying, I suppose, not to perpetuate that within our training and our conversation. And indeed, I suppose as far as Good Future foundation is concerned, one of the first steps for us is identifying an AI 101 by breaking that term down. What does it mean? What is a large language model? How has this become part of our day to day conversation in almost any sector, not just in education, but at the same time trying to keep people on site, keep them interested, keep them engaged and focused on what it is we're talking about. But it isn't until people see, for example, a generative AI tool that they can understand how it might prove to be useful in their workspace. And when we're talking to teachers, that's ultimately about reducing their workflow on administrative and bureaucratic tasks that detract from the thing that made them passionate about being teachers in the first place. So we're looking at quite specific examples. I suppose when it comes to generative AI tools, however, it feels as though there's a much bigger conversation around that. There is a much bigger conversation around that when it comes to how these tools are being used, what information is being input into these tools. And so when we're looking at best practice, for example, we have our five golden rules, for example, about personal information, private company or school data. Essentially, anything that you put into the generative AI tool goes into a black hole, and you can't be sure as to how it's going to be used. Moving forward, how much of, I mean, you were talking earlier about definitions and being careful about terminology. When we're speaking about artificial intelligence or generative AI or machine learning or large language models, how important is it, do you think, to break these down? How much detail do we need to go into when we're speaking to people who perhaps are approaching this subject for the first time. They know it's important, but they're not perhaps sure how it fits into their day to day life. Is there still a degree of importance there? And I'd love to get your thoughts on it.

Rohan Light 14:33

This is one of the big questions. How does it fit into day to day life? One of the interesting observations in the industry right now is the large numbers of failure rates with large language model applications. So it tells me with my background that we have possibly an implementation with quite a narrow use case, or it might, it might just, it might fluctuate regularly, or it might actually fluctuate irregularly, which is the hardest thing to provision. So I would, I would say that actually, if, if the explanation of the tool that's coming down to you as a teacher really doesn't make sense, then it probably just doesn't make sense. You may not need it. Number two, though obviously the kids are all using it. We know this, we know this. Students teach the teachers as much as the other way around. So then it goes to the extent to which, as you said, the degree with which the technology can return time to the key interaction. And one of my notes here was actually one of the, surely one of the key interactions is still the group, simply put, sharing time, learning time with each other. So we then go, if large language models are being touted as a solution to back office administrivia, is that back office administrivia not already under your control? In which case you can possibly look at what you're doing without a large language model doing it for you. So that goes to a very utilitarian side for the teacher right now, let's just look at the element of a prompt into a large language model. It's a giant dice roller, and out the back you will get a response that may never recur. And also the dice you rolled, my metaphor is going to break down, but it's accessible to many other people in that same dice rolling room. Now I say follow the boosters. Now, Ethan Mollik. Ethan Mollak is a really good, articulate booster. He's mainly going to give you what works. But I say at the moment, because overall, the overall geopolitical climate's not great. So things are feeling bad, people are going to feel bad about AI. We can't actually afford to have that happen. So it's very important to follow erudite in really good boosters. Ethan Mollock is one, then. Yes. So my point there was they will tell you where the actual useful use cases are. You know what? It might be okay for a class to just bounce words at each other via a large language model. So what? And yeah, maybe quite easily you could imagine, I certainly can imagine, the ability to engage people of the link between art and words. Many of these generators have great artistic capability, obviously. They also have, you know, terrible, terrible applications as well. And this is what does make it a double edged sword. It does feel like to me it is a topic that requires honestly engaging with kids who have hit a large enough age to get a sense of the consequences of actions.

Daniel Emmerson 18:46

What about an institutional level then? Because we're finding as well that often decisions around whether or not a tool might be implemented, for example, in a school setting, lies with the decision of perhaps two or three people in a senior leadership capacity. What would you say are the key things for them to be looking at when it comes to implementation?

Rohan Light 19:11

Number one is the implementation is going to be as successful as your engagement to date has been long. So if you're suddenly dropping it on people, it's just going to hit the floor. So if you've left your engagement too short, your implementation is going to not really go well. However, if you're in your local school environment, being realistic, there have been technology upgrades going in there since the eighties. Why have we suddenly, you know, we've suddenly woken up to one part of it, one element of it. I hope that it leads to a good topic of discussion between the teaching community and parental community, because the kids are balanced between those two and where there is a big disparity. I think you have some problems emerging just for the kids.

Daniel Emmerson 20:09

And from a technical perspective, should or is there a role, perhaps even for the government, when thinking about it from an institutional perspective? And what does that role look like, do you think?

Rohan Light 20:22

Okay, this is one of the questions of the age, obviously, regulation, intervention by the state. Some people run towards it, some people run away from it. It depends on the risk state. If you're suddenly in danger, then you usually want help.

Rohan Light 20:41

After many years in public service. Usually, yes, there is a role because the upside is so significant. The advantages need to be clearly spelled out and the marketing hype is out of control, so it needs to be tamped down. And there are unique combinatorial risks to these things.

Rohan Light 21:07

So all three of these areas are areas where governments do make statements. They depend on the government. Ours down here we have several different official organs of government making stepwise changes. Growth in the data governance toolbox, toolbox that, for instance, I can bring to bear.

Daniel Emmerson 21:33

Right.

Rohan Light 21:33

The issue is, because data of which AI is made can be infinitely re combined out the backend, there are so many unknown use cases, and this is the argument of the don't regulate too much approach. Let us see where the use cases develop, because there are a large number of risk and assurance people like me saying, tell you what, the use cases are going to be quite smaller than what we've been promised. And at the same time we have to also bear in mind that we can pour water on really good developments, so we have to find ways to safely test them. Now this comes to active rollouts in schools and every board and every management team, you know, they've got an investment schedule. Three years ago they put a few million quid into this contract and three, and it's now year three and it's starting to pay out. In those two previous years, the investment was a net cost. It comes back to this point that we have had a happy or an unhappy coincidence of events, depending on how you look at it. And what we can't do is think the system we had in place will perpetuate unchanged.

Daniel Emmerson 23:04

It feels like a bit of a wild west at the moment though, right? To piggyback on what you were saying about how it's being marketed and how it's being, it almost feels like as a school leader, you need to be picking this up and doing something with it. There are so many different stances as far as governments are concerned with what best practice might look like. There isn't really a gold star that you can point towards and say, okay, if I'm going to bring this into my school or my company or whatsoever, I need to be looking at this set of criteria when it comes to an ethical approach or safety or a data policy, is that something that you think is likely to change as governments perhaps get a firmer hold on what the level of expectation is, particularly if you want to play in the k twelve school space.

Rohan Light 23:56

Gotcha. Right. So the first point is actually, governments usually have a ton of existing regulations they can bring to bear. There's nothing actually uniquely special about AI. It's software plus data plus machine learning and some statistics. I greatly. Yeah, it's not that it's more than that, but it doesn't have its own particle, it's not its own unique thing. We can govern it with what we already have. This goes back to the failure rate of the use cases. So the second part of your question was, if you're entering this space, if you're wanting to constructively build data infrastructure that can carry a safe interaction between society and human and AI. What do you do? How do you approach it? So the first thing is, in the UK particularly, you've got a really good Privacy Commissioner ICO. That's right, ICO. You got him from us. So I like him. He's really good. Mr Edwards. Great person. But you've actually got a really good watchdog there. Really good advice. Follow that. Anything that can go wrong with data will go wrong with AI, just times ten. So if you already have flaky privacy practices and you bring in AI, it's just gonna, it'll blow it up really quickly. But if you've got really solid fundamentals, you'll be okay. And this is the, again, the message is safe to experiment. What we have in place is already really good. Public enemy don't believe the hype, but experiment into your own space. Experiment in. And the reason why you have to experiment and chuck some science at this is that you are looking for your own use case. It is likely to be about time saving. And unless you can really get an understanding of your workflows, how you can change your workflows, how you can save time, how you can waste time, how you can generate work for yourself, that takes you nowhere. This assessment of your own human technological workspace is the way to go. You don't need to pay a lot of money on consultants, you don't need to. If you're aware of what you do, you can find a way to make it work, as long as you remember it's made of data. So follow all the usual data rules and look at her and follow a good booster like Ethan Mollik.

Daniel Emmerson 26:52

That is an awesome place for us to wrap up. Rohan, thank you ever so much indeed for the conversation and for your time. It's always such a pleasure listening to you. Really, really appreciate it.

Rohan Light 27:04

Thank you. I've loved it.

Voice Over 27:07

That's it for this episode. Don't forget, the next episode is coming out soon, so make sure you click that option to follow or subscribe. It just means you won't miss it. But in the meantime, thank you for being here and we'll see you next time.

‍