6.3 Can we put humans in the loop? - Video Tutorials & Practice Problems

<v ->One thing to think about</v> when we're thinking about people and AI is a technique called human in the loop AI. And what I wanna do is to give you an example of how that works, because when you can add humans to your AI process, so it isn't completely automated, that's gonna give you some types of advantages that you might not have with other approaches. So let's look at an example of human in the loop AI. It's an example we talked about before, where you're classifying content to try and match what the audience interests are. So we might wanna understand what their topics are or what content types or industries. So there's all sorts of ways of classifying content. So let's look at how we do this. Maybe look at a traditional ways of doing it and then look at how human in the loop might make it better. So you start out with certain types of taxonomies. So you might know that word from science, where they have genus and species of animals. And so you have the same type of thing going on with content. You might have a topic taxonomy where all the different subjects have some type of hierarchy, same thing for a content type. So the form of the content, whether it's a white paper or a case study or a product specification, these are all types of content and they might have their own taxonomies as well. But also you could look at industries. So there are certain types of standard taxonomies that you have for industries. There are North American Industry Classification System, the NAICS codes, there's all sorts of other ones, SIC codes, Standard Industry Classification, and visitor identification systems for B2B often tell you what industry someone's in. So maybe you could use that to show better content on the fly to a visitor to your website. And so these are all reasons that you might wanna have a taxonomy where you're going to tag your content so that you know what it's about. So let's first just look at the example of website topics, so that topic taxonomy. So topics can have many uses, so knowing the subject of a page on your website, you can use it in site search for facets, for drill down. You can use it in personalization so that you're showing content, that's the right topic that people are interested in. Your CMS might help you assess whether you have coverage of all the topics you think you do by looking at how many pages there are out there. And that can help you when you're trying to cover topics for search engine optimization as well, and maybe just to standardize your nomenclature. You can have one central group that's kind of naming certain things so that you're using it consistently through your content. So there's all sorts of reasons to have that type of topic taxonomy. And in fact, just that site search use alone of facets, which you've probably seen in lots of different sites search facilities, that can be valuable enough all by itself for you wanna have a topic taxonomy and for you want your pages to be tagged with those topics in an accurate way, so that the search engine knows which pages are about which topics. And so how could you do this? Well, a lot of the times the way you would do that is to really start with the content that you already have and try and understand whether your content is authored in such a way that the search technology could maybe figure out what the page is about. So sometimes you might not need to tag the pages by topic, if your search technology is working well enough, but often it can be helped by adding metadata to your page, that that's taxonomy data as well. And by tagging the page, by having an author go into the page and say, I'm selecting this topic or these topics about that page. Now, in order to do that, you have to have some kind of standards. So you remember when we talked about having coding standards, having instructions for people so that they do it consistently? Well you'd need to do that. Now the problem is though that a topic taxonomy often has dozens of different topics in it sometimes even more, and that can kind of be hard for people to do. And so that's something that you really need to think about. We make the assumption that when a person tags a page by hand, that they're gonna get it right, but this is actually something that's very hard for people to do. One of the problems is that there's kind of too many categories. If there are dozens of choices, there are many situations in which a couple of them might be kind of similar to each other and it's pretty easy to get things wrong. The other problem though, is that people are inconsistent. And with a lot of these systems, they don't use that inter coder agreement technique that I've been hammering away at. They just have the author tag the page, one person's opinion. So there have been studies done that show that if you show the same content to the same person, a couple of days apart, 35% of the time, they actually disagree with themselves as to what label it should get. And so if a person can't even agree with themselves, two days later, you could imagine how bad it is when you try to check to see whether people agree with each other. And so human tagging can actually be very problematic for some of these situations. It isn't that the computer using AI will always tag it correctly. But what is true is that if you have errors in your AI process, using all of the techniques we've talked about before, you can actually correct them by making the AI more consistent by actually improving and attacking those errors. But the problem with people is you can't make people more consistent. If this task is just really hard for people to do, it's really hard for you to make improvements in the process so that they get better at it. And so let's look at kind of traditional ways that computers have done what's called topic modeling in the past. They use a technique called clustering. So what they do is they actually look at all of the different topics that might be out there and they don't really know which ones make sense. So they're just looking for patterns of pages and grouping them together. And sometimes they make sense and sometimes they don't. And so that can be a problem because you wouldn't wanna show the ones that don't make any sense to people. Or they do manual taxonomies. The problem is that those can be really slow and you get a lot of disagreement. If you lay five subject matter experts end-to-end, they all point in different directions. And so it can take a long time to get agreement within your organization that these are actually the right categories. The other problem is that if you do things manually, then you really want those taxonomies to reflect the documents you have. Sometimes we've seen clients that show us what their taxonomy is, and one of them is sustainability. And we go and look, and we see that they've got 40,000 pages on their website and five of them are on sustainability. And it's like, well, I get that you wanna be talking about sustainability, but you kind of, don't, that's kind of more of an aspirational topic for you. And so it probably doesn't belong in your taxonomy if it's only about five pages. The other thing that's hard is that if you do try to automate your classification with a manual taxonomy, it's pretty hard to do because there's a lot of overlap between topics so that the patterns aren't very clear to your AI model as to which topic it belongs in, because there's so much overlap that you've created with manual topics. And that's actually another reason why it's hard for people to classify into it. And so then what would you do instead? So here's an area we're adding human beings to an AI process can actually make it a lot better. So first you can start with the machine. So first you start with that kind of clustering approach that we talked about, and it's gonna come up with all sorts of groups of pages that it thinks are similar to each other. But as we mentioned, the problem with that is that a lot of them don't make any sense to people. I remember doing a project when I, for a tech company, in which one of the things we did was we ran this clustering approach, and we said, okay, so what kind of topics do we see? And so the first one we got was relational database. So it's like, well, that looks pretty good because they actually have products that are relational databases. The next thing we got was a topic that looked like technical support, and like, all right, well that's kind of odd technical support relationship with databases, they don't even really, they're not even remotely the same kind of topic. That's kind of weird. Then we looked at the next topic and it looks like, well, it was pretty clear what that one was, French. It was all pages written in French. And it's like, okay, well, this is not helpful at all. And so what, if you had a page that was about technical support for a relational database written in French. I mean, wouldn't, all three of those topics could be valuable. And instead what this is doing is it's showing us three topics that don't even seem like they're of the same character as each other. And so just using automation to do this doesn't work, but then what you can do is you can put a human in the loop. Have the human being take a look and say, hey, that topic stupid, get rid of that, we don't want that one. Hey, these two topics really are the same topic, why don't you merge those together. And so by doing that, by putting a human in the loop, you get all the power of the machine, actually putting topic taxonomy together, based on your actual contents, you don't have that problem of those aspirational sustainability topics, they won't show up. So it's based on your real content, but it's also putting together patterns that will make it very easy for them to classify later because the topics don't overlap very much and what the human is doing is making sure that these topics make sense to people. And so putting a human in the loop can really, really help. So, first the machine identifies the topics from your actual documents, your website say, and then the human actually changes them. And then you rerun it again, and then you rerun it again. You do it as many times as you need to, until it's producing a very small set of topics, you know, a few dozen, maybe rather than hundreds, that really make sense to people. And this is gonna be something where now you've got a taxonomy that you can very accurately classify into. And it's really machine learning that makes all of this possible. So you have humans in the loop at each step, they're training the model, they're labeling the data, they're assessing it, and they're coming up with the taxonomy itself. And this is a way for you to put humans in the loop to get much more accuracy versus just trying to have a pure technical solution.