Tag: AI

To read the popular press, AI can outdo humans at anything, but the truth is far more complex. AI applications are typically to do just one thing well, such as when Watson took on all comers in the Jeopardy game show. But while many of us are fine with letting computers play games, polls show that many of us are distrustful of self-driving cars. And that trust is a key issue, because otherwise valuable applications will be slowed down or even stopped if people don’t trust the technology. Research also shows that when AI systems give incorrect answers too frequently or answers that make no sense to humans, it reduces their trust in the system.

So, how do you build human trust in AI?

Explain the system’s decisions. There are calls for explainable AI, where the system must provide an explanation of how it came to its decision. This technique is still R&D, because techniques today are notorious for being proverbial black boxes. The problem with the research is that when you force systems to use only those techniques that are explainable, they inevitably work worse. Someday, this might be the answer, but not today.

Improve the system’s accuracy. The reason you want AI to explain itself is so you understand how mistakes happen. If you can make it work well enough, maybe no one needs an explanation. After all, most is us don’t know how our cars work, but we trust when we apply the brakes that it will stop. Hardly anyone knows how Google’s AI works, but we trust that our searches will get us good results, so we keep using it.

Reduce the really big mistakes. Watson once gave a really bad answer to a Final Jeopardy question in the category US Cities, providing the response of “Toronto.” We call that a “howler”–an answer so bad that even if you don’t know the correct answer, you still know that response is wrong. You can actually tune the system to reduce howlers by scoring bad answers as worse responses than wrong answers that are “close.”

Put humans in the loop. This might be the simplest of all. Instead of treating every AI system as one to replace humans, maybe it is easier, safer, and more trustworthy to set up the AI to help the humans do their jobs. Watson is being used to diagnose diseases, but rather than replacing doctors, Watson shows the doctors possible diagnoses based on the symptoms, with the doctor making the final decision. When decisions are so high stakes, this might be the most prudent approach.

AI is no longer science fiction, but people are understandably nervous of this kind of powerful force that works in mysterious ways. We need to pay close attention to building human trust in the system to see AI used in the safest and most valuable ways possible.

I grow weary of reading the simplistic headlines around the impact of AI. Some people say that AI will put many of us into a new leisure class that doesn’t need to work. Others argue that AI will make us all unemployed. They are both saying the same thing, actually, so it is just a personality test to divide optimists from pessimists. But there is no technology that in the past had that kind of impact, so why is this one different? It probably isn’t.

What is much more likely is that as machines do more, we humans will do something else. Something machines can’t do yet. That’s the way it has always been, so I think that is the way to bet.

What fuels my belief that this is true is that the most powerful AI we see today depends on human judgement. No, I don’t mean the highly-paid data scientists and AI engineers that are all the rage these days. Sure, they are important, but I am talking about ordinary people doing ordinary jobs using judgement that computers just don’t have. This technique is called semi-supervised machine learning or active learning.

Here is how it works. Supervised machine learning is what most AI applications use. They need human judgement, too. But they use it only at the beginning. They ask humans to tell the system the right answer to a question–for example, whether a tweet has positive or negative sentiment. You pile up enough tweets with human answers and use that to train the AI system. So, far, so good. But that is where most systems stop.

The most powerful systems keep getting better, using semi-supervised machine learning. The secret is something called the confidence score. Most AI systems can do more than just answer the question. Beyond telling you that they think this tweet is positive or that tweet is negative, they can tell you how confident they are in that opinion. So, the system might be 90% confident that this tweet is positive and just 60% confident that another tweet is negative.  This provides some interesting possibilities for semi-supervision.

You can set up your system so that your system handles automatically any tweet with over 70% confidence. If it is that sure of itself, let it provide that answer on its own. But if it is less than 70% confident, you can refer that tweet to a human being to check its answer. Is that tweet negative–the one with 60% confidence? Checking the answers the system isn’t sure of is semi-supervision, and it has two benefits. The first is that the system is more likely to get the answers right if it can ask a human to check its work.

The second benefit is that each new human answer is new training data that the system can use to improve its model. By constantly asking for help with the answers it is least sure of, the system is improving itself as rapidly as possible. You can add more training data at any time to any machine learning system, but if your new training data is merely adding more examples of what the system is already doing well, it doesn’t cause any improvement. Only by adding new training data in the areas that the system is getting wrong does improvement happen.

So, yes, machine learning is very important. But semi-supervised machine learning is what provides that most rapid way of continuously improving your machine learning application. If your team isn’t using that approach, it might be time to ask why not.