Toasters for AI

Some Background

I was tester for brief period of time in 2011. I was also part of an organization where quality of products has been rigorously tested.

Later, around 2014, chose Artificial Intelligence (AI) as my field that I want to excel in and working in it still in 2020.

In the last decade, I have built teams of AI, delivered some of the awesome projects for various clients & build products. Be it predictive machine learning, recommendation engines, chatbots, deep learning applications, big data applications. I tried my hands wherever I can.

The need of testing Bringing Quality in AI products

The experience I gained – taught me lot of about value of testing these products. While I was building products and delivering projects – I simply couldn’t find a tester who can do testing of my AI products. This was around 2018. I tried asking few thought leaders into testing & started working towards how we can bring effective quality procedures for AI applications.

I also found there aren’t much information available on internet who can help me learning testing for AI. It made my task harder and challenging. Fortunately – one of good piece that I found is Martin Fowers CD4ML – where he has written extensively about building a continuous delivery application for Machine learning systems. Please read it, it’s a great article.

Toasters – to be toasted by testers!

Title of the post – Toasters of AI , is an analogy of scenario where AI could fail. In fact we can fool AI to recognise/behave in certain way without explicitly mentioning.

Imagine you are building a image classification or video object detection applications using deep neural networks. And you have a test dataset where you can evaluate an accuracy of model by observing predictions on test dataset.

Now you tweaked few images to make it look like a toaster. But they aren’t actually a toaster. If you pass the image through model – you will probably get recognition as a toaster, when there is nothing.

“Toaster” is just an analogy and frequent word in AI teams to mention that machine learning model that was trained is not robust yet and can fail to noisy signals or data.

In upcoming blog posts – I will be introducing good number of ways you can toast your AI applications – by bringing (or adding) toasters. I will be writing about AI and specific things testers needs to learn so that they can be well-equipped with knowledge before starting a testing project for AI.

Hope you enjoyed till here – if you are a tester , do subscribe for the future blog posts.

Until then, stay safe and healthy!