Business, Machine Learning

Alien, Zombie, Astroid and now “AI Bias”


In 2016, a team of scientists from Microsoft Research and Boston University researched how machine learning runs the risk of amplifying biases present in data, especially the gender biases (Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings] ). The research team revealed that word embeddings trained even on Google News articles exhibit female/male gender stereotypes to a disturbing extent.

Until 2009, Amazon was de-ranking LGBT books by mistakingly classifying them as “adult literature” ( Amazon stated: “We recently discovered a glitch to our Amazon sales rank feature that is in the process of being fixed. We’re working to correct the problem as quickly as possible.”

Amazon maybe fixed the problem in one algorithm but in 2016, Bloomberg analysts revealed that Amazon prime same-day delivery service areas excluded ZIP codes by race to varying degrees (

Even today, in 2018 we still see gender bias in machine learning powered general tools as Google translate. The underlying algorithm associates doctor with men and nurse with women when translating between gender-neutral and gender-inclusive languages.

I’m confident that most of the solutions don’t have the intention to include biases, but ignorance is not the same as innocence. The human has a long history of violence and discrimination, and the default tendency of a machine-learned system based on human data is inheriting these biases, causing disastrous effects on the so-known data hungry Artificial Intelligence (AI).

The problem is much broader than to be solved just by changing the algorithms; it is related to every part of businesses, from engineers to product managers and executives. In this article, I’ll surface some of the causes of the “bias” problem and provide few suggestions to prevent it. I’ll be using the term “bias” as a disadvantageous treatment or consideration towards anybody or any group; meaning, being treated worse than others for some arbitrary reason.


From the civic announcements at the Agora in ancient Greece to your news notifications on your mobile app historically information has mostly been “served” to you. It means that unless you did your scientific research and experimentation the information you receive always has some bias. And, unlike some observational error, you would otherwise encounter, the bias in served information will have much more complicated reasons. To start with, I’ll divide the bias in served information into two categories: Intentional and Unintentional.

An example of Unintentional bias is maybe the gender bias being seen between the relationship of the words taken from Google news. An example of Intentional bias is the incredible amount of cryptocurrency news or sided political news I see nowadays.

Until information was in manageable amounts the only source of Intentional and Unintentional bias was human, but once we crossed the line where we had more information than we could consume we gave rise to technologies which added error as well as amplified the source bias significantly. Today, these technologies are in our search engines, news feeds, social news feeds, translators and in many more tools.

One of these technologies is recommendation engines. Recommendation engines solve a massive problem of the digital age: the Information Overload. Even though they can also introduce a filter bubble, it is still the best technology which is available today. For example, at Yahoo, the recommendation engine we’ve developed was able to select handful of news articles with outstanding relevance, out of a million news articles, in single digit millisecond time frame and for each of the billion users. In the absence of a recommendation engine, you would need to read every day through a million news articles to find the relevant articles to your liking. This approach is similar to your LinkedIn, Twitter, Facebook feeds and even for the search engines you use for a general search on your hotel and flight search.

Based on my observation, all information I’m receiving on the internet using standard tools is from one or another search or recommendation machine learning algorithm. And, underneath the surface of these machine learning algorithms, I see four different areas where we need to monitor and control bias:

  1. Data source
  2. Data processing
  3. Model
  4. Inference

Since machine learning algorithms model the data, which can be anything from digitized information to the environment and digital bits, identifying the bias in the source information is very crucial for the down-stream systems to function in a fair way to the human.

Drilling little more into the concept of data sources, I see that there are at least three types of machine learning data sources when it comes to serving information to human: Content, Context, and Activity. Content is the data which is produced with the purpose to be presented directly to a human. Context is the state of the environment relative to the content. And, Activity data is generated in the result of the interaction between a user and content in a context.

Every one of us has a temporal objective, a subjective and intersubjective worldview which we carry into everything we create; into our articles, photos, movies, songs, music, paintings, software and more. On the intentional side, we are consciously aware of worldviews we choose, but there are also some worldviews we are oblivious, the unintentional, which originate from our paradigms and our social circles. These biasses can arise from cognitive error, conflicts of interest, context/environment or prejudices. For example, an analysis done on user comments on daily popular news articles revealed that the average user comment always has a negative tone regardless of the news topic. In contrary, most of the people would disagree that they are negative.

Given the problems in the human history, these biases are not surprising, but things get little dangerous when we use these unintentionally biased content to create machine learning models; the models inherit the biasses with a degree of error and operate on it. Unfortunately, these biasses carried into machine learning algorithms are not visible to human eye unless we deliberately expose them.

Black box AI is the name we give to machine learning models we don’t care to understand. I said “we don’t care” because in most cases, like deep learning, it can become very labor intensive to explain every factor. Especially understanding the bias is a project on its own.

Systems like recommendation engines mainly try to predict user’s behavior based on historical and collaborative internet activity. This approach causes algorithms to create information isolation named Filter Bubble ( The nature of this isolation depends on the representations of the user activities in the system and can be anything from social, cultural, economic, ideologic to behavioral. If not given attention, filter bubbles can be intentionally or unintentionally used to control public opinion towards a particular bias. For example, in 2013, Yahoo researchers found out that web browsing on Yahoo Finance can anticipate stock trading volumes ( This means a bias in the financial news ranking could affect user activity and hence affect stock trading volumes.



It is every data scientist, product manager, and engineer’s responsibility to have a robust strategy to detect, expose and debias the biases in AI products and services. While there are hundreds of possible biases, I think the following most critical biases are a good start for every content based machine learning system:

  • Racism
  • Sexism
  • Cynicism
  • Framing
  • Bullying
  • Favoritism
  • Lobbying
  • Classism
  • Polarity

One of the ways to detect these biasses is to model the bias by using a class of NLP (Natural Language Processing) techniques named Sentiment Analysis ( Today, sentiment analysis is possible by using human-provided training data (e.g., sentiment labels) as well as unsupervised learning techniques like Unsupervised Sentiment Neuron ( Also, in the recent years, RNN (Recurrent Neural Networks) algorithms became very popular in solving NLP problems.

Preventing Filter Bubbles

One approach to avoid filter bubbles is building exploration and exploitation tradeoff strategies. Exploration and exploitation tradeoff allows the system to create a balance between serving information “from outside” and “more about” the filter bubble. Some techniques involve addressing the problem using Multi-armed bandit solutions (

Glass Box AI

Today, we see more and more researchers and companies moving into this area, and creating technologies to explain machine learning models. One of these technologies is LIME ( LIME is based on the paper, and currently, it can explain any black box classifier, with two or more classes.

Another step towards transparency is DARPA’s Explainable AI (XAI) program ( which aims to produce “glass box” models that are explainable to a “human-in-the-loop” (read more about XAI at .) Also, leading researchers like Kate Crawford ( are studying the social implications of AI and bringing more and more awareness to the industry.

On the commercial side, companies like Optimizing Mind ( develop technologies which understand how deep learning models interpret each component of the input.


While we are introducing more an more AI technologies into our processes, it is everybody’s responsibility to understand the bias issues and take necessary precautions.

In this article, I presented just a few aspects of the dangers of artificial intelligence solutions. In our AI courses for Product Managers and C-level executives, we provide strategies to prevent AI issues in the organizations, products, and services. If you are interested, please email me or check out our courses at and


Product Management skills to survive in the AI era


In 2016, tech giants such as Baidu & Google alone spent $20B-$30B on AI, and 62% of all enterprises expect to hire a Chief AI Officer in the future. The share of jobs requiring Artificial Intelligence skills in the US has grown 450% since 2013 and corporations are seeking relentlessly for technical professionals as well as product leaders who can utilize AI technologies on their products and services to improve the company’s bottom line or top line. It is named the Fourth Industrial Revolution, and it is happening right now, right here.

However, as a product manager (PM) or as a potential product manager, how do you gain the necessary knowledge to analyze, understand, plan, and design products based on Artificial Intelligence technologies? Since you can not get today a college degree in AI Product Management, how do you adapt to this rapid change?

As an AI consultant in Silicon Valley, I get to talk to many C-level execs, product managers as well as software engineers who want to move to product management. These are professionals from all size of companies and have two things in common: First, they see the fourth industrial revolution happening and want to make smart moves into AI domains and technologies at the right time. Second, they have a hard time defining a framework and where their responsibilities start and end in this new domain.

Regardless if your organization is willing to use AI technologies in their products, services or internally, the PM role alone is hugely impacted, and drawing the lines between technical responsibilities and product responsibilities is not very easy. To help you with both of these questions as well as guide your way to become an AI Product Manager (AIPM) I want to introduce you a four-step framework.

In the rest of this article I am giving an overview of this framework, and if you are interested in more details, please feel free to contact me. Alternatively, please check out our AIPM certification course where we give the entire training in-depth:

AI Product Manager’s Skills:

  1. Have solid core Product Management skills
  2. Have industry specific business domain expertise
  3. Gain specific AI solution understanding
  4. Gain AI Product Lifecycle knowledge

1. Solid Core Product Management Skills

In the AI era, It is crucial to success not just to understand the technology but also to have the core PM skills. Since AI solutions can touch the vision, strategy, team, product, marketing, partnerships, and support, it is essential to have the understanding how these business aspects operate together. For example, trying to implement an AI solution without the know-how to balance between customer needs, team capabilities and business constraints it is almost guaranteed that the time-features-cost-quality equation will be unbalanced.

Product Managers Need To Be Able To Balance Contradicting Business Aspects

The ways to gain core Product Management skills is beyond the scope of this article. In addition to online product management courses, I have seen that the Business Model Canvas (by strategyzer, frameworks and techniques help enormously to see the bigger picture and be able to put on the second CEO hat in any organization.

Business Model Canvas (

2. Industry Specific Business Domain Expertise

Without specific domain knowledge, a PM will not be able to ideate, design, create and release viable products in that domain. The domain is, in this case, a specific industry or product. This requirement is no different with AI Product Management; the market, the regulations and the business model of the organization need to be understood.

However, even with core business model understanding, it is not always possible to implement AI solutions in those businesses. The reason is that unlike other technologies, AI can bring change to every aspect of the organization and can require different business perspectives. Therefore, it is ubiquitous that the business processes need analysis with an AI perspective. Take for example a visual quality inspection process. The solution is not as isolated as it sounds; we could integrate a feedback mechanism and make the whole production line automatically optimize itself. In such case, not just the end but also the rest of the processes needs evaluation.

Below I will go more into the details how to analyze from an AI perspective, but in general, a Business Analysis Framework like in the following diagram helps to organize gathered information.

Business Analysis Framework

Based on my experience I have seen that for any given business process there are at least four AI opportunities. It is the product manager’s role to go over each business process and identify which of these opportunities are available:

  1. Automation Opportunities
  2. Optimization Opportunities
  3. Expansion Opportunities
  4. Innovation Opportunities

Automation opportunities exist in proven and well-working business processes. One can not automate a broken process. Therefore it is essential to understand the requirements and performance metrics before an automation decision. Some examples of these processes are where human error is high, human performance is too low or recalls rate is low.

Optimization opportunities exist in well working automated processes where usually the software and hardware technology is old and new alternatives are available. PMs need to know the baseline key metrics, the goal and be able to walk through. During optimization projects, the interaction with the science team is usually more frequent than during automation projects. I will mention more about the AI product lifecycle later but at a very high level the PM needs to be able to follow a rapid experimentation cycle with various AI solutions. Also, optimization efforts get more difficult when the actual performance approaches Bayes error, which is the lowest possible error rate for your AI solution. For example, object detection task in the Large Scale Visual Recognition Challenge (LSVRC) Competition already exceeded human performance ( and improving this algorithm further requires effort and new approaches.

Expansion opportunities arise when the goal is to apply working automated processes to different geographical regions, to different product or services. These opportunities are common in large organizations, where in some cases AI capabilities are underutilized and newer technology is available. For example, applying a chatbot solution of one product to another product, after making small changes. Alternatively, expanding an e-commerce recommendation engine to international markets.

Innovation opportunities arise when a new and maybe unproven business process is needed. These opportunities are comparable to creating a new product or starting a new start-up where there is a continuous search for a model. It is an iterative process of defining, measuring, validating various hypothesis to achieve the desired goal. On the technical side, in most cases, a new approach or algorithm is needed which increases uncertainty and overall project complexity.

Complexity is a significant factor when deciding on a technology, methodology as well as team structure in every AI project. It is usually the case that research-oriented solution is more complicated in the areas of the organization, technology, process, and regulation. The complexity of the four AI opportunities is in the diagram below:

Complexity In AI Projects

It is also essential to know the driving factors of an AI project. Since these factors differ from organization to organization, and even from project to project, it is the PMs responsibility to identify them. Below are the top five elements I’ve seen:

  1. Competition
  2. Customer Demands
  3. Market
  4. Corporate Goals
  5. Venture Capital

3. Specific AI Solution Understanding

Today, there is endless information available about AI Solutions on the Internet, and articles range from marketing solutions to how to train an image recognition algorithm. But, the commonly missing information is the strategy and explanation about how a PM can design a solution for their specific business. The truth is that there is no one-size-fits-all solution, and the PM needs to gain the necessary understanding and equip themselves with the right framework and techniques to build a custom strategy, per project basis.

Below is an excellent framework to follow to match any business process to the AI solution space:

  1. Study the relevant AI technology landscape
  2. Study the corresponding AI solution domain
  3. Evaluate AI solution alternatives

The first step is to study to study the AI landscape to understand where the AI technologies related to the business are standing. When we look at different companies and even different business processes, we see that technology did not evolve equally in every area. For example, fraud detection and even advertising are using today very advanced algorithms. One of the reasons is that these industries have been competing for a decade. On the other hand, some industries like healthcare and especially drug development is not using AI as much as advertising.

We dive into the aspects of our AIPM course ( but to give an idea I provided below a high-level timeline for the PMs to determine the stage of the business process.

The AI Timeline

The second step is to study the relevant AI solution domains. AI technologies can be utilized to play the following roles in any business process:

  • Task Processing
  • Decision Support
  • Decision Making

These roles directly map to AI stages shown in the previous diagram and are not always viable for every business process. For example, it looks like critical healthcare systems will never play the 3rd role. On the other hand, today, online advertising automatic bidding systems are making every second substantial number decisions. Therefore, the PMs role is to consider these roles when looking for an AI opportunity and analyze aspects like operability, reliability, and compliance related to each of these roles during requirement analysis.

Nowadays, we categorize the AI solution domain based on three fundamental techniques:

  • Supervised Learning
  • Unsupervised Learning
  • Reinforcement learning

The PM needs to have an understanding of the business solutions in each of these categories and be able to foresee the project lifecycle involved with it. For example, a solution to predict diabetes cases from historical patient lab data will fall into supervised learning algorithms and the nature of this category is to have labeled data and therefore PM has to plan for the labeling aspect.

4. AI Product Lifecycle knowledge

If you are an experienced PM, then you most probably know that when it comes to project methodologies, frameworks, and techniques, there is no one-size-fits-all solution. As soon as I say I’ve seen it all, another method comes along. I think the most crucial PM trait is to know the pros and cons of each methodology and then work with the project manager.

What is the right project management methodology, framework or technique?

AI projects tend to be different from regular software projects by having procedures like data analysis, model research, and experimentation, etc. Therefore, given the requirements and business case, the PM is responsible for identifying the right method for the project.

Below is an example AI product lifecycle with the research and A/B test iterations. I find it also valuable to adopt an agile model and adhere to agile software development manifesto.

product lifecycle2

In this article, I presented just a few aspects of the broader frameworks and methods we teach at our AIPM course. If you are interested in an AIPM career, please email me or contact at ( Also, please join our LinkedIn group at “From Engineering to PM” (

Business, E-commerce

If Your E-commerce Sales Didn’t Grow Last Year By 30% Here Is How Artificial Intelligence Can Help


The $410 billion E-commerce sales are only 9% of all retail sales in the U.S. and it keeps increasing by a whopping 14% YoY. It is predicted to grow to $600 billion by 2020. Perhaps you may not have seen a 32% growth in your e-commerce revenues like Amazon did last year, but did you at least hit the 14% average? If not, here are a few possible reasons why and how Artificial Intelligence can help.

You probably have thousands of products, a website with the latest e-commerce features, a mobile app. You may also have invested millions in brand awareness, user acquisition, and user behavior analysis but did not see any significant lift in your top-line.

When it comes to marketing funnels there isn’t really a magic formula; the success comes by doing small things in a big way. There are countless aspects like customer service, return policies, deals, customer reviews, seller reviews, autocomplete suggestions, finding items, displaying customer reviews, product promotions, price comparison, aggregated listings, rich product information, discounted shipping and much more which you need to get right.

In fact, take a look at the solutions of the top e-commerce retailers today. Their product offerings, websites, and apps are very similar on the surface yet Amazon has 43.5% market share compared to 6.8% of its closest competitor eBay: (source Recode )

  • 43.5% Amazon
  • 6.8% eBay
  • 3.6% Apple
  • 3.6% Walmart
  • 1.5% Homedepot
  • 1.4% Best Buy
  • 1.2% Macy’s
  • 0.9% Wayfair
  • 0.9% Costco

But, what are these small differences between solutions and how can you improve upon them? Let’s take a very simple feature like “search autocomplete suggestions”, and see how this technology differs. Below are the results of “search autocomplete suggestions” feature for the “Desk lamp” keyword on three different websites:

One easy-to-spot difference is that Wayfair is expanding the query by prefixing categories, whereas Amazon and Walmart are using some kind of intelligence. At a very high level, the intelligence seems to be a combination of past public query patterns and maybe some additional logic. Since “Desk lamp white” is not even a correct English sentence I would say there is a flavor of similarity based on language understanding. The “Desk lamp for an office in home” top suggestion of Wallmart is a very niche category and isn’t anything I’m looking for. It is also surprising that neither of the solutions seems to have personalization built-in; regardless of the account, the autosuggest results are the same. It is possible to fix the irrelevant suggestions with a personalization solution which includes user’s own query and purchase history.

There are also many more data points we can associate with a search term to make it more intelligent. These include:

1) the quality and quantity of the product data related to the search term,

2) the cumulative product review scores related to the search term,

3) the cumulative product sales volumes related to the search term.

In summary, optimization objective of a “search autocomplete suggestions” is maximizing sales and that requires finding the balance between many attributes.

Another example of intelligent features on Amazon is its real-time price comparison. We know that Amazon does not always have the lowest prices but it has the intelligence to fine-tune the prices in real-time such that it will produce wider margins on less competitive items and private-label goods where the user is less likely to compare prices.

This kind of intelligence requires vasts amount of data extracted from pricing, customer reviews, product rank, customer-product behavior, collaborative aspects etc. and a vast amount of processing power to support the volume of 183 million monthly active users. This is where artificial intelligent solutions come into the picture.

In addition to Amazon, many large tech companies like Apple, Google, Facebook, Intel, and Microsoft have invested decades into artificial intelligence and now they are riding the wave by reaping the benefits. Currently, the reason that the sales gap between Amazon and its competitors is growing is due to Amazon’s ability to collect vast amounts of data and use artificial intelligence to understand it and then feed it back to the marketing and sales channels. Hopefully, this is not a complete surprise to you but perhaps you feel constrained in jumping on the artificial intelligence bandwagon due to constraints such as domain expertise, resources, and costs.

(Image from AI Index 2017 Annual Report


According to Stanford University’s 2017 AI Index Report between the years 2000 and 2017 the number of AI papers increased 8 fold, annual VC investments into AI startups increased 6 fold and the number of startups increased 14 fold. In 2017 alone a record $6.5B of capital was deployed across 650+ deals, surpassing all 2016 numbers.

Today AI is increasingly being viewed as the 4th industrial revolution; it is the new electricity.
(Image from AI Index 2017 Annual Report


As with any new technology, especially at this scale, we see more questions than answers. The most common question is “how can I move my company to the AI era?”. One of the reasons for the abundance of questions is the lack of AI expertise in the market. The demand for AI experts is high but there are not enough of them out there. The AI Index report sheds light on this expertise shortage and shows that Stanford had only 1500 AI/ML course enrollments in the last year. Comparing this number to 3.8 million software engineers in the US it seems like it will take quite a bit of time to have AI skills commonly available. Also, you will be competing with the large tech companies like Apple, Google, Amazon, Facebook, Intel, Microsoft and others for such talent. Which brings us actually to another shortage, that of AI product and service companies. AI is trending and this has affected the start-up ecosystem as well. According to CBInsight in 2017 alone, over 55 private AI companies were acquired, mainly by the most active large tech companies Google, Apple, Facebook, Intel, Microsoft, and Amazon. Until the large companies are saturated with enough AI startups we will have a shortage.

So, what should you do?

Luckily, even if you don’t have an AI lab with hundreds of experts with PhDs, most of the AI components have now become mainstream cloud technologies and you can work with an AI integrator to speed up your integration. Today, e-commerce solutions like the ones below have become readily available to you at affordable prices:

  • Personalized search
  • Optimized rankings
  • Email recommendations
  • Behavioral triggers
  • Smart content personalization
  • Content sequencing
  • Product recommendations
  • Personalized engagement
  • Persistent visitor intelligence
  • Actionable insights
  • Conversion rate optimization
  • Omnichannel marketing
  • Price Optimization
  • Product insight

Ironically, you can compete with Amazon by using another Amazon product, namely Amazon Web Services (AWS). Besides its broad solution space, AWS has a set of AI solutions which can be combined and integrated easily in a matter of weeks. Going back to our example, AWS even has a commercial“search autocomplete suggestions” solution you can integrate into your system.

As a final note, I want to share with you a list of almost all Amazon AI services ready for integration as of today ( Of course, as much as there are other cloud AI service providers there is also a great deal of AI technologies more on the horizon to come.

Amazon Rekognition Solutions

  • Image Object detection
  • Image Face detection, search, tracking, compare
  • Image Celebrity detection
  • Image Unsafe content detection
  • Video Object detection
  • Video Face detection, search, tracking
  • Video Celebrity detection
  • Video Unsafe content detection
  • Video Person detection

Amazon Comprehend Solutions

  • Key phrases extraction
  • Sentiment Analysis
  • Entity Recognition: places, people, brands, events
  • Language Detection
  • Topic Modeling

Amazon Lex (Chatbot) Solutions

  • Automatic speech recognition (ASR) (speech to text)
  • Natural language understanding (NLU)

Amazon Polly Solutions (~25 Languages)

  • Text-to-speech
  • Text-to-speech marks

Other Amazon AI Solutions

In my next article I’ll cover more of the solution space and examples. If you want to learn more about how your business can benefit from AI please visit “Move to AI” at, or stop by at one of our Silicon Valley Executive meetups.