What keeps challenging the Machine Learning industry

Today, the fourth edition of Applied Machine Learning Days kicked off its keynotes. Here are my top three takeaways.

Meaningful data remains a challenge

Data keeps being the main asset, but at the same time making this data F.A.I.R (findable, accessible, interoperable and reusable) keeps being a main challenge. Companies and their C-suit seem to realize this more and more.

• Christopher Bishop of Microsoft Research Lab made (again) a great case for compute & data beating domain knowledge in the long run, referring to the “Bitter Lesson”, a recent blog post of Rich Sutton. Worth a read!

• Asif Jan presented how Roche is running a dedicated program, Enhanced Data and Insights Sharing (EDIS), integrating clinical trail data to create large, high-quality, disease specific data marts. He says a good curation pipeline is key. Roche goes so far as to get permission from patients to collect data on behalf of them from every clinic the client has been to before.
• Vas Narasimhan, CEO of Novartis, recently highlighted the importance of maintaining outstanding data in an interview with forbes:

“The first thing we’ve learned is the importance of having outstanding data to actually base your ML on. In our own shop, we’ve been working on a few big projects, and we’ve had to spend most of the time just cleaning the data sets before you can even run the algorithm.”

Takeaway: Collect and curate the data!

Integration of ML into production is hard

Jeffrey Bohn from Swiss Re Institute reiterated on the (still) large failure rate of ML projects, if we measure success as ML-integrated-into-daily-processes. “Probably in the 90%s” of projects fail in Jeffrey’s own experience from Re/Insurance, and 85% of all AI projects are estimated to be failing by Gartner. Jeffrey sees the key reasons around misconceptions, missing explainability and bad customer adaptation, among other things.

Too often we design unwanted Skeumorphism. Skeumorphisms are things that mimic an old process using new technology. A good example would be old-school managers printing E-Mails, reading and answering them by dictating to their secretary.

To prevent this kind of thing, Jeffrey highlights a few points to assess the readiness of a project: Yes, Model performance matters (at multiple time horizons), but it’s not everything. Re-engineered end-to-end process time, robustness and diagnosability as well as data reliability & cost (e.g. licenses, guarantees) should all be considered when designing and evaluating a new ML solution.

Takeaway: Always assess the end-2-end process, not just the machine learning (“Kaggle-Part”)

Explainability and end-user-acceptance of ML is key

• Marinka Zitnik from Harvard presented her work on predicting links between drugs and diseases using graph ML. Her key: They let domain experts look at data that is responsible for the prediction using GNNExplainer.

• Causal inference (in my view the holy grail of explainability in a way) was brought up by Jeffrey Bohn. People keep quoting Judea Pearl here. What is missing for me is the crucial step from theory to practice: Where are the relevant Python frameworks? Where is the guide for the analyst or researcher on the ground? Where are the industry use cases?

Takeaway: ML often cuts accross silos and requires corresponding collaboration and eductation!

PS: I’ll do a series of deep dives on some of these as well as other topics from the conference in the next few days. Stay tuned!