This transcript was autogenerated. To make changes, submit a PR.
Hello, I'm Karl Weinmeister with Google's developer app SC team.
Today we're going to talk about ten things that can go wrong with ML projects.
Let's get started. So machine learning practitioners are
solving important problems every day, running into a
set of unique challenges. Today we're going to talk about the best practices
and tools that can help address them. The issues we're
going to discuss fall into four categories, building a model,
model accuracy, transparency and fairness,
and mlops. So let's start with our first problem.
It's all about these business problem you're trying to solve.
So many organizations are transforming and
really changing how they do things with machine learning, but we see
other companies that are really struggling to get value out of those machine learning
projects. So it's key that your machine learning model is
aligned with your goals. It's also important that
when you're figuring out if you're doing well, you have a baseline
where you can evaluate how your model is doing. You need to know
how existing approaches are working, whether they're manual,
whether they're implemented by traditional software development
systems, or in a previous form of machine learning.
You need to know what your starting point is, to know how much you're improving
based on that. So it's key to know what that baseline is and the goals.
So let's talk a little more about that. So I recommend watching
this video coming from DeepMind, Google's research
organization, where there's a great video on product management
for AI and the speaker talks about a lot of topics.
It's about a 30 minutes video. Some of these key takeaways that I took from
that were staying focused on your tools
as a project, but being flexible on the tactics.
It's inevitable that you're going to run into things that don't
work along the way. So you just need to keep adapting but not losing sight
of what your ML solution is aiming to achieve
for your users. Secondly, from a scheduling
perspective, it can be hard with machine learning because it's
often a large research and discovery. Part of the mission
is where you start things, where you're not sure exactly how things
are going to work. You might not have an answer.
You're learning a lot as you go, and that's hard to plan around. So setting
milestones that you can adapt and change
as you go is fine, but it's important to plan things
out with milestones as you go to sort of have some structure
on your project. Finally, having users involved
at each stage of your project is important. So you
may still be doing a lot of research work, but having
some early insights and lessons to share with your users to
ensure you're on track is often very valuable. So some
of these insights will help ensure that you're getting value out of your AI
projects. The next thing I want to focus on here is
that your problem needs to be a good fit for machine learning.
This isn't an exhaustive list, but here are a few things that
you might want to keep in mind when you're wondering, is this an
area where machine learning can help? The first group is
predictive analytics, and these are problems. Theyre you have historical
data and you want to look at trends that are going to happen in the
future. Whether it's looking at past,
say, transactions and trying to figure out if a new one
is fraud, looking for equipment,
if it's going to fail over time by looking at information about
heat or vibration, et cetera, being able to extract those
patterns for when the systems are going to fail, being able to fix them beforehand.
There's all kinds of situations like this. We're using historical data to predict
some new future. There's another class of problems around unstructured data.
This is where you have data that's not in the tabular
format that fits into, say, databases. This is where you
have images, text, et cetera. There's a
variety of different use cases here, but just a few examples
of things like maybe triaging emails if
there's a large load for the customer service organization, being able to
cluster and move items to the right
place. Another area could be automation, where you have
a manual process and you're trying to automatically fulfill
some step of that process, and you see a few examples of that.
Finally, personalization, where you want to understand how your
users tick and you want to be able to provide
them useful information. Next steps that help them
in your application to achieve what they're trying to do faster,
easier, et cetera. All right, so let's move on to the next
part of building a model. A huge problem can be jumping straight into
model development with these prototype. A machine learning project
is an iterative process, so you start with something simple and
you refine it as you go, and many times something
simple if it achieves your goals, is fine, but it's
always good to start small and expand from there. A quick prototype
can tell you a lot about challenges. Those could be access
to data as you start to build out that initial model. Maybe there are teams
you need to work with to request data or integration points
that you weren't aware of. Maybe you really struggle with getting
a decent model accuracy. There's all kinds
of other questions you can find out and that will help with scoping out
the length of the project and be able to understand what you're
getting into. Starting with that prototype, there are
a couple of tools that can help. So let's first start with Bigquery ML.
So Bigquery ML allows you to create machine
learning models directly from the data warehouse in Bigquery.
So you'll see a little bit in this animation where you can
write some SQL statements for deep neural network
models, logistic regression, all these different
types of models, even time series forecasting. It allows you
to get started quickly. By the way, this is a full fledged production
system as well. If you want to run your models out
of bigquery completely, a great option for that,
but allows you to move quickly, try some things out, see what your
baseline accuracy is. Similarly,
AutomL is another great option. If you want to write
a custom model, theyre you take your own data. Use AutoML to handle
the training, deployment and serving steps, and then generating a rest API.
It can wrap all that into one user
interface or SDK to enable you to do all these steps quickly
so that can serve as performance baseline.
It also can help with explainability where you can get feature importances
so you can know maybe where AutoML is focusing in theyre
there's signal. Maybe that's an area where with your data engineering, you can dig
in a little bit to extract some more features from it.
Problem number three, model training taking a long time.
Not sure if you've ever run into this. If you're working with a larger model
where it can take days or even weeks in some
cases to run, and that just really mlops down your team where
you might reach a point where you just have to wait until tomorrow
to find out what you can do next. So that slows down that process of
innovation and can really harm
the success of your project. Serverless training with Vertex
AI can be very helpful for this because it allows you to submit
training jobs across a distributed infrastructure using
gpus and other custom
chips that you'd like to use, and building models
in all different kind of frameworks. Using a container image as your
base allows you to do even things like hyperparameter tuning
where you can create multiple models and find out
the one that's working best. And this really allows you
to speed up that training time. And as
well as you see here in these screenshot, things like getting access
to logs, downloading your model
storing it in these cloud and managing on the cloud it will take care of
for you as well. Another option as far as
training quickly is cloud tpus or
tensor processing units. And those are custom
chips that are built for machine learning workloads.
They allow you to create models
very quickly at high scale, and can speed up that training
even more if that's something you want to use. All right, let's move on to
the next group of issues around model accuracy. So,
first type of issue could happen when you have an imbalanced data set.
So many machine learning tasks have many
more examples that fall into one category
than the other, right? So let's take fraud detection as an example
where fortunately, say, most of the data is benign,
theyre they are not fraudulent transaction, and you have a few that
are. It's like finding a needle in the haystack. So we
could have a trivial model that simply predicts that everything is benign
and I would have a good accuracy, but that's not going to add
any value, right? So what you
want to do is apply some techniques to make sure
that your accuracy is good across both of the
classes, even if there's not a lot of data for them. So let's
look at this next resource here,
which is a tutorial for dealing with imbalanced data.
So this is on the Tensorflow website, and it provides some of these
different techniques. So things like weighting different classes differently,
basically applying a greater penalty for
mistakes on, say, the class with
fewer examples. Other things
you could do are oversampling and undersampling, where you take
with the existing data you have, you basically can duplicate some
of those records so that there's a more balanced
number between the classes. Or conversely, under sampling,
where you remove examples from the class with more.
Finally, you could consider generating synthetic data. So there
are packages in Python, like one called smote, for example,
that can look at the distributions of your data and generate
data that's similar to what's in your training
sets. All things to try. Personally, I've had
the best success with waiting classes
to help with that issue.
And automl has some ability to help with this
as well, without even changing some of the class
weighting that we discussed previously. So when you're training
a model, there's something called an optimization objective, and this is
what you're optimizing for. And so
you can see here there are several different options.
And if you switch to something called the
area under the curve for precision recall, that is
generally better for helping with that
class with fewer examples in it, but you see there's
a spectrum of possibilities depending on if you're trying to maximize
accuracy for all of the data
or create a more balanced result,
et cetera. You can just customize using this.
It's under advanced options for model evaluation.
There is something you can do too, is where you create a model,
and then you can review the accuracy at different thresholds.
You can then review the confusion matrix.
And if you see that below, this is an example of flight delays.
And again, very good to see that most of
the time, flights are on time, although it might not always feel that way.
We can see here that there's definitely a difference in the
accuracy with flights that are delayed,
a little bit harder to pick those out with this particular model.
So you could go through this process,
look at the results for each of the different classes,
and then perhaps come back and adjust that optimization objective
if you'd like. So, model accuracy, this is a huge
one. And this really can be where projects might
not even succeed. They get completely stuck, right? So you are
creating a model and you just get to a point where you can't improve the
model accuracy anymore, and it's not good enough. It's not going to really add any
value for the business. And sometimes
it just is what it is, that it's a hard problem to solve,
and there's not that signal available in the data,
but often, with some creative thinking,
you can move past those obstacles. So let's go back to this example
of flight delays. So, on the left, I have a
research paper that talks about historically, what are
the reasons for flight delays. So I was actually looking at modeling
this problem, and I started with data around
the start and end times of the different flights,
the carrier, et cetera. And that gave
me some information. But what I did was I augmented
my data with weather information. So I took a bigquery
public data set of weather.
I had Latlong coordinates, I joined
that against the airport. So I know, okay, well, the arriving flight
is going to have hail in it, things like that. And that definitely
improved my model. No, it actually wasn't a huge
increase. And this data kind of shows that
6% of the root
cause of flight delays and cancellations due to extreme cancellations,
but the biggest one is aircraft arriving late. And this would be an
example of if you have better data,
you might be able to work on some data engineering to
look at the whole flight graph and
what flight is coming into the flight that you're trying to predict,
even multiple flights back, using that as information. So the
point here is really understanding these problem is
key. It's not just about the algorithm and the math
that you can often make a much bigger difference by
understanding these domain that you're in. So that's my number one
tip is look at improving domain expertise for the
problem you're trying to solve. Really dig in, ensure you have the right experts
on the team and as a data scientist learn
the domain the best you can and you'll probably think
of some things that are going to help you. Secondly, and sort of
related to things, including more data, that always
helps improve your accuracy, of course, and varied training data of
different types for different tables that are going
to add some diversity there for your model.
Feature engineering of course is always useful to
unlock information from some features. Maybe I'm
making this up. You have a date field and you want to extract
whether it's a weekend or a weekday. There might be
different patterns for that, all kinds of different things
that you can do with your data to ensure that
your model can take advantage of it.
So looking at some of the other things, consider removing some of the features that
are causing overfitting where you're sort of locking into
noise. I might suggest with what we talked
about earlier, start with a smaller model and then incrementally
add features back. Also try different model architectures.
This is traditionally what we do in data science. Try a
different model architecture, different number of layers,
hyperparameter tuning, ensembling your model after
you've done some of these more fundamental things. Finally,
just for a gut check, always doesn't hurt to try automl
to see what kind of performance is possible
with the input data that you have. Let's move on.
Transparency and fairness, our 6th issue,
your model doesn't serve all of your users well.
And so this issue
is a very important and complex
issue that when you look at how you're dealing with AI
development, you want to look at a responsible AI framework to help with
this. So here are a few questions to ask as you're building your
model to help. So the
first is around some of the business questions we asked before.
What is the problem you're trying to solve? Who's your user?
Moving on. Things like the risks, success factors. Now we're
starting to move on to data. How was the training data
collected, sampled, labeled. There's all kinds
of issues that can pop up into this phase of the project.
It's key to ensure that you're collecting data
and those sampling and labeling processes ensure you
have a representative sample. I'm not going to
go through all the points here, but it's worth, as you go through the training
problems, the evaluation process,
you are considering all of these important questions when
the model might have some limitations
that you want to document and you want to document how
you collected your data from end to end, so that you
can continue to assess where you're
at from a responsibility perspective and keep improving on
it. There are a couple of tools that I'd like to mention here
that can help. The what if tool allows you to
slice your data by various factors
to see why predictions happen the way that
they did. This can give you a much more
detailed understanding of your model accuracy versus a simple statistic
like it was 98% precision
or something like that. And you can also things is why
it's called the what if tool. Actually change some values and
see what happens. If I change a value slightly,
does that change my prediction? So it's almost like a debugging
tool for your model. Tensorflow model analysis is
also helpful. So what this can help do is it
can produce estimates of
performance by slice of the data. So let's take a look at
what that really means. Here is an example of Chicago
taxi trip data, where we're estimating what
the tip is going to be for a taxi trip. And you
can see some statistics here. The bar graph shows
you the number of samples at different hours of the day.
So we're seeing that 05:00 in the morning,
06:00 in the morning, much lower number of trips.
Well, that might impact the accuracy. It's an example of not
having a balanced data set, so it's actually going to slice per hour
and give you statistics on that accuracy.
So this will allow you to see by different
dimensions of your data how balanced the errors
are. So it can be a very useful tool. So we talked
about assessing the accuracy and equitability
of the model. Now let's look at how to document
it. So ML models often get distributed
as is. Here's these model, it just works. Okay?
And think about it. With software development, they always
documentation explaining each
part of the user interface, and glossary, et cetera.
Let's think about these same concept applied to ML models.
Right? So first is being able to explain
what's happening under the hood. So for a variety of different
data types, it's important to look at why
these predictions happened the way they did. And explainable
AI can tell you which features are
driving the decisions. So you see some examples
here, maybe in an image. What is it about the image that was critical to
making that determination, or in, say, in tabular data.
We see that distance was the most important factor in our
model's predictions. Today on Google Cloud, we support explainability
across multiple layers of the
platform, from automl to prediction to using
our sdks to perform explainability.
From your notebook and model cards allow you
to document your model. You can specify information
such as how you collected the data. You can
put graphs around your performance curves,
the model architecture. This is what we've done for the
object detection API.
And you can see there's a model card toolkit that allows you to generate
these model cards based on information or model, or even
attach it to your Tensorflow extended pipeline to generate one of
these automatically. All right, so our final class of problems,
ML Ops, are machine learning operations.
So what if you built a model that just
was a bad model? It was built on some training data
that had just a bunch of
data quality issues, and it somehow got into production. All kinds
of users were impacted by that. Definitely not something
that you want to have happen. Vertex pipelines can help with that by
allowing you to codify the set of steps to build a machine
learning model and providing guardrails
around deployment so you can implement steps,
or rather processes like continuous integration,
continuous deployment, and include tests along
the way for each of these different steps.
Pipelines allow you to build a custom machine learning pipeline.
Next issue, your model accuracy is drifting downward. So most
software projects you can have or should have unit
best on them, where you can evaluate whether your code is working or
not, and the unit test will tell you if it's working or not.
It's binary, it works, or it's broken. It's a little
more subtle with machine learning, where things might drift
slightly, the data distributions,
for whatever reasons, for whatever you're modeling over time,
may change due to outside conditions changing.
So how do you detect and manage that?
There are a couple of processes to consider. One is continuous evaluation,
where you're regularly sampling your model's predictions, comparing those
to ground truth, and these assessing
the accuracy. Another thought is continuous
training, where you deploy an
NML pipeline that extracts the data, trains a new model,
tests the model, of course, and then deploys it to production.
And each of these different steps can
work together to look for issues
when you've drifted away from a certain threshold and
then preventing the issue by training models
on a recurring basis or when a
certain amount of data changes. So you want to find that right sweet spot,
not create too many models with every small change that's happening,
but not let your models get too stale where their
performance starts getting impacted. Vertex model monitoring
can help around detecting these issues as
far as drift or training serving SKU.
So maybe you're starting to see where your users
are making a lot of prediction with data that's much
different from what you originally trained on. That might be a warning
signal that your model isn't quite as applicable
as it could be. So this gives you an additional layer of confidence
in your model reliability. Now,
our final issue is around model inference.
So this is when you've built a model, it's deployed and
people are using it, or they're making inferences or predictions on your
model. So it's a success. You solved the problem at
the accuracy level you were looking for. It's integrated
into widely used application. Now, how do you handle the
spiky workloads that may result? And how do you avoid
over provisioning infrastructure while preventing errors
or high latency? If you don't have enough infrastructure
set up, vertex prediction can help with that, because that can allow you
to set up an online endpoint where you can
serve your model, and it will scale automatically based on
your traffic. So you can set up, say, a minimum number of nodes,
a maximum number of nodes. It will scale those up and down based on various
utilization thresholds. It will help you with logging,
it'll provide you the option of using
some powerful GPU chips,
and so you'll be able to ensure that you're serving the
right amount of requests with vertex
prediction at an optimized cost. So that
wraps up the ten different issues. I hope that was helpful. Let's look
at a few resources. Vertex AI is
the AI platform that we discussed today that can help with several of
these problems. Codelabs are a way to dive in and
use notebooks and building
models and basically get some training. They're free resources
at Codelabs, at developers google.com, and if
you're into learning via video, like this one,
AI Adventures is one of our video series that has a lot
of different resources around using Google Cloud for
AI. And that concludes our
presentation today. So I thank you for watching.
I hope you have a great day.