8 Metrics to measure in your Machine Learning projects to bridge the communication between Business
4 min readJun 8, 2019
Machine Learning projects are based on a considerable amount of research, infinite trial and error cycles, and never enough data. None of these adjectives has anything in common with Return of Investment, profitability, predictability, which are needed by any successful business.
Below is a list of metrics that I measure in Machine Learning projects:
- Time to market — how long does it take from the moment the user wants a specific functionality to the moment it can be used in production? As an investor or Product Owner, you’ll need to know what expectations to set to your clients when it comes to releasing times. Maybe your clients will expect it fixed by the next day, but we are fixing the way the machine learns, not an algorithm or a design. More you know about this process, more you can educate your market.
- Training time — how long does it take to train a machine learning model
- Data — how much data and of which quality does the solution need to improve.
- Minimum data — the challenge of machine learning is that you need a lot of data, but what is the minimum amount of data to have a solution that can be released to the market?
- Algorithm improvements when doing maintenance updates. Keeping track of the upgrades and stages is something that can standardize and predict your development process. It also helps to give direction and in the roadmap decision making process.
- Number of training cycles before a solution is ready to be released. After about three significant releases, you’ll start to see trends on how many learning cycles your model needs for a considerable version of a new feature.
- Performance — how long does it take to perform a specific operation? Can the technology scale? What are the scaling points? How many users can we support at a given time, and with what cost? The product price, the sales, and the entire business strategy are depending on the answer to these questions. The stakeholders will know that they’ll have to change the business approach when they reach a certain point. On the other hand, the development team will see what technical solutions they need to research to meet the business targets.
- Hardware cost vs. new technology upgrade? Everyone wants to work with the latest technology, but does it make sense of the update? Is it really what provides most of the business value? The cost of the hardware will skyrocket with the success of your machine learning solution. That’s a given. When there is a new technology, evaluate how much you need to invest in hardware on each user threshold and what is the hardware reduction provided by software development. In most of the situations, it will be cheaper to pay for the development team to upgrade the technology that buying new GPU.
When should you start tracking these metrics?
I do it from day one in the project, and when the team has gained enough maturity, to can allocate time to track metrics.
You might be interested in:
