Startup CTO Performance Metrics and KPIs
We're looking to set KPIs and performance metrics for startup CTO and wanted to get some recommendations. Some we are considering:
% Accuracy of estimates
Team attrition rate / eng team satisfaction
Uptime/downtime rate
Bugs over time
Critical bugs over time
The first one is really most important for us as we are often seeing delays in features getting released and not happening according to schedule. At the same time, I worry about setting this kind of KPI as it could lead to sandbagging.
Thoughts?
Trending on Indie Hackers
Post-launch lesson: traffic came, activation didn’t
I Stopped Browsing Reddit Randomly. Here's the Keyword Monitoring System That Actually Gets Me Customers.
For indie hackers: Outsource marketing or do it yourself?
You will need to set KPI's otherwise it would be difficult to improve so you are doing the right thing however it would not be on CTO instead it would be for engineering organization as a whole as that is what you want to improve. KPI's get bad reputation because people start fixating on the KPI's and sideline the human element. Goal is to improve the productivity over time, ensuring engineering is performing at acceptable rate etc.
Here are my thoughts about your existing KPI's
Accuracy of estimates: I would not use this kpi it only tells me that someone is bad in estimating it neither helps me in estimating new features nor does it help me in imrpoving it. Instead i will get the estimates from the past deliverables and use them to estimate new features. CTO would be involved in selecting the past features that were delivered that are closest to the feature i am trying to estimate. I will than keep an eye on these estimates and if they are delayed or completed too early than it would act as a conversation starter on deviation. If there is a delay than there should be some reasonable explanation else something needs to be fixed in your engineering team.
Team attrition/team satisfaction: This is a good metric ideally you don't want people to leave as hiring is costly.
uptime: This is also good metric can be easily measured with tools and directly impacts business.
Bugs over time: What will you measure here? bugs are part of software so fixating on it doesn't makes any sense it will start affecting 2. What you should measure is turn around time so that once bug does happens you are able to respond to it in timely manner and kpi can be set for that 2days/3days/1 week according to you org policy same goes for critical bugs. Critical bugs would have shorter kpi in days than normal bugs.
By the way if you are using github i have created a tool it's half baked right now but can help in getting this data and more. Its free of charge of course as i want to see what issues people face with it let me know if you are interested in testing and providing feedback.
Thanks @pdyc
Very helpful.
Though on the first bullet, wouldn't I want to know if my CTO is bad at estimates? That seems pretty important and something we would want him to improve on. We do sprint planning every month and determine what we can/cannot work on based on estimates provided by our CTO. If the estimates are not accurate then features are often getting shipped later than we anticipate, which would affect all kinds of dependent planning.
Regarding bugs over time:
you are correct about having good estimates and all your arguments related to it are valid that is slippage would lead to cascade effect on your future planning however it is not a good metric to judge CTO since estimates also depend on the team that is implementing it. So for e.g. your CTO might estimate something is going to take two days because presumably because it would take him 2 days to implement(he might have seen the problem before would be fmailiar with the stack etc.) however when it would come to actual implementation you might find that it is slipped by a week because person who is implementing it is not familiar with the stack or is familiar with the stack but seeing the problem for the first time or some other issue so you will end up firing CTO(extreme case) and bringing in new CTO and same pattern would be repeated since now new CTO has to learn existing state of project etc. and it goes on however this did not solved your problem you are still getting bad estimates and schedule slippage because there are multiple variable factors that are not getting counted so what you really need is a good way to get estimate without having human bias so that if CTO leaves tomorrow or hit by a bus it does not affects your companies ability to estimate and deliver.
You are correct about bugs also and your arguments are also practical and valid. If you keep on spending time on bugs than when would you have time for features. So what you need to check is how much time is getting spent on features vs bugs and also how many bugs are getting carried forward if rate of incoming bugs is bigger than unsolved bugs getting carried forward than theoritically you will never be able to fix the situation so you would need to either add resources or postpone the features until system is stabilized.
:-) i have been through this so many times that i can just keep on writing about it. I even wrote my own tool to get bird eye view of project because i faced the same problems that you are facing.
Also, how can we check out your Github tool? Is this only for CTOs or also for any developer? @pdyc
i will send you video demo on your email. Its for CTO/Engineering managers etc. for people supervising the developers. I don't see developers having much use other than for information.
OK, I take your point regarding estimates BUT presumably a good CTO when getting estimates would either a) have a good sense of his team's capabilities and track record and therefore wouldn't be estimating how long it would take him/her to do it but rather his team or b) would actually ask his team to estimate and would be training his team to also provide accurate estimates. If his team isn't good at providing accurate estimates then he is not coaching/training them properly. @pdyc
yes that would be correct however it requires stable environemnt(i.e. same team members no churn) and time (multiple cycles) to assess and correct the capability of engineers. so ime approx 6 months to a year to get things right.
It depends how big the Engineering team is they'll be managing. Those KPIs seems more like health checks for the Engineering team than how the CTO is performing in their role. A CTO would usually set their own team metrics. I'd be reviewing the CTO themselves on more standard leadership measures through regularly performance reviews. Things like:
etc.
Thanks @jmacgregor, that's helpful.
We are a small startup at the moment the CTO is managing 3 devs and a QA engineer.
The items you listed in the bullets are good qualitative measures, but we also want quantitative performance metrics so that everyone on the team is held accountable.
If they seem too much like health metrics, what would you change? Everyone else on the team has numeric KPIs that they are working against.
OK, got you.
For our Engineering team the important stuff is:
Uptime, bugs etc. get tracked but they're not a good measure of how the CTO is performing for us.
Super helpful, thanks