Data Leadership in Engineering, A How To

Written By Raul Maldonado

Summary

Having a successful engineering executive group requires leading a scalable and agile organization that is not only innovative, but making data-driven decisions.

At the least, the executive group must intentionally make informed decisions that are positively impactful that aligns with short and long term business objectives.

This article details that in the following:

  • Goals & Tips

  • Strategic Focuses (the specifics)

    • Alignment

    • Impact

    • Metrics

Goals & Tips

Engineering Executives must first ensure that they have fundamentals down. This could depend on your type of business model like purchasing licenses or subscription model, but such fundamentals are

  • Ensure to have rich software quality and reliability

  • Have the ability to measure effectiveness with multiple data sources

    • (e.g. AWS Cost, Cost Center Information, CSAT, API usage)

  • Having Humility and Curiosity to meet Business Objectives

Anti-patterns to avoid

  • Missing Context

    • Metrics are not 100% true. Identifying the root cause requires both data and context, but when the prior does not exist, we rely more on the latter. However, data is a guardrail to minimize number of mistakes, and get closer to a correct devision. Context is critical to understanding issues, when they arise.

  • Backlogging technical debt

    • In some cases, data quality concerns arise in the midst of bugs and outputted logs. This results in poor data that creates uncertainty, plausibility, and slowdown in informed decision makings. When it comes to data quality concerns, it is affected by upstream engineering implementations, It’s good to crack down in these novel but impactful bugs that could affect your metrics

  • Having Individual Performance Metrics

  • Measuring Everything

  • Not accepting criticism that could impact Business Objectives

Tips

A big tip is that this will be a challenging efort to kick off with, but rewarding for not just the engineering division, but your customers as well. Make sure you have the following:

  • Leadership Buy-in

    • Get consensus and sponsorship to convince other leaders business impact to measure goals. Make sure there is a mapping to these goals, which we talk about later

  • Flexibility to unknowns: There could be factors that make decisions change or impact metrics. Be ready for such flexibility, in this faster development lifecycle that you’re committed to

  • Tooling: Make sure yo uhave the right tools purchased or built out to operate at the scale, speed, or impact that you need (e.g. building out LLMs vs buying Github enterprise co-pilot licenses)

  • Decision making

    • Sometimes metrics are not possible to develop, finalize, or even acquire. Don’t let limitations like this stiffen efforts for measuring things you need for your business objectives.

Strategic Focuses

Now, let’s go into the important details YOU are here for.

Mapping Key Results with Roadmap

Once receiving the roadmap for long term, short term, or current fiscal year, start building out your OKR -> Metric framework with a data strategy, like Metric trees. This mapping would help with building effective business strategy to meet your business goals.

In this process, ensure that pillars/centers of excellence, sub group initiative maps directly map to your business objective. This could look like revenue growth, customer retention & churn, or just optimization. This requires repeated efforts of building alignment of communication for such goals.

Encourage Agile & Decisions

This sounds intuitive, but organizations must be willing to accept agile methodology and also making explicit, nuanced, and detailed decisions that continue the metric development process. If this does not pan out, then mapping key results with an engineering roadmap will be successful for the timeline of your organization.

This indecision or rate of making decisions can be shown in examples like saying “I want XYZ”…but it being 10% successful to build towards that goal. It could also like “I have XYZ”…but who will own it?

Business Impact

Decide on what your business impact will be. This could look like one of the following, with respect to business growth:

  • Revenue

    • How do engineering initiatives and products drive growth?

  • Cost Management

    • Are costs optimizations implemented to increase Profitability?

  • Customer Satisfaction

    • Is your engineering group ensuring they are bettering customer experience received from Sales or Customer Success feedback? What about offerings?

  • Optimization

    • Are efficiency gains added to improve architectural improvements to reduce long term issues, costs, or rate of development?

Customer and Developer Satisfaction

Investment Straetegy: Ballancing Innovation and Sustainabiklity

What do do?

Engineering Leadership Dashboard

Metrics

Cost Optimization

  • Cost Cost: Example unit economics, AWS cost, etc

  • Cost Per feature: What is the cost for an added effort?

  • Return on Investment (ROI): Measuring total ratio of revenue to that of infrastructure investments or additions

Customer Satisfaction

  • Customer Reported Issues/Bugs

  • Customer Satisfaction Scores (CSAT)

  • Stickiness Ratio: DAU / MAU

    • Daily Active Users / Monthly Active Users

  • Retention Rate

Developer Productivity

Using some metrics associated to DevOps Research and Assessment (DORA) to AI

  • Total Deployments

  • PR Review Times

  • Merge Velocity

  • Total Backlogged Feature Requests (e.g. Jira tickets)

  • Surveys of developers

  • Co-pilot/AI adoption rate

Revenue

  • ARR Annual Recurring Revenue

Reliability

  • Post-release defects

  • Customer Reported Issues/Bugs

  • Resolution time

Conclusion

Building a successful engineering executive group hinges on the ability to lead a scalable, agile, and innovative organization that prioritizes data-driven decision-making.

By focusing on strategic goals, aligning initiatives with business objectives, and continuously measuring impact through relevant metrics, engineering leaders can drive significant business growth and customer satisfaction. These principles will not only enhance the engineering division's performance but also deliver substantial value to customers and stakeholders alike.

Raul Maldonado
Next

Data User Experience: First Few Columns Can Make or Break you