9 Reasons Metrics Can Be Controversial
Metrics can be a surprisingly controversial topic for companies, especially when you consider the benefits. Geckoboard's survey reveals that 50% of companies tracking real-time metrics achieve all their goals within a year, compared to only 24% of those not using real-time tracking. Tracking metrics isn't solely about company growth and revenue; it can also boost employee satisfaction by aligning priorities and providing a real-time view of goal attainment, helping employees see the impact of their contributions.
Subjectivity: User experience is inherently subjective, as it involves individuals' feelings, preferences, and perceptions. These feelings can vary across demographics such as gender, age, race and income. No one wants to limit their product’s success by focusing on one group of the population. In addition, it’s important to ask demographic questions when surveying because research shows that some demographic groups such as Asian people tend to avoid extreme responses on “agree-disagree” scales.
If we consider demographics when interpreting data, we can get closer to understanding the whole picture and how it relates to the customer segment.
Lack of Consensus: Many organizations lack alignment among their teams when it comes to determining the most suitable metrics for assessing a positive user experience. For instance, the customer experience team may prioritize metrics like Net Promoter Score (NPS) and Satisfaction, while the user experience team might emphasize usability benchmarks and System Usability Scores (SUS). A helpful approach to bridge this gap is by employing a regression equation to convert NPS into SUS, especially for specific user journeys that intersect with the products under the research team's evaluation. There are other ways to convert other metrics if both parties don’t align on metrics.
Context Dependency: Have you ever received a survey asking if you would recommend a product to a friend or family member within seconds of visiting a website for the first time? The timing of data collection is just as important as what you choose to track. Often, business units may not coordinate their in-product surveys effectively, resulting in surveys popping up across various products during a single visit, which can be a less-than-ideal user experience. It is crucial to establish a unified surveying strategy across your product to ensure not only valuable feedback but also a positive customer experience.
Trade-offs: Certain metrics, such as efficiency, may sometimes clash with others, like satisfaction. There exists an old UX adege that suggests users should find what they're seeking on websites within three clicks. However, this notion has been debunked due to a lack of supporting data. Rather than dismissing it entirely, it is essential to assess the efficiency and findability of content for users, as these factors can be correlated with lower satisfaction scores
Incomplete Picture: Relying solely on metrics might not offer a complete grasp of the user experience. Qualitative research and user feedback serve as crucial complements to quantitative data. However, it's worth noting that these methods can be both time-consuming and costly. Therefore, it's advisable to leverage multiple data sources when making significant decisions, especially those that could entail substantial engineering investments. In my view, a guiding principle is to refrain from making major decisions unless backed by at least two sources of data.
Misinterpretation: Metrics are susceptible to misinterpretation or improper utilization. For instance, an exclusive focus on enhancing a solitary metric, such as the conversion rate, could inadvertently result in the neglect of other vital facets of the user experience. An illustrative example of this scenario involves a high conversion rate coexisting with a low user retention rate. Concentrating solely on the conversion metric may paint an inaccurate picture of your site or product's overall performance.
Unintended Consequences: Excessive focus on specific metrics can encourage behavior that deviates from the overarching goals of enhancing the user experience. For instance, urging users to submit more reviews may inadvertently result in the generation of spammy or insincere feedback, as seen in the case of Google reviews. Google requests customers to rate a business immediately after visiting, potentially introducing a recency bias, where users might provide positive feedback to avoid causing offense to the business.
Lack of Standardization: Customer Experience (CX) and User Experience (UX) are distinct fields with a shared objective: comprehending customers and enhancing product quality. However, CX encompasses a wider spectrum, encompassing all brand touchpoints. When metrics are gathered differently by the CX and UX teams, discrepancies can arise, particularly in areas such as the phrasing of in-product surveys. This disparity poses challenges when attempting to compare results across various studies or organizations.
Resistance to Change: When it comes to introducing new metrics or altering existing ones, organizations often encounter resistance. Team members tend to be attached to familiar metrics and may hesitate to embrace novel approaches. For instance, as per Lumoa, 75% of companies monitor Net Promoter Score (NPS), while 50% track Customer Satisfaction (CSAT), and 14% monitor Customer Effort Score (CES). Additionally, individuals transitioning between organizations may have a preference for specific metrics they wish to monitor.
In summary, achieving alignment across the organization regarding what to track, how to track it, and how to socialize the data across teams is essential to mitigate controversies surrounding metrics. Research and Customer Experience professionals must navigate these challenges to ensure that metrics serve their intended purpose of enhancing the user experience.