Helping companies create feasible AI Strategies
Aries-The making of an NLQ
Background
Electronic Arts data is siloed across the business, not to mention across studios. Many analysts and non-analysts spend their time buried under a mountain of spreadsheets, which prevents them from working on the most valuable part of their jobs—answering complicated questions. When it comes to working with data, things get more obtuse. Extracting, modeling, and analyzing data requires specialized skills, which can be time-consuming and complex.
Challenge
To analyze data, most businesses rely on SQL (Structured Query Language) to extract data from databases. While SQL is a powerful tool, it has its drawbacks. For one, learning SQL takes time for non-analysts, so the brunt of the work goes to analysts who become the bottleneck. How might we lower the bar for analyzing data and free up time for analysts?
Solution
Our team decided to explore the potential of NLQ-to-SQL systems. We utilized the Jobs to be Done framework in combination with LLMs to create business value while addressing common challenges associated with LLMs, such as hallucinations (where the system might generate inaccurate or misleading results).
We developed an experience that teaches users how to write better prompts for NLQ, which is different than if they were prompting strictly an LLM. By focusing on these challenges and leveraging data pipeline as well as domain knowledge, we created a more intuitive and efficient way for our users to interact with data, ultimately driving better decision-making across the organization.
Outcomes
The implementation of NLQ-to-SQL systems within Pricing (our first cohort) has transformed the way analysts and non-analysts work with data. Business users can now make informed decisions more quickly and with greater confidence without specialized technical knowledge. Meanwhile, our analysts can focus on more strategic initiatives, driving further innovation and value for the company.
This case study highlights the importance of simplifying data access and the impact that innovative solutions like NLQ systems can have on business operations. By bridging the gap between technical complexity and user-friendly interfaces, we have empowered our team to work smarter and more efficiently.
Enabling better anomaly detection
Background
Global Audit is investigating how to use Corporate Credit Card data to improve our audit capabilities, strengthen risk assessment, and enhance fraud prevention. By analyzing different card types (Travel & Entertainment, Purchasing Cards, Managed Cards, and Ghost Cards), we can better manage risk across the organization and gain valuable insights for specific audit initiatives.
Challenge
Discover use cases that add real value and help the team scale up the employee population that they test.
Solution
We developed a self-service analytics tool that leverages an LLM to detect data anomalies, plus provide deep domain knowledge to junior and less experienced auditors.
Outcomes
We were able to launch to users outside of the testing program two weeks before our deadline.
Microsoft GitHub Copilot-Adoption Strategy
Background
GitHub Copilot is a generative AI code assistant that lives in your IDE and offers real-time AI-based suggestions.
As player expectations and demands continue to evolve, teams with the resources they need to meet those expectations. The vision for AI is a human-first — one where AI is an accelerant and amplifier of human creativity.
Challenge
The product team wanted to create meaningful touch points with users to discover how Copilot is delivering on its value propositions of making coding more enjoyable, saving time, helping developers write better code, and producing a better outcome.
Solution
We developed a pulse survey administered to 500 developers using Axiomatic every quarter to track metrics such as NPS and satisfaction. Additionally, we tested survey questions and asked follow-up questions to identify areas of opportunities for training, especially prompt engineering.
Outcomes
Our research indicated opportunity areas for training and marketing outside of groups. We were also able to establish benchmarks for key metrics and realize that over 45 percent of participants saw time savings that exceeded our goal of 10 percent.
We were also able to influence Product Leadership to focus on some key metrics that were more relevant to their business goals.
Neurodiversity Experience Heuristics GPT
Background
Heuristics for designing an experience have existed for years for neurotypical people, but there wasn’t anything consistent and robust for people with a condition that falls under the neurodiversity umbrella. Plus, recruitment for neurodiverse people can be tricky because of legal issues.
Challenge
Develop a way for designers to understand if their designs would cause friction for neurodiverse people that is low cost and fast.
Solution
I created 10 Neurodiversity Experience Heuristics + a GPT that allows designers to upload designs and receive an evaluation score plus recommendations that will improve the usability of digital products. George Washington University is helping evaluate the GPT and we have used it at Electronic Arts to determine if our wireframes for a new portal would impact the 33% of people who self-selected as neurodiverse in segmentation questionnaires.