A Process Framework for Evaluating GenAI Adoption and Use in Software Engineering

Generative Artificial Intelligence (GenAI) is increasingly adopted in software development for faster ideations, code assistance, and automation, from early exploration to operational deployment. However, organizations are uncertain about how to evaluate product quality and quality-in-use when using GenAI across these stages. Standards, such as ISO/IEC 25059, distinguish between software product quality (e.g., usability) and quality-in-use (e.g., satisfaction). However, there is a lack of empirical studies on applicability, usefulness, and evaluation methods in industrial practice. These insights are important for the trend of growing GenAI adoption. In this study, we investigate GenAI adoption and use, in particular, how quality is evaluated for adoption decisions, monitoring, and assessment of product quality. We conducted 19 semi-structured interviews, supported by archival data analysis for triangulation (15 documents and 184 web pages), in two companies belonging to a large organization, to understand adoption steps, quality concerns, evaluation practices, and role responsibilities. Our findings describe a three-phase adoption process – Ideation, Development, and Operation – highlighting where and how quality evaluations occur, the criteria used, and the distribution of responsibilities. We synthesize these insights into an evidence-based quality evaluation framework. Through a GenAI for Software Engineering (AI4SE) use case, we conclude that our framework demonstrates its applicability in industrial practice for guiding structured quality evaluation. Furthermore, the results highlight a need for a coordinating role – the GenAI Quality Lead – to improve accountability and traceability in AI-based systems. This contributes to Software Engineering for AI (SE4AI) by clarifying responsibilities when developing GenAI-based systems.

Download paper here