Modern web applications feature dynamic and visually complex interfaces that challenge the reliability and maintainability of traditional automated testing frameworks. These conventional approaches often rely on fixed document object model locators, such as XPath or CSS selectors, which are highly sensitive to minor user interface changes. As a result, tests frequently break, leading to increased maintenance costs and reduced accessibility for non-technical users. To address these challenges, we propose a unified and lightweight framework that combines a domain-specific restricted natural language for intuitive, high-level test specification, with a vision-language module capable of real-time user interface element detection directly from live screenshots. This eliminates the brittleness of fixed locators and enables robust test execution even under frequent user interface changes. To validate our approach, we implemented and deployed a fully functional prototype, demonstrating that th... Read more

Modern web applications feature dynamic and visually complex interfaces that challenge the reliability and maintainability of traditional automated testing frameworks. These conventional approaches often rely on fixed document object model locators, such as XPath or CSS selectors, which are highly sensitive to minor user interface changes. As a result, tests frequently break, leading to increased maintenance costs and reduced accessibility for non-technical users. To address these challenges, we propose a unified and lightweight framework that combines a domain-specific restricted natural language for intuitive, high-level test specification, with a vision-language module capable of real-time user interface element detection directly from live screenshots. This eliminates the brittleness of fixed locators and enables robust test execution even under frequent user interface changes. To validate our approach, we implemented and deployed a fully functional prototype, demonstrating that the system can operate efficiently on consumer-grade hardware without reliance on cloud services offering a highly cost-effective solution suitable for enterprise use. A key enabler of our framework is the streamlined dataset construction process: we introduce a hybrid data acquisition strategy that combines automated annotation from large-scale web crawling with a scalable human-in-the-loop refinement pipeline, significantly reducing manual effort while maintaining high data quality. Notably, the framework is domain-agnostic and can be readily adapted to other disciplines requiring efficient vision-language fine-tuning, broadening its potential impact beyond web testing. Experimental evaluations show substantial improvements in test robustness, maintenance overhead, and execution efficiency, establishing our framework as a practical and scalable solution for next-generation automated UI testing. Read less

Automated testing is essential for maintaining software quality and robustness in modern web development, where applications are increasingly dynamic and complex. However, traditional script-based testing tools like Selenium pose significant challenges for non- technical users due to their reliance on fragile web element locators such as XPATH and CSS selectors. These locators are prone to failure when application structures change, requiring frequent updates and substantial technical expertise. To address these challenges, this paper introduces a Domain-Specific Language (DSL) designed to simplify web testing by enabling non-technical users to define tests using natural, declarative syntax. The system leverages Large Vision-Language Models (LVLMs) to dynamically locate web elements based on user descriptions, enhancing the resilience of test scripts and reducing the need for manual intervention. By abstracting over traditional web testing technologies and integrating modern AI techniq... Read more

Automated testing is essential for maintaining software quality and robustness in modern web development, where applications are increasingly dynamic and complex. However, traditional script-based testing tools like Selenium pose significant challenges for non- technical users due to their reliance on fragile web element locators such as XPATH and CSS selectors. These locators are prone to failure when application structures change, requiring frequent updates and substantial technical expertise. To address these challenges, this paper introduces a Domain-Specific Language (DSL) designed to simplify web testing by enabling non-technical users to define tests using natural, declarative syntax. The system leverages Large Vision-Language Models (LVLMs) to dynamically locate web elements based on user descriptions, enhancing the resilience of test scripts and reducing the need for manual intervention. By abstracting over traditional web testing technologies and integrating modern AI techniques, this approach aims to make automated testing more accessible and robust, supporting both technical and non-technical stakeholders. Read less