Many apps have basic accessibility issues such as missing labels or low contrast. Automated tools can help app developers detect basic issues, but they can be laborious to run or require writing dedicated tests. In this work, we develop a system for generating mobile app accessibility reports through a collaborative process with accessibility stakeholders at Apple. Our method combines various data collection methods (e.g., app crawling, manual recording) with an existing accessibility scanner. Many of these scanners rely on scanning a single screen, and a key problem in app-wide accessibility reports is effectively removing duplicate issues and summarizing those collected across an app. To this end, we develop a screen clustering model with 96.9% accuracy (F1 score 88.8%) and UI element matching heuristics with 97% accuracy (F1 score 98.2%). We combined these technologies into a system to report and summarize unique issues in an application, and evaluated our system with 18 accessibility-focused engineers and testers. The system can enhance your existing accessibility testing toolset and address key limitations in current accessibility scanning tools.
Figure 1: The prototype interactive HTML report generated by our reporting system. The interface has three main areas: a) A carousel displaying all the screens captured in the report, b) A menu to toggle between the currently selected “Summary” view and screen-by-screen views represented by thumbnail screenshots, and c) a table of summary results of the issues found, grouped by category. The report provides actions that users can take, such as ignoring, viewing suggestions, reporting bugs, and enlarging the screenshot.
Figure 2: Our approach to accessibility reporting collects screenshots and accessibility audits via an app crawler or manual recording tool. From the collected data, a report generator uses a screen grouping model and UI element deduplication heuristics to generate an interactive report.
Figure 3: Xcode Accessibility Inspector in Single Screen (SS) mode with scanner results for an iPhone. The inspector has four main areas: (A) a table of detected issues for the current screen that a user requests by clicking the “Run Audit” button, (B) remediation suggestions that appear when a user clicks the “?” icon on a selected row, (C) a preview window with a screenshot of the affected UI element, and (D) a device (either in a simulator or live) or application window. Our study evaluates two modes: SS, which provides AD features, and MS, which provides AE features. (E) adds a history of scanned screens and headers to summarize result counts by category.
