Machine learning models have been trained to predict semantic information about user interfaces (UIs) to make applications more accessible, easier to test, and easier to automate. Currently, most models rely on data sets collected and labeled by human workers, a process that is expensive and surprisingly error-prone for certain tasks. For example, it is possible to guess whether a UI element is “touchable” from a screenshot (i.e. based on visual signifiers) or from potentially untrustworthy metadata (e.g. a view hierarchy). , but one way to know for sure is programming. Tap the UI element and see the effects. We built Never-ending UI Learner, an app tracker that automatically installs real apps from a mobile app store and crawls them to discover new and challenging training examples to learn from. Never-ending UI Learner has crawled for over 5,000 device-hours, performing over half a million actions across 6,000 apps to train three computer vision models for i) tap-ability prediction, ii) drag-ability prediction, and iii) screen similarity.
