Design sharing sites provide UI designers with a platform to share their works and also an opportunity to get inspiration from others' designs. To facilitate management and search of millions of UI design images, many design sharing sites adopt collaborative tagging systems by distributing the work of categorization to the community.
However, designers often do not know how to properly tag one design image with compact textual description, resulting in unclear, incomplete, and inconsistent tags for uploaded examples which impede retrieval, according to our empirical study and interview with four professional designers.
Based on the deep neural network, we introduce a novel approach for encoding both the visual and textual information to recover the missing tags for existing UI examples so that they can be more easily found by text queries. We achieve 82.72% accuracy in the tag prediction. Through a simulation test of 5 queries, our system on average returns hundreds of times more results than the default Dribbble search, leading to better relatedness, diversity and satisfaction.
To demonstrate our task, we first show some examples to illustrate what is the fundamental limitations in existing design sharing platforms.
Figure 1 shows an example design from the design sharing website Dribbble of which tags illustrate the two problems with tagging-based search.
Fig. 1. An example design from the design sharing website http://www.dribbble.com/ of which tags illustrate the two problems with tagging-based search.
First, users may use different words to describe the same UI design based on their own background knowledge. The increase in the use of incoherent tags may hinder the content searching or navigation. For example, when designers want to search for UI designs with "user interface", this design tagged with "ui" will be omitted.
Second, users may extract different topics from the same design according to their own understanding, hence missing some closely related tags associated with the GUI design images. When tagging the UI design, users tend to provide only a few keywords that describe the most obvious visual or semantic contents, while content in other aspects may be simply omitted. For example, although this design is related to the "food" category, this label is missing from the tags.
Some example UIs with complete or incomplete tags from these designers can be seen in Figure 2.
Fig. 2. Examples of UI examples with complete&incomplete tags.
In this work, we select Dribbble as our study subject. Since its foundation in 2009, Dribbble has become one of the largest networking platforms for designers to share their own designs and get inspirations from others'. Only professional designers can be invited to upload their design works for sharing with the community, leading to the high-quality design within the site.
We build a web crawler to collect designs and associated metadata from Dribbble from December 27, 2018 to March 19, 2019 with a collection of 226,000 graphical design.
We show some examples in our dataset, top is the designs and bottom is its metadata.
Fig. 3. Example and Metadata of our Dribbble dataset crawled from December 27, 2018 to March 19, 2019
The full dataset can be downloaded via Dataset || Metadata
We adopt the Association Rule Mining and Community Detection for visualizing the landscape of UI tags. Figure 4 shows the UI-related tag associative graph.
Fig. 4. The UI-related tag associative graph from December 27, 2018 to March 19, 2019
We adopted a consensus-driven, iterative approach to combine the observed tag landscape with existing expert knowledge documented inbooks and websites such as Mobile Design Pattern Gallery and Google’s Material Design.
Figure 5 shows the categorization of some most frequent UI-related tags. For example, the APP FUNCTIONALITY category contains "MUSIC", "FOOD & DRINK", "GAME", and the subcategory "FOOD & DRINK" contains UI design tagged with "Food", "Restaurant", "Drink", etc.
Fig. 5. The categorization of some most frequent UI-related tags.
We adopt a semi-automatic method (1) train a word embedding to extract semantically-related words like "minimal" and "minimalistic" (2) define a set of rules to discriminate the abbreviations (3) manually check the vocabulary
Figure 6 shows the 40 most frequent UI related tags with their abbreviations and synonyms and in brackets indicate the number of occurence.
Fig. 6. The 40 most frequent UI related tags with their abbreviations and synonyms and in brackets indicate the number of occurence.
The full UI category can be viewed Here
Figure 7 shows the overview of our approach. We first collect all existing UI design with specific tags identified in our empirical study, and then develop a binary tag prediction model (predicting the image is or isn't belonging to the tag) by combining a CNN model for capturing visual UI information and a fully-connected neural network for capturing textual information of existing tags. Additionally, to understand how our ensemble model make its decisions through the visual information, we apply a visualization technique (Saliency Maps) for understanding which part of the figure and which keyword leading to the final prediction.
Fig. 7. The architecture of our tag prediction model.
Figure 8 shows the statistics of our dataset for each tag. The dataset contains 50% positive and 50% negative samples.
Fig. 8. The number of instances per tag in the proposed dataset.
Please follow the Readme.md instruction in RecoverTags folder.
Note that as the splitting ratio may influence the final results, we experiment four splitting ratio (training : validation : tesing), 50%:25%:25%, 60%:20%:20%, 70%:15%:15% and 80%:10%:10% for each model respectively.
We further set up several basic machine-learning baselines including the feature extraction (e.g., color histogram) with machine-learning classifiers (e.g., decision tree, SVM). We further set up different settings of data splitting.
Results show that the improvement of our model is significant in all comparisons and in all data splitting.
Fig. 9. Tag classification accuracy for four dataset splitting ratio in different methods.
The detailed results can be viewed here.
Fig. 10. Tag classification accuracy in four splitting ratio.
Figure 11 shows some predicted additional tags for example UI designs by our model.
Fig. 11. The predicted tags by our model for complementing the original tags.
Figure 12 shows the visualization of salient visual and textual features in our model leading to the final predictions.
Fig. 12. Visualization of the salient features in our model leading to the final predictions.
Some common causes for tag augmentation failure.
Fig. 13. Examples of the three kinds of prediction errors.
We conduct a pilot user study to evaluate the usefulness of the predicted addition tags for boot-strapping the UI design retrieval. Figure 14 and Figure 15 provides initial evidence of the usefulness of our method for enhancing the performance of tagging-based search. For more detail of this user study, please see the website.
Fig. 14. The random queries for searching UI designs.
Fig. 15. The comparison of the experiment and control groups. ∗ denotes 𝑝<0.01, ∗∗ denotes 𝑝<0.05.
