Redesigning Reverse Image Search
How do you describe an exotic fruit or the face of a stranger to a search engine? Images often contain visual information that is beyond words. One way of querying images is to use reverse image search, which enables the user to input an image as a query, rather than describing what they would like. This content-based image retrieval (CBIR) method removes the user's need to provide appropriate keywords and can potentially improve search results in situations when textual descriptions fall short… but it is an under utilized search technique. We set out to build our own model and to create a user friendly search engine.
Traditional text-based image retrieval has many limitations and image search is a developing technology that is underutilized, partially because users often don’t know it’s an option or find it hard to use the feature.
In this project, we build and evaluate a reverse image search engine that takes an image as query and returns a ranked list of images sharing similar visual properties and class labels— in a user friendly way.
In this team of 3, I played the role of user researcher and designer while my team mates built the reverse image search model.
Lit Review \\ Qualitative Sessions \\ Competitive Analysis \\ Key Insights
For this project, we utilize a combination of qualitative sessions and market data from the Google Play app store in order to measure efficiency, effectiveness, and sentiment for existing reverse image search tools. The primary goal is to create a user-friendly platform. Using information from the qualitative sessions and competitive analysis, we address the following research questions:
Which reverse image search tools have high effectiveness and efficiency and positive sentiment for users?
Which device are these tools designed for?
Do users prefer a general search tool or a topic specific tool?
What are problems users would like to solve with reverse image search tools?
What are features of successful reverse image searches?
We reviewed papers in the fields of 1) user experience and usability frameworks, 2) content based information retrieval (CBIR), and 3) image classification machine learning models. Overall, our research revealed that there was almost no user experience research for this form of search, though the models at our disposal are highly sophisticated. Essentially: the technology is here but few users have caught on!
At the start of this project, we established a desire to improve on both usability and UX for reverse image search. Combining these two aspects of user testing, we create a framework of three variables that need to be explored. These include effectiveness, efficiency, and sentiment. Below are short definitions of each term used in the context of this project.
Effectiveness: this measures the the user's ability to achieve a task.
Efficiency: this measures the amount of resources needed to complete a task. Examples of resources may be time, trials, and cognitive effort.
Sentiment: this is typically a self- reported response reflecting the emotional reaction to performing the task.
I conducted five qualitative sessions with voluntary participants in the Austin Public Library and three at the Pompeu Fabra University in Barcelona (I was there for a conference). Sessions lasted between twenty minutes to an hour and were semi-structured. Each task was timed and I took notes and recorded audio. Participants talked through four tasks and then were asked open ended questions and Likert scale questions about their perception of the task.
Based on initial qualitative session results, mobile apps had higher usability and UX outcomes than websites on desktop or mobile. To further explore mobile apps, I scraped 100 apps from the Google Play app store that was returned by the query "reverse image search." This was achieved via a data scraper created by Github user Facundo Olano. This scraper included variables such as app ratings, number of installations, descriptions, and images of the app. By analyzing most popular and least popular apps and comparing aspects of those apps, we informed our own design choices.
Based on our competitive analysis and qualitative sessions, we extracted 4 key principles:
Specific topic, specific tasks.
The worst rated reverse image search tools were too broad in nature, leading users to get lost in the myriad of options. Several of the most successful apps were Pinterest and Picture This, in which users have a clear goal in mind (and hardly realize they are using a reverse image search). In the qualitative sessions, users also had high efficiency and effectiveness for tasks when the app was topically focused and had an interface that clearly directed users towards a certain task. Sentiment was also more positive because the user interface design met user expectations and because users felt more confident.
"User history," "my observations," "my garden," "add to collection," "download to phone" --- this feature had many names and forms. All successful apps had some form of knowledge management, which increases user satisfaction. An additional advantage of this feature is that it provides relevance feedback that can be integrated into the ranking of results/training model.
Quality in, quality out.
To increase quality of inputted images coming from the user camera, it is necessary to ensure that the main subject is the focus of the picture. Strategies include cropping pictures to relevant subjects or guiding users as they take pictures (i.e. “put the main subject in the dotted frame” and providing user friendly tips.)
Results should be more than similar pictures.
Successful apps added contextualizing information to images. For example, when a user searches for a picture of a plant, the returned similar images include names of plants, a map of where they were found by crowd sourcing data, and plant care tips.
Pitch \\ Design \\ Performance
Using PIC Search, users can instantly identify landmarks and find out more information, including similar images and historical information. Users can do this on the go, drawing from their albums or with their phone camera— and they can rest assured that their results will be saved for future reference.
The code used for the back-end portion of our project are available at https://github.com/SanatSharma/RevSearch and an interactive mock-up of our app is available at https://invis.io/QFP09FZURSX. The qualitative session guidelines and post-task survey is available at shorturl.at/hlEK4.