Sketchy Collections is an AI-driven image search tool that makes it possible to explore select museum collections by drawing a picture. The tool analyses the sketch and images of museum objects, predicting text descriptions based on each museum's own terminology, and compares those to find the top match. It makes exploring both cultural heritage and AI technology more intuitive and fun!

The tool uses an AI model called CLIP to turn the images into long lists of terms with probabilities. In the app, you can see the five top terms for both the sketch and the matching images. Each museum has its own vocabulary, some of which are tags made for the public or terms used for cataloguing the museum objects.

When comparing the images and tags across different museums, we can see how the AI model sees images and how it describes them given different languages of museums. All of this is shaped by different factors: What was the model trained on initially? What objects are included in the museums? How do museums talk about their collections?

by Polo Sologub – polosologub.com