Matching animal photos with embeddings: matches and deduplication in “Find a Friend”

For “Find a Friend,” we built embedding-based photo matching: an animal photo is turned into a vector, and the system then searches for nearest neighbors and uses that signal in ranking.

The problem looked simple only on the surface. A user sees a listing for a found cat and hopes it is their missing pet. If there are two orange cats from the same neighborhood, the system should not confidently suggest a match just because the pictures are similar in color and setting. And conversely, if one found cat was photographed by several people, those listings should not live as independent entities and dilute the results.

Before: similar photos produced too much noise

The project already has inherent ambiguity: animals of the same breed, same color, and from the same area can look almost identical in poor photos. An orange cat on asphalt, an orange cat near an entrance, an orange cat in a carrier — to a person these are different situations, but for automated search based on a single feature it is easy to get an overly confident match.

Another pain point is duplicates. One found cat can appear in the service multiple times: several people photographed it, someone posted a closer shot, someone else a side view. Without visual similarity, it is hard to merge such listings: the text may differ, the description may be short, and the coordinates may be close but not identical.

So we decided to use photos as a separate signal, but not as the final verdict.

Now: every photo is searched as a vector

We added a pipeline where each photo is turned into an embedding — a numeric vector that stores the image’s visual features. During search, the system takes the vector for a new photo and looks for nearest neighbors among already known photos.

This gave us two useful scenarios.

The first is ranking potential matches. If a user adds a missing cat, the system can rank found animals with visually similar photos higher. But this is ranking, not certainty: two orange cats in the same neighborhood do not automatically become “found.” That reduces false hope, especially on repetitive-looking photos.

The second is deduplication of found listings. If one cat was photographed by several people, visual similarity helps show that this may be the same object across different listings. Such candidates can be grouped and presented more cleanly, without forcing the user to go through identical cards one by one.

A practical note: in early discussions, we wanted visual matching to be the main signal. Then we quickly ran into orange cats. They have too much in common at the image level, and the cost of a mistake for the user is unpleasant: a person starts hoping too early. So embeddings remained a strong factor, but not the only one in scoring.

Time also affects relevance

For matches, we added temporal decay: match relevance decreases over time. The logic is practical. A photo can be very similar, but if the listing is aging, its weight in the results should drop.

This does not cancel visual proximity. An older listing can still appear among the candidates if the photo is truly close to the query. But in scoring, time acts as a separate coefficient so that fresh and visually similar listings do not get buried under older cards.

Temporal decay came with a separate trade-off: if you suppress old listings too early, you can lose an important lead; if you suppress them too little, the results start clinging to old matches. That is why decay was moved into scoring, where it can be tuned independently of visual similarity.

What changed

Before: the system sees a set of listings where two similar orange cats from the same neighborhood and several cards for one found cat all look like equally likely options.

After: every photo has an embedding, nearest neighbors are found from it, and visual similarity then participates in ranking and deduplication. Listing age is accounted for through temporal decay, so match relevance gradually decreases.

For the user, this appears without unnecessary technical exposition: similar animals move higher, duplicates of found listings are easier to group, and older matches do not pull the entire result set toward themselves.