The pitfalls of CV-based lost-pet search in the “Find a Friend” project

“What if the dog is found dirty and thinner — will your model say it’s a different dog?”

That question captures the problem in the “Find a Friend” project well. A user uploads a photo of a lost or found pet, the system turns the image into a vector, and then searches for similar profiles using cosine similarity and ANN. After that comes a push notification if the match falls within the configured radius.

That sounds neat until you start looking at real photos. The same dog in a home photo, after a week outside, and in the rain — those are three different visual objects for a naive model. Below are the pitfalls that are easy to run into with a task like this.

Pitfall 1. Train the model as a breed classifier

The first obvious idea is to identify the breed, coat color, size, and then search among similar ones. That works for a breed catalog. It does not work well for finding a specific animal.

A lost dog rarely shows up in the system as “Labrador, yellow, no special features.” It may have a white spot on its chest, a torn ear, an asymmetrical mask on its face. For a cat, it may be stripes on the paws, the shape of a spot near the nose, or a rare coat color. These details are often more important than the breed.

So we did not turn this into a classification problem. The model was supposed not to answer “what class is this,” but to map photos into a vector space so that similar animals end up close together. The query looks like this:

1. animal photo;
2. image embedding;
3. nearest-neighbor search with cosine similarity / ANN;
4. match verification with geography taken into account.

This approach fits reality better: the database can contain mixed-breed animals, animals without an exact breed, different angles, and phone photos. We do not need the class “similar mutt.” We need the nearest visual candidates.

Pitfall 2. Treating a clean photo as the baseline

The most unpleasant thing about the dataset is that it has to look like real life, not like an animal show. A lost pet may lose weight, get dirty, end up in the frame in the evening, stand sideways, or be partly covered by grass or a leash.

If you train the model only on neat photos, it starts overvaluing coat condition, lighting, and angle. For a person, that is noise: “yes, the dog is dirty, but the spot on its face is the same.” For a model without preparation, dirt can easily become a new feature.

We built training around specific features: breed, coat colors, unique markings. We also separately accounted for transformations that happen to an animal after it goes missing: weight, dirt, and changes in appearance. The goal was not to make a pretty comparison of two identical images, but to avoid missing a match when the “before” and “after” photos look like they came from different stories.

A practical detail: in discussions, people argued most not about ANN, but about what counts as a stable feature. Coat color — yes, probably. But wet dark fur suddenly breaks confidence. A marking on the face is stronger. The shape of a spot is even stronger, if you can see it.

Pitfall 3. Sending a push for every similar photo

Similarity search gives you candidates, but it does not guarantee that this is the same pet. Especially if the area is dense, there are many users, and there are enough similar animals in the database.

The project had 50,000+ users, and notifications could not be turned into noise. If a push arrives too often, people stop reacting to it. If the filter is too strict, you can miss a needed match.

So the match was tied not only to visual similarity, but also to a radius. The model searches for similar profiles, then the system checks geography and only after that sends a push. ANN is needed here for fast vector search, cosine similarity for ranking similarity, and the radius for filtering out visually similar but useless options from a search perspective.

The trade-off was an engineering one: don’t try to guess the owner from a single score. First narrow the candidates quickly by image, then filter by location, then show the person the match. That path is what produced hundreds of reunited pets, not a nice demo with two matching photos.

The next layer worth checking separately in systems like this is how push quality changes as the database grows and as the density of listings increases in a single area.