The system can then create a description of the picture

May 29, 2015 07:29 GMT  ·  By

Microsoft Research has developed a new system that can automatically identify the objects in a photo and caption them, using the generated keywords to then create a detailed description of the picture.

While such software might not seem helpful at first, Microsoft says that it represents a huge opportunity in the field of artificial intelligence or can be used to develop more advanced systems that see, hear, speak, and even understand just like a human being.

And that's not really out of this world because, as Microsoft explains, the new system was specifically designed “to understand how a human understands the image.”

“For example, when given a picture of a man sitting in front of a computer, the image captioning technology can accurately recognize that it should focus on describing the man in the foreground, not the image on the computer in the background. Because the man has facial hair, it also knows that it is a man, not a woman,” the company explains.

The new software is based on the concept of neural networks, which are made of computing elements developed to work like the human brain and bring together vision and language. This way, error rates are smaller, Microsoft says, and the system is much more effective.

“A woman holding a cat”

As you can see in the photo attached to this article, the system isn't perfect right now and can still incorrectly caption some objects shown in the photo.

For instance, this woman's curly hair has been detected as “a cat,” so the generated description is “A woman holding a cat.” But based on the generated keywords, the system can automatically re-rank sentences and build descriptions that seem to be more relevant, in this case “A woman holding a camera in a crowd.”

At this point, the new system is still in its early stages, but Microsoft is working together with researchers to improve it to the point when it'll be finally ready to be implemented in products available to everyone.