I tried out the multimodal capabilities of GPT-4-Turbo, Gemini 1.5 Pro, and Claude 3 Opus.
I am currently experimenting with multiple AIs using KingJoeBot (a LINE Chat Bot). Since I was able to integrate the Claude 3 API this time, I tested the multimodal capabilities of the three major AIs (based on my own arbitrary evaluation).
Here is the image I had them read. For a source that is a bit hard to see even with the naked eye due to subtle lighting, I entered the same prompt:
"Please provide a concise explanation of this photo."
as the input.
GPT-4-Turbo's response
This photo shows a product box labeled "TEREA for IQOS ILUMA" with a white pattern on a blue package. IQOS ILUMA is a heated tobacco device, and TEREA is a dedicated stick for it. A warning in Japanese is printed at the bottom of the box, stating, "The smoke (vapor) from heated tobacco products may have negative health effects on those around you. Please be considerate of your surroundings while using it, as the possibility of health damage cannot be ruled out." This is a warning to the user that the product is not completely harmless.
Gemini 1.5 Pro's response
## Photo Explanation
This photo is of a package for "TEREA" tobacco sticks for the "IQOS ILUMA" heated tobacco product.
The package contains the following warnings:
* **The smoke (vapor) from heated tobacco products may have negative health effects on those around you.**
* **You cannot smoke in places where it is prohibited by the Health Promotion Act.**
In other words, it warns that even with heated tobacco, there is a risk of health damage to those around you, and it cannot be used in places where smoking is prohibited.
Claude 3 Opus's response
This photo is a shot of the package for an IQOS ILUMA heated tobacco product.
The package is labeled "TEREA for IQOS ILUMA," clearly stating the brand and product name.
Additionally, the explanatory text on the package states, "The smoke (vapor) from heated tobacco products may have negative health effects on those around you. You cannot smoke in places where it is prohibited by the Health Promotion Act," indicating precautions for using heated tobacco.
The package design is simple, based on blue, with a swirling pattern in the center. It can be said that the package design gives an overall sophisticated impression.
Impressions
All three AIs are excellent and correctly recognize the "TEREA" for "IQOS ILUMA." GPT and Claude correctly identify it as "TEREA for IQOS ILUMA" as shown on the label. GPT is the only one that misread "smoke (vapor)" as "leaves (stems)," but I suspect it might have associated "tobacco" with "plant" and then with "leaves." Regarding the text read from the package, GPT and Gemini describe it as a "warning," while Claude limits it to "precautions." It is a minor difference, but personally, I think "warning" is more appropriate. I was particularly impressed by Gemini's "In other words..." sentence. I felt that it digested the content written on the package and explained it in a very easy-to-understand way. While I think GPT and Claude also convey information accurately and "matter-of-factly," I think Gemini is the one that explains the interpretation well. Conversely, it is interesting that only Gemini omits the description regarding the "appearance." Even among GPT and Claude, which do describe the "appearance," GPT limits itself to a somewhat plain expression like "a white pattern on a blue package," whereas Claude uses an emotional (?) expression, saying it "gives an overall sophisticated impression." Since the prompt instructed them to be "concise," I think GPT is the one that answered most faithfully to that instruction.
