You Can Now Upload Images to Google Bard; Here Are Some Cool Examples

Google has been quietly improving Bard and addingnew featuresevery few weeks, bringing its capabilities up to par with ChatGPT. Now, the company has added the ability to upload images to Bard for a much broader experience besides text. Make no mistake, Google Bard is still a text-onlylarge language model. However, the search giant has integrated Google Lens, reverse image search, and a few VQA systems (Visual Question Answering) to make Bard feel like a multimodal model. Nevertheless, Bard’s current vision capability is indeed somewhat surprising, and we have tested it below to learn about its capabilities. On that note, let’s take a look at some cool examples of image uploads in Google Bard.

1. Extract Texts from Images

The best utility of Bard’s image-handling ability is that now you can upload an image by clicking on the(+)button. It can then quickly grab texts from the uploads. Google Bard thenautomaticallyperforms OCRand does an accurate job. That being said, despite a long list of language support in Bard, currently, the OCR functionality only works for the English language. I tried multiple international and regional languages, but it failed to grab texts from scanned images. Nevertheless, for quick text extraction from images, Bard can be very helpful.

2. Extract Tables with Formatting Intact

We all struggle when we have to extract tables from scanned images or documents. However, Google Bard can effortlessly extract tables with the formatting intact. In fact, you canexport the table to Google Sheetsas well and do further editing or data crunching. How cool is that? Having said that, currently, Bard hallucinates a lot, and in some cases, it fills the cells with the wrong data, so make sure to verify them before exporting it.

3. Generate Code for Websites/ Apps Using Mockups

To showcaseGPT-4‘s multimodality feature, in March 2023, OpenAI demonstrated how its modelunderstood the scribbled noteand quickly created a mockup of the website from a piece of paper. While the multimodal feature is yet to come to GPT-4, Google Bard is able to generate code that matches the mockup. Keep in mind thatBard is not a multimodal modelbut uses image segmentation viaGoogle Lensto understand the image. Nonetheless, Bard surprised us with its results.

I uploaded a screenshot of the Facebook landing page, and it quickly generatedcode in HTML and CSSthat looked somewhat similar. I also uploaded an image of a simple website that I drew on paper, and Google Bard did a good enough job of recreating it. Further, you can use similar methods for recreating UIs for smartphone apps and other websites as well.

4. Google Bard Can Explain Images

Google Bard is good at explaining images and summarizing what is going on in them. You can upload obscure images, and it canproduce reliable informationquickly. I uploaded a low-quality image of a biological mechanism, and it correctly identified it as Cell Mitosis. It further explained the process step by step.

In another example, Iuploaded a chart,and it correctly understood the image and explained the data. It even created a table of the data points so that I could work on it in Google Sheets. Particularly for students, Bard can be helpful in understanding concepts in science and other topics. You can simply upload an image and ask Bard about it.

5. Get Nutritional Information from Images

Using Bard’s image-handling capability, you can get thenutritional values of food. Simply upload the image of food on your plate, and it willcalculate the total caloriewithin seconds. This can be immensely helpful for people who are on a regulated diet.

In my testing, it couldn’t gauge the portion size but gave examples so that you could calculate the total calorie intake by yourself. It seems Google is using image segmentation to categorize food items and come up with nutritional information.

6. Improvise Food Recipes

Another excellent use case is to add the image of raw food items and ask Google Bard to come up with various food recipes. You can also add images of food items in your refrigerator, and it will effortlessly createpersonalized recipesfor you. Furthermore, you can ask Bard for particular cuisines from various parts of the world. And if you are on a diet, you can ask Google Bard to create fat-free, low-calorie food recipes for satiety.

7. Solve Mathematical Questions

You can use Google Bard to solve mathematical questions as well. You can upload an image of your maths problems to Bard, and it willtry to solve the questionfor you. In my testing, Bard’s approach was right but due to notation issues, it came up with wrong answers only. I think it will require an update to its vision system to make Bard more suitable for handling mathematical notations and questions.

8. Explain Memes and Jokes

Google Bard can also explain memes and jokes. You can upload images of funny memes and cartoons andask Bard what is funnyabout the same, and it will provide its own interpretation. I uploaded the same image that OpenAI demonstrated during theGPT-4unveiling, and Bard rightly understood the hilarious absurdity behind the image.

In another instance, I uploaded an image to Google Bard from The New Yorker Cartoons and asked it toexplain the joke. However, this time, it simply explained the scene and couldn’t tell why the image was funny. It entirely missed the email phrase that is commonly used in workplaces. I will suggest you try Google Bard yourself and check if it’s intelligent enough to understand wit and humor.

9. Translate Equations to LaTeX

Simple#Bardtip, just grab the image of an equation and have it rendered in Latex.pic.twitter.com/XXvuEW1LQB— Carlos E. Perez (@IntuitMachine)July 15, 2023

It’s no secret that many people find it hard to write inLaTeXand prefer to use word processors. However, for scientific research papers and academic writing, LaTeX is required foradding complex equationsand high-quality typesetting. In such a scenario, Google Bard can be helpful. You can add images of equations, and Bard can translate them to LaTeX code. That’s amazing, right? So, go ahead and translate the equations to LaTeX code in no time.

10. Upload Medical Reports and Ask Questions

Finally, you can upload images of your medical reports and scan them to Google Bard. You can then ask medical questions based on them. Some physicians on Twitter have shown that Bard is quitedecent for differential diagnosis. It can also help users to understand their health and make sense of medical reportsGoogleのbard凄い & 怖い ！！！Bardによる脳CTの診断！｢画像には脳の白い部分が写っていると言えます。。。脳腫瘍、脳卒中、出血など、さまざまな原因が考えられます｣答えは脳出血です。その鑑別診断を挙げるだけでも凄いですね。特化型で学習しているわけではありませんので。pic.twitter.com/aEdF5xtlqt— 河野 健一　生成AI ✕ 医療に注目！ 手術支援AI CEO 脳外科医 (@CeoImed)July 14, 2023

Googleのbard凄い & 怖い ！！！Bardによる脳CTの診断！｢画像には脳の白い部分が写っていると言えます。。。脳腫瘍、脳卒中、出血など、さまざまな原因が考えられます｣答えは脳出血です。その鑑別診断を挙げるだけでも凄いですね。特化型で学習しているわけではありませんので。pic.twitter.com/aEdF5xtlqt— 河野 健一　生成AI ✕ 医療に注目！ 手術支援AI CEO 脳外科医 (@CeoImed)July 14, 2023

That said, do keep in mind Google Bard is running on ageneral-purpose LLMcalledPaLM 2. The search giant has developed a separate medical-domain Med-PaLM 2 model, which is quite accurate and advanced, but it’s not available to general users yet. So I will recommend users stay away from any kind of self-diagnosis using Bard. It’s strongly recommended to consult a doctor. And finally, if you upload your personal medical reports to Bard, make sure todelete Bard chatsto protect your privacy.

Arjun Sha

Passionate about Windows, ChromeOS, Android, security and privacy issues. Have a penchant to solve everyday computing problems.

Add new comment

Name

Email ID

Δ

01

02

03

04

05