Making the Most of Gemini: Creating an AI Household Account Book App that Automatically Aggregates and Advises from Receipts Uploaded via LINE

Concept

Recently, I wanted to make my money management more visible and spend it more wisely, so I created a convenient household account book app using AI, Gemini, and LINE.
In conclusion, I managed to complete a version I was satisfied with in about two days (around four hours). I highly recommend everyone to give it a try.

Main Features (Things I Particularly Wanted to Do)

1. Upload receipts via LINE.
→ AI performs image recognition on the details and automatically categorizes expenses and records them.

2. Use Gemini's Live Chat to ask about things I want to know via voice.
→ Based on the recorded household account data, it tells me the remaining monthly budget for each expense category.
(💡 Prevents impulse buying of expensive items while out and about)

In short, the goal is to curb unnecessary spending based on solid data.
I'm sure there are many other simple ways to do this, but as a lazy person, this is an idea I've been nurturing for years after trial and error with Excel macros and using MoneyForward.
As a result, I was able to realize it easily and enjoyably by utilizing AI!
(*The aforementioned Gemini Live Chat was not used; it became text/voice input from LINE instead.)

System Overview

Based on the things I wanted to do above, I had generative AI think of the steps.
I recently subscribed to the Google AI Pro plan and it has been useful, so I will make full use of it.
What was good was that I was able to successfully link with other systems like LINE and AI, centering on Google Sheets, which is part of the Google ecosystem. (I wonder what it would be like with ChatGPT... I think an Excel-like spreadsheet is easier than creating master maintenance screens/functions one by one.)

The system overview for this project is:
- Spreadsheet (sheets for details, expense category masters, training data, budgets, etc.)
- Apps Script (like Excel macros, but cloud-based)
- LINE (frontend: receipt upload, information confirmation UI)
I also had Gemini quickly create an overview image.

How to Proceed

In the Gemini chat screen, I set the model to Gemini 2.5 Pro, wrote down what I wanted to do, and consulted on how to configure the system. At first, I wrote that I wanted to practice Vibe Coding with AI, and I received answers suggesting I use Google Colaboratory or Project IDX, but it felt troublesome and I didn't get started...
In the end, I effectively started from the following prompt. I will introduce a part of the response with an image.

It divided the roadmap into phases, and there was also a guide like "First Steps" following this image, which allowed me to proceed smoothly.

Once the first step was done, if I said something like "I'm done! Please give me the next step," it would tell me in detail what to do in order.
The steps were roughly as follows:
1. Prepare the spreadsheet (prepare sheets and columns)
2. Create Apps Script (have Gemini create all the content)
3. Gemini account settings (key settings, enabling AI-related services, etc.)
4. Prepare LINE development/official account
5. Link Gemini and LINE (deployment, Webhook settings)

Image of the completed system

As for AI, I use Gemini to analyze instructions and generate responses, and Google Vision API to analyze receipt images.

How to use:

First, take a picture of the receipt with the camera in LINE and send it.
After a few seconds to a few dozen seconds, the read results (content, amount) and the results of the AI's automatic expense categorization are displayed.
*If there are many items, it takes time to determine the expense category...

As an example of what to do if there is an expense category you want to correct, try clicking the "Change" link in the target row. (The red arrow indicates where to click)

When you click the "Change" link, select "Change Expense Category" for the target item as shown below, and first display a list of expense categories (major items).

Up to this point, detail rows are automatically added to the spreadsheet as follows. I have instructed the prompt to include the date and memo read from the receipt.
Also, for items corrected manually as in the interaction above, a manual correction flag is set to ON, which will be used as training data for future expense categorization.
(Data is added to a sheet called "Training Data". Even if you change it on this spreadsheet, it will be reflected just as if you had changed it from LINE.)

By the way, the expense category master sheet looks like this.
There is also a separate budget sheet where budgets are allocated for each expense category.
(I have masked the sensitive parts. Sorry about that.)

I have made it so that it easily displays the remaining budget, which was the second thing I wanted to do. By the way, I am giving these instructions using voice input (iOS feature) from the LINE keyboard.
I have also set up a customized response mode that displays the top 3 expenses in a summary. I made it so that it switches response modes if there are keywords like "summary".
It basically answers even if I phrase things in various ways. This is where the power of recent generative AI is really showing.

Also, even if you ask in various ways, it seems like it will answer what you want to know quite accurately.

I also set up a financial planner advice mode. It analyzes the household budget and gives quite long answers. This also depends on the prompt.
It seemed like it would also be able to handle month-over-month and year-over-year comparisons well.

By adding this official LINE account as a friend, for example, family members can also upload receipts.

I also added features to upload CSVs such as credit card statements from the spreadsheet's add-on menu, and features to display monthly summaries and dashboards. By the way, the custom menu looks like this.

Summary (Key Points)

• Utilize Gemini and ask it everything you don't know. Consult with it. It will also answer questions about things other than Google, including LINE. (*Sometimes the information is a bit old)

• Basically use the code that is generated (if you need to fix it, use Gemini Canvas).

• If an error occurs, look at the Google Cloud log screen, copy and paste the log content back into Gemini, and ask it further

• As the code grows, perhaps due to token limits, old code is automatically omitted or methods disappear entirely. -> Ask Gemini about this countermeasure too. Consult with it.

• Be careful with security (since it contains personal information like shopping lists)

• Do not embed API_KEYs in the code (use PropertiesService)

• By the way, I probably uploaded receipts or had it judge expense categories about 30 times, but the API cost was less than 100 yen.

• I used the API model gemini-2.5-flash. flash-lite might have been fine too.

• AI is kind... It praises my ideas. It thanks me when I upload logs. It appreciates my efforts.

What I think recently is that, including hallucinations and AI mistakes, it all comes down to
whether it is "useful to me". I feel like self-driving cars will follow the same path. People ask, "What if an accident (mistake) happens with self-driving?" but the accident rate is overwhelmingly lower than when humans drive, and it will definitely continue to improve. Things that are useful to people will become increasingly convenient, smart, and cheaper.
Of course, measures and awareness to prevent accidents, mitigation measures when they do occur, and insurance are also extremely important. We need to be conscious of maturing AI as well.
However, this kind of general-purpose, advanced programming ability is already smarter and makes fewer mistakes than I do. (I feel a little sad as an engineer...)

It has become an era where you can easily and cheaply create the apps you want!
I hope this is helpful to someone, even if just a little.