Accelerating Business Hypothesis Validation by 1000x with OpenAI o1 — Entering an Era of Solving Every Problem with Mathematical Models

Having recently used OpenAI's latest model, 'o1-preview', I experienced a revolutionary change that improved operational efficiency by 10 to 1000 times, so I summarized it in a note.

Click here for the English version

Revolutionary Utilization of OpenAI o1

Since I started using OpenAI o1-preview, tasks that previously took weeks or even months can now be completed in just a few minutes.

Especially in intellectual production work. The time it takes to find clues for a hypothesis has changed dramatically.

Seeing is believing.

I will introduce this through 7 case studies across 4 types of business operations.

1. Historical Research

2. Current Situation Analysis

3. Future Forecasting

4. Verifying Strategy Probability

1. Historical Research

Case Study 1: Exploring Success Factors in the SaaS Market

You are a researcher on the team of a major Japanese SIer company. This company provides a one-time purchase CRM tool. You have been instructed to investigate why the recently booming SaaS model is growing and what its success factors are.

"Should our company also transition from a one-time purchase model to a recurring model (SaaS/subscription)?"

You must provide an answer to this question.

As a researcher, you intuitively feel that the subscription model is superior, but you are not convinced that your company, as an SIer, should change.

Therefore, you decided to research the history of Salesforce, which transitioned from a one-time purchase model to a subscription model and achieved the No. 1 market share in the CRM industry.

How will you proceed with your research?

Do you collect books and online articles?

In such cases, try giving o1 the instruction: 'Define an econometric model and elucidate the reasons why Salesforce captured market share.'

It then defined an econometric model and devised variables as follows. A list of plausible-looking variables has been generated.

Once the variable definitions were complete, it assumed and inserted initial values.

Then, insights based on the simulation calculation results were output.

I thought it would be better to shift to a subscription business, but I couldn't come up with the logic to explain it.

However, by looking at these results,'I see! Indeed, as mentioned in 5. Pricing Strategy, subscriptions are overwhelmingly cheaper and easier to cancel than on-premise solutions. As a result, the structure is designed to attract a wide range of customer segments!'I realized,

I found a clue for a strong subscription theory.

Moreover, what is surprising is that this only took about 5 minutes. I was able to obtain clues in an instant.

However, what is written is a bit abstract and doesn't quite feel right. So, we will improve the accuracy in the next step.

Case 2: Improving Simulation Accuracy by Adding Contextual Information

Salesforce is a company with a long history, and there are many historical documents and online articles. Various reference information can be obtained regarding the reasons for its success. Therefore, creating a model based on actual success stories will likely improve accuracy.The excellent point about o1 is that it can effectively combine mathematical problems with contextual information.

For reference, let's input the information from a historical study on customer success that I conducted in the past.

You just need to pass the context and send the instruction to GPT as follows.

Considering the history of customer success and how Salesforce achieved success within it, reconstruct an econometric model based on the success factors.

Then, the effect of customer success initiatives (CSPractices) was added to the independent variables. I am curious about how much effect can be obtained by engaging in customer success.

The calculation results are out.

It seems that by engaging in customer success, the churn rate was significantly improved.

If you were told by your boss at an SIer, 'Customer success is fine, but is there really a return on investment?', you have now gained another clue for your logic.

Customer success is directly linked to churn prevention, and therefore, revenue. Moreover, it exerts its effect gradually through compound interest.I have gained clues that turn doubts into conviction.

2. Analysis of the Current Situation

Case 3: Analyzing the Company's Position in the Market

Here is the next case. You are a marketer managing an SNS service for a major company. This SNS service provides text-based communication and boasts a large market share. However, market competition is fierce, and you never know when you might lose that share. Amidst this, the rapidly growing video platform TikTok is targeting your market share.

I had heard from an in-house data scientist about a method to express retention rates using a probabilistic model. However, because I do not have advanced mathematical knowledge, it remained just knowledge.

I have research data for both the competitor and my own company at hand.

Now, let's use o1 to attach the paper title as follows, feed it the research data, and instruct it to analyze the market structure.

This paper was one I referred to previously when I simulated retention rates using a probabilistic model.

Insights that took me over a week to analyze in the past were output in just a few minutes.

Honestly, I was surprised by this.

Moreover, with o1, you don't need to know the inner workings of the model. This is because it explains even things you don't know in an easy-to-understand way.

Case 4: Identifying Root Causes Through Large-Scale Interviews with Organizational Members

Here is the next case. You are in charge of everything from new product development to production and distribution at a major automobile manufacturer.

In automobile manufacturing, you must create a production plan in advance, secure factory operating conditions, have the supply chain manufacture parts, and only then can you begin assembly, so you cannot easily backtrack like in the startup industry. However, with the wave of EV transition approaching, innovative product development is required.

While the planning department wants to verify value quickly and in an agile manner, the development department insists on proceeding carefully using the waterfall method.

Because of this, the two departments cannot reach a compromise, and various problems are occurring within the organization. You conducted interviews with each stakeholder once to grasp what kind of problems are occurring overall, and summarized them in meeting minutes.

Reading the meeting minutes, it became clear that various problems are occurring in many areas.

However, I realized that the chain of problems—where one problem creates another, which in turn creates yet another—is intricately intertwined.

I am at a loss, not knowing which problem to tackle first.

Such issues are known as complex system problems. Intricately intertwined events mutually influence one another. Therefore, it is necessary to identify the root cause that acts as a key factor, from which all other problems ripple out.

So, let's proceed as follows. Since this is a problem of an interaction system, we will use the "Systems Thinking" approach to have it simulated as a computational problem.

The results have been output, and it has become clear.

Among the various problems occurring,the most fundamental factor was "lack of coordination between departments."According to this simulation result, if coordination between departments does not go well, the efficiency of the development process decreases, and it seems that development resource priorities cannot be set appropriately either.

Indeed, now that you mention it, I can think of instances where this is true, and it makes sense. You decided to first raise this issue and move toward a solution.

3. Future Prediction

Case 5: Growth Forecast for Emerging Markets

You are a CMO at an early-stage HR startup trying to enter the spot-work market. The spot-work market appears to be dominated by incumbents like Timee and Mercari. However, you believe in the future with a reasonably high resolution that spot-work will grow into a massive market.

However, investors point out, "Isn't it impossible to enter now?"

You must logically explain the future market size and the prospects for winning to achieve fundraising.

However, you lack the logic to explain it well. This is especially true because it is a new growth market, making it difficult to explain using existing examples.

Therefore, you decided to forecast the market size of the spot-work market 10 years from now.

Since it is a medium- to long-term future, you thought it would be appropriate to use a humanistic approach (= consideration from the perspectives of technical, social, economic, and cultural backgrounds, as well as consumer needs) rather than thinking based on actual numbers, and to use a method of backcasting the image of society 10 years from now from a broad perspective.

Furthermore, I wanted to incorporate logical rigor using the recently trending "probabilistic marketing model based on scientific empirical rules."

I prompted o1 as follows to build a humanistic and scientific market structure model and forecast the market size.

The model selected was the NBD-Dirichlet model. This is a famous model for explaining market size, known from "Probability-Based Strategy" and the Ehrenberg-Bass Institute. (*While famous, it requires advanced mathematics to master, and only a very small number of companies have been able to put it into practical use...)

The results are in.

I found that the spot work market will grow into a 400 billion yen market in 10 years, and the figures obtained were consistent with the backcasted future predictions I usually consider as a CMO and the gut feeling I experience in my daily activities, which has led to increased confidence.

However, this is merely a humanistic backcasted future prediction. Just to be sure, let's also grasp the current market at the category level. I will have the NBD-Dirichlet model confirm the current market share by treating the part-time job market and the spot work market as a single market.

In this way, I was able to visualize the current state of the part-time job and spot work market.

Furthermore, let's check the extent to which the customer base is shared among each brand.

Now that I understand the state of the market, given that the part-time job market is currently 900 billion yen, it is convincing that the spot work market will grow into a 400 billion yen market in 10 years.

After that, I continued to verify from various angles, and thanks to this, I also found clues for the logic to explain to investors.

4. Verifying the Probability of Strategy

Case 6: Strategy Verification through Sensitivity Analysis

As a CMO, should I immediately start working on digital marketing measures to capture the spot work market?

No. It is important to first identify the conditions for capturing the market based on the market structure and confirm that it is a battle that can be won before taking it on. Also, since competitors are ahead, it is necessary to think about the order in which to deploy strategies to capture the market and clarify that path.

I had some image of the necessary conditions in my head, but I became anxious about whether I had overlooked anything.

To grasp all the conditions for capturing the market, I decided to parameterize the conditions and perform a sensitivity analysis to confirm the validity of the market strategy.

The conditions necessary for market capture have been converted into variables.

The calculation results are output, and ultimately, it even formulates an execution plan for the strategy.

I understood the importance of acquiring end users and clients.

However, thanks to this analysis, I realized that I had completely neglected 'reliability and compliance enhancement'.

Thanks to this, I have a grasp of the strategy and the order in which resources should be deployed, and I have gained more material to persuade investors.

Case 7: Updating the Business Financial Plan

Now, the business financial plan I had originally created in a Spreadsheet (Excel) aimed for 30 billion yen in sales after 10 years.

The spot work market is an area where network effects are at play. Because it is a market where a virtuous cycle—'users gather because many companies use it, and companies gather because many users use it'—is easy to generate, there is a risk that one to three companies could take it all.

Could I somehow strive to aim for a sales scale of 100 billion yen (about 25% market share) in 10 years?

I decided to update the business plan. I also wanted to anticipate how much funding would be needed in which series if I were to aim for 100 billion yen in 10 years.

I converted the Spreadsheet Excel into a CSV, pasted it into the text, and had it revised so that the sales after 10 years would reach 100 billion yen. I also decided to have it propose business and financial strategies toward that goal.

First, it seems it correctly understood the current business plan figures.

Furthermore, it also proposed a funding plan.

It also tells me about the strategies that should be changed.

Finally, a business financial plan with the challenging goal of 100 billion yen in 10 years was also output.

Through these efforts, the CMO was able to instantly create a market structure, current positioning, business strategy, and a draft of the business plan. Based on this, we decided to first talk to investors and proceed with the strategy deployment sequentially.

Conclusion

So, what did you think?

Until now, it was impossible to verify the probability of ideas using such advanced mathematical models even when they occurred to you.

In a sense, the democratization of mathematical models is shocking, as it allows for the completion of analyses that previously took over a month in just five minutes.

Also, it doesn't matter if you don't understand the meaning of the mathematical formulas. I am not good at math either. If you ask OpenAI (o1, 4o) about the meaning of the model's formulas, they will explain them to you carefully.

Abstract hypotheses and intuitions about the future can arise suddenly, but verifying whether they are correct requires a lot of effort. By using o1, you can confirm their validity in just a few minutes.

Instantly turning flashes of insight into conviction.

I felt that possibility.

Random thoughts

• I felt that the greatest brilliance of OpenAI-o1-preview is its ability to create a mathematical model that reflects context by providing contextual information, hypotheses, minutes of internal meetings, and records of user interviews. For example, if you have a strongly held hypothesis, inputting all the contextual information creates a quantitative model that faithfully reproduces the mental model in your head. When I examined strategic options through sensitivity analysis, the simulation results revealed that it was actually better to solve the problem in the order of A→C→B rather than A→B→C, and I was surprised to find a sense of conviction in such high resolution.

• Also, it is known that o1 is not good at word problems. Therefore, it was necessary to define them as mathematical problems as much as possible. For example, when solving the interaction model for organizational analysis in Case 4, it seemed that if I passed the minutes as they were, it would be processed as a word problem. By converting the minutes into a list of state variables S in advance, I was able to calculate it as a system optimization problem (differential equations, SD-Model, SEM, etc.).

  • In fact, since it is difficult for humans to do this "mathematical problem formulation" work, I used multiple LLMs. I also took the steps myself: 1. Human consults LLMa about the problem → 2. LLMb identifies the type of problem and proposes an appropriate analysis method → 3. LLMc creates a mathematical problem using that analysis method → 4. LLMd (o1) solves the mathematical problem → 5. LLMe (4o) explains the calculation results in an easy-to-understand way. (It seems that such LLM workflows will become important in the future.)

• Also, the mindset of believing in the AI was important. I felt that prompt engineering was required more strongly than with 4o, as I had to believe that it could definitely produce excellent output and use techniques to repeatedly request it,in other words, it was more demanding.

  • Perhaps this is because if it is not abstracted and structured to some extent, it is difficult for it to be recognized as a mathematical problem.

  • When an unsatisfactory result is returned, repeating interactions such as "You are a genius! You can definitely do it!" or "That's 20 points. Do it over (Fukatsu-style? Power harassment prompt)," and even "Focus more on the XX context" will dramatically improve accuracy, but if you give up halfway, it will settle into a simulation that does not reflect reality. It seems that the question is whether you can believe in the potential of AI to the end.

• I also had 4o explain the simulation results of o1, and if there were any changes to the hypothesis, I had o1 run a different simulation. Going back and forth between o1 and 4o was also effective.

• In conclusion, rather than "being able to verify hypotheses 100% correctly," I got the impression that "I have gained a good sparring partner who gives me sharp clues."

Finally, if there is anyone who uses similar methods or has other interesting ways of using it, please let me exchange information with you! I would like to explore various things myself.
(*Although I cannot use it for about 5 days due to usage restrictions)