Survey of Research on Mathematical Optimization Using LLMs
Survey of Research on Mathematical Optimization Using LLMs
Supply Chain Optimization Using Large Language Models
In recent years, supply chain operations have become increasingly complex, and their decision-making processes involve a wide variety of challenges. Over the past few decades, advancements in computational technology have brought significant benefits to supply chains, enabling a transition from manual labor to automation and cost-effective optimization. However, in business settings, significant effort is still required to explain optimization results to stakeholders and gain their understanding.
Against this backdrop, Beibin Li et al. (Microsoft Research) focused on Large Language Models (LLMs), which have seen remarkable development in recent years, and researched their potential to bridge the gap between supply chain automation and human understanding and trust. The paper summarizing these results is "Large Language Models for Supply Chain Optimization" (Beibin Li, Konstantina Mellou, Bo Zhang, Jeevan Pathuri, and Ishai Menache, 2023).
In this paper, they propose a framework called OptiGuide. OptiGuide receives queries input in plain text and outputs insights regarding the underlying optimization results. This framework does not abandon state-of-the-art combinatorial optimization techniques, but rather leverages them to quantitatively answer "what-if" scenarios (e.g., "How would costs change if we used Supplier B instead of Supplier A for a specific demand?").
A notable point is that OptiGuide is designed so that sensitive data does not need to be sent to the LLM.
This is a significant advantage, as it can be a privacy concern in certain situations.
The authors demonstrate the effectiveness of the proposed framework using actual server placement scenarios within Microsoft's cloud supply chain. Furthermore, they have developed a general-purpose evaluation benchmark to assess the accuracy of LLM outputs, which is expected to be applicable to other scenarios as well.
The contributions of this paper are mainly summarized in the following three points.
Presenting a new way to utilize LLMs in supply chain optimization: They demonstrated the potential to utilize LLMs not just as text generation tools, but as tools to support human understanding of complex optimization problems.
Designing a privacy-conscious framework: The ability to provide insights on optimization results without sending sensitive data to an external LLM is extremely important for practical use.
Developing a general-purpose evaluation benchmark: By standardizing the performance evaluation of optimization support systems using LLMs, it is expected to contribute to the future development of research.
This paper can be considered a very interesting research achievement that suggests the potential for LLMs to bring innovation to the field of supply chain optimization. In particular, by enabling business personnel to more easily understand and utilize optimization results, it is expected to lead to improved efficiency and faster decision-making across the entire supply chain. Future research is expected to apply frameworks like OptiGuide to a more diverse range of supply chain challenges and verify their effectiveness.
Diagnosing Infeasible Optimization Problems Using Large Language Models
Decision-making problems can be expressed as mathematical optimization models and are widely applied in fields such as economics, engineering, manufacturing, transportation, and healthcare. An optimization model is a mathematical abstraction of the problem of making the best decision while satisfying a set of requirements and constraints. However, one of the major barriers to utilizing these models in practice is the difficulty in helping practitioners understand and interpret the model, especially when it is infeasible, meaning there is no solution that satisfies all constraints. Existing methods for diagnosing infeasible optimization models often rely on expert systems and require deep expertise in optimization.
Against this backdrop, Hao Chen et al. (Purdue University) propose OptiChat, a natural language-based system equipped with a chatbot GUI for interactively communicating about infeasible optimization models. This paper was published as "Diagnosing Infeasible Optimization Problems Using Large Language Models" (Hao Chen, Gonzalo E. Constante-Flores, Can Li, 2023).
OptiChat can explain the optimization model itself in natural language, identify parts that may be causing infeasibility, and make suggestions to make the model feasible. The implementation of OptiChat is based on GPT-4 and works in conjunction with an optimization solver to identify the minimal subset of constraints that make the entire optimization problem infeasible (called an Irreducible Infeasible Subset (IIS)).
To enhance the reliability of OptiChat, the authors utilize techniques such as few-shot learning, expert-led Chain-of-Thought, key retrieval, and sentiment prompting. Experimental results showed that OptiChat helps both expert and non-expert users deepen their understanding of optimization models and quickly identify the causes of infeasibility.
The main contributions of this paper are as follows.
Proposing an infeasible optimization problem diagnosis system that can interact in natural language: Even users without specialized knowledge can now intuitively understand the problems in an optimization model using natural language.
Building a system that combines GPT-4 and an optimization solver: By combining the advanced language processing capabilities of LLMs with the rigorous problem-solving capabilities of optimization solvers, effective diagnosis is achieved.
Enhanced system reliability by utilizing various prompting techniques: By using techniques such as few-shot learning and Chain-of-Thought, more accurate and human-like responses are generated.
This paper can be considered pioneering research demonstrating that Large Language Models (LLMs) can be applied to the task of diagnosing optimization problems, a domain previously reserved for experts. In particular, the fact that users without expertise in optimization can easily understand the issues in complex models through a natural language interface is of great practical significance.
Future research is expected to focus on handling more complex optimization problems and developing more accurate diagnostic systems that take into account the history of user interactions. The widespread adoption of systems like OptiChat is expected to expand the scope of optimization technology and improve the quality of decision-making in various fields.
AI-COPILOT for Business Optimization: A Framework and Case Study in Production Scheduling
Business optimization refers to the process of finding and implementing efficient and cost-effective operational methods to enhance a company's competitive advantage. Problem formulation is an essential part of business optimization, and it has traditionally relied on humans to construct problem formulations using optimization languages. However, with the advancement of Large Language Models (LLMs), it is becoming possible to minimize the human expertise required for problem formulation.
Against this backdrop, Pivithuru Thejan Amarasinghe et al. (La Trobe University) have proposed an AI-Copilot for problem formulation using LLMs. This paper was published as "AI-COPILOT FOR BUSINESS OPTIMISATION: A FRAMEWORK AND A CASE STUDY IN PRODUCTION SCHEDULING" (Pivithuru Thejan Amarasinghe, Su Nguyen, Yuan Sun, Damminda Alahakoon, 2023).
There are several challenges in developing a problem formulation system using LLMs, such as the lack of training data, token limits, and appropriate performance evaluation metrics. Regarding training data, the approach of fine-tuning pre-trained LLMs for downstream tasks, rather than training LLMs from scratch for specific tasks, is attracting attention. This paper adopts this LLM fine-tuning approach and proposes an AI-Copilot for business optimization problem formulation.
Regarding token limits, it introduces modularization and prompt engineering techniques to modularize complex problem formulations and synthesize the modules to fit within the LLM's token limits. Furthermore, it designs more appropriate performance evaluation metrics to assess the accuracy and quality of the problem formulations.
Experimental results show that by using this approach, it is possible to synthesize complex and large-scale problem formulations for typical business optimization problems in production scheduling.
The main contributions of this paper are as follows:
Proposed an AI-Copilot framework for business optimization: Presented a framework that leverages LLMs to automate and support the problem formulation process that previously relied on human expertise.
Addressed token limits through modularization and prompt engineering: Divided complex problems into small modules and maximized the capabilities of LLMs through effective prompt design.
Designed performance evaluation metrics suitable for problem formulation: Proposed metrics to more appropriately evaluate the accuracy and quality of the generated formulations.
This paper is an important research achievement that demonstrates the potential for LLMs to support the task of problem formulation in business optimization, which has previously relied on human experts. In particular, it is expected that the AI-Copilot will accelerate the business optimization process by working in collaboration with human experts to perform problem formulation more efficiently and effectively.
Future research is expected to include application to a wider variety of business optimization problems and system improvements that incorporate user feedback. As systems like AI-Copilot become more widespread, the barrier to entry for business optimization will be lowered, allowing many companies to benefit from it.
From Large Language Models and Optimization to Decision Optimization CoPilot
Significantly simplifying the creation of optimization models for real-world business problems has long been a major goal for widely applying mathematical optimization to critical business and social decision-making.
The capabilities of recent Large Language Models (LLMs) provide a timely opportunity to achieve this goal. Segev Wasserkrug et al. (IBM Research - Israel) have proposed research at the intersection of LLMs and optimization, aiming to develop a Decision Optimization CoPilot (DOCP). DOCP is an AI tool designed to assist all decision-makers, aiming to understand business problems through natural language dialogue and to formulate and solve corresponding optimization models.
This paper, "From Large Language Models and Optimization to Decision Optimization CoPilot: A Research Manifesto" (Segev Wasserkrug, Leonard Boussioux, Dick den Hertog, Farzaneh Mirzazadeh, Ilker Birbil, Jannis Kurtz, Donato Maragno, 2023), outlines the vision for DOCP and identifies several fundamental requirements for its implementation. Furthermore, it explains the current state through a literature review and experiments using ChatGPT.
The authors show that a) LLMs already provide substantial and new capabilities relevant to DOCP, and b) there are still major research challenges to be addressed. Furthermore, they propose possible research directions to overcome these gaps.
This paper is also positioned as a call to action to unite the LLM and optimization communities to pursue the authors' vision, thereby aiming to enable broader improvements in decision-making.
The main contributions of this paper are as follows:
Presenting the vision of a Decision Optimization CoPilot (DOCP): It outlines the concept of an AI tool that interacts with users in natural language, understands business problems, and builds and solves optimization models.
Identifying the fundamental requirements for realizing DOCP: Based on the current state of technology, it clarifies the functions and performance required for DOCP.
Clarifying the current state of technology: Through a literature review and experiments using ChatGPT, it reveals the current capabilities and limitations of LLMs.
Indicating future research directions: It proposes concrete approaches to overcome the remaining research challenges.
Calling for collaboration between the LLM and optimization communities: Cooperation among researchers in different fields is essential for the realization of DOCP, and this paper serves as a call to action for that purpose.
This paper suggests that combining LLMs and optimization technology has the potential to significantly change the way decision support is provided. If tools like DOCP are realized, even users without specialized knowledge will be able to utilize advanced optimization technology to solve business problems.
Future research is expected to follow the proposed research directions, such as developing prototypes of DOCP and improving the system through user interaction. This paper can be said to be an important first step in opening up the future of decision support through the fusion of LLMs and optimization.
LM4OPT: Unveiling the Potential of Large Language Models in Formulating Mathematical Optimization Problems
The field of natural language processing is evolving rapidly, but among these, formulating content described in natural language as a mathematical optimization problem is a very difficult task that requires advanced understanding and processing capabilities from Large Language Models (LLMs).
Tasnim Ahmed et al. (Queen's University) compare the performance of LLMs on this task and explore their potential. In this paper, "LM4OPT: Unveiling the Potential of Large Language Models in Formulating Mathematical Optimization Problems" (Tasnim Ahmed, Salimur Choudhury, 2023), they compare major LLMs, including GPT-3.5, GPT-4, and Llama-2-7b, in zero-shot and one-shot settings.
The results of the study revealed that, especially in one-shot scenarios, GPT-4 shows superior performance. The core part of this study is the introduction of "LM4OPT", a framework for incrementally fine-tuning Llama-2-7b using noisy embeddings and specialized datasets. However, this study revealed that, especially when processing long and complex input contexts, smaller models like Llama-2-7b have a significant gap in context understanding ability compared to larger models.
Through an empirical investigation using the NL4Opt dataset, the authors revealed that GPT-4 achieved an F1 score of 0.63 based solely on natural language problem descriptions, surpassing the baseline performance established in previous studies. This was achieved without relying on additional named entity information. GPT-3.5 also showed performance exceeding that of the fine-tuned Llama-2-7b, yielding results close to those of GPT-4.
The main contributions of this paper are as follows:
Comparative evaluation of the performance of major LLMs in the task of formulating mathematical optimization problems from natural language: It revealed that GPT-4 shows particularly superior performance.
Proposing "LM4OPT", a framework for incrementally fine-tuning Llama-2-7b: It suggests the possibility of contributing to the performance improvement of small-scale models.
Pointing out the difference in context understanding ability between small-scale and large-scale models: It clarified that there are challenges in processing capabilities, especially for long and complex inputs.
Demonstrating that GPT-4 can formulate optimization problems with high accuracy from natural language alone: It achieved performance exceeding previous studies without additional information.
This paper can be called a pioneering study that clarified what capabilities LLMs have and what challenges exist in the formulation of mathematical optimization problems. In particular, showing that GPT-4 can formulate optimization problems with high accuracy from natural language alone will contribute significantly to the development of future research.
In the future, it is expected that by further developing frameworks like LM4OPT and improving the contextual understanding capabilities of smaller models, optimization problems can be formulated more efficiently and effectively. This paper can be considered an important step that opens up the potential for a new field of application: the formulation of optimization problems using LLMs.
ORLM: Training Large Language Models for Optimization Modeling
Large Language Models (LLMs) have emerged as powerful tools for tackling complex Operations Research (OR) problems by providing the ability to automate optimization modeling. However, current methodologies rely heavily on prompt engineering using proprietary LLMs (e.g., multi-agent collaboration), which raises data privacy concerns that can be a barrier to industrial application.
To address this issue, Zhengyang Tang et al. (The Chinese University of Hong Kong, Shenzhen) propose training open-source LLMs for optimization modeling. In this paper, "ORLM: Training Large Language Models for Optimization Modeling" (Zhengyang Tang, Chenyu Huang, Xin Zheng, Shixi Hu, Zizhuo Wang, Dongdong Ge, Benyou Wang, 2023), they identify four key requirements for training datasets for OR LLMs and design and implement OR-INSTRUCT, a semi-automated process for creating synthetic data that meets those requirements.
Furthermore, the authors introduce IndustryOR, the first industrial benchmark for testing the performance of LLMs in solving real-world OR problems. They apply data obtained from OR-INSTRUCT to various 7B-sized open-source LLMs (referred to as ORLM), significantly improving their optimization modeling capabilities. The highest-performing ORLM achieves state-of-the-art performance on the NL4OPT, MAMO, and IndustryOR benchmarks.
The main contributions of this paper are as follows:
Proposing open-source LLM training for optimization modeling: Demonstrated the potential to mitigate data privacy concerns and facilitate industrial application.
Identifying four key requirements for OR LLM training datasets: Provided guidelines for creating high-quality training data.
Developing OR-INSTRUCT, a semi-automated process for synthetic data creation: Enabled the efficient and effective creation of training data.
Introducing IndustryOR, an industrial benchmark based on real-world OR problems: Enabled the evaluation of LLM performance from a more practical perspective.
Significantly improving the optimization modeling capabilities of various open-source LLMs through the proposed method: Achieved state-of-the-art performance and demonstrated the effectiveness of the proposed method.
This paper is a significant research achievement that opens up the possibilities for optimization modeling using open-source LLMs. In particular, the fact that it achieves high performance while mitigating data privacy concerns is of great significance for industrial applications.
Future research is expected to include application to larger models and support for a more diverse range of OR problems. As technologies like ORLM develop, the automation of optimization modeling is expected to advance, leading to more efficient and sophisticated decision-making in various fields.
The code and data for this paper are available at https://github.com/Cardinal-Operations/ORLM, contributing to the reproducibility and advancement of research.
OptiMUS-0.3: Modeling and Solving Large-Scale Optimization Problems Using Large Language Models
Optimization problems permeate every field, from manufacturing and distribution to healthcare. However, many of these problems are still solved based on manual heuristics rather than being optimally solved using state-of-the-art solvers. This is because the expertise required to formulate and solve optimization problems hinders the widespread adoption of optimization tools and technologies.
To address this challenge, Ali AhmadiTeshnizi et al. (Stanford University) propose OptiMUS-0.3, an LLM-based system designed to formulate and solve (mixed-integer) linear programming problems described in natural language. The details are explained in the paper "OptiMUS-0.3: Using large language models to model and solve optimization problems at scale" (Ali AhmadiTeshnizi, Wenzhi Gao, Herman Brunborg, Shayan Talaei, Madeleine Udell, 2023).
OptiMUS-0.3 can develop mathematical models, write and debug solver code, evaluate generated solutions, and improve the efficiency and accuracy of models and code based on these evaluations. A major feature is the adoption of a modular structure, which allows it to process problems with long problem descriptions or complex data without requiring long prompts.
Experimental results show that OptiMUS-0.3 outperforms existing state-of-the-art methods by more than 12% on simple datasets and by more than 8% on difficult datasets (including NLP4LP, a new dataset released in this paper featuring long and complex problems).
The main contributions of this paper are as follows:
Proposed an LLM-based system that formulates and solves (mixed-integer) linear programming problems from natural language descriptions: OptiMUS-0.3 automates the process of formulating and solving optimization problems, enabling users without specialized knowledge to utilize advanced optimization techniques.
Adopted a modular structure to handle long problem descriptions and complex data: A major strength is its ability to process problems efficiently without requiring long prompts.
Realized a system that integrates mathematical model development, code generation, solution evaluation, and improvement: It completes the entire series of processes required for solving optimization problems within an LLM-based system.
Demonstrated performance exceeding existing methods through experiments: It showed its effectiveness, particularly in datasets containing long and complex problems.
This paper can be considered an important research achievement that demonstrates how LLMs can contribute to solving optimization problems. With the widespread adoption of systems like OptiMUS-0.3, it is expected that the scope of application for optimization technology will expand significantly, improving the quality of decision-making in various fields.
Future research is expected to address non-linear and stochastic optimization problems, as well as system improvements through user interaction. This paper can be called a pioneering study that opens up the possibilities for solving optimization problems using LLMs.
CHAIN-OF-EXPERTS: When LLMs Tackle Complex Operations Research Problems
Large Language Models (LLMs) have emerged as powerful technologies in various natural language processing tasks, such as mathematical reasoning and plan generation. However, in the automated modeling and programming of complex Operations Research (OR) problems, there remains a heavy reliance on domain experts, which is a challenge in many industrial sectors.
Ziyang Xiao et al. (Zhejiang University, Huawei Noah's Ark Lab) propose the first LLM-based solution using a multi-agent collaborative framework called Chain-of-Experts (CoE) to solve this challenge. The details are explained in the paper "CHAIN-OF-EXPERTS: WHEN LLMS MEET COMPLEX OPERATIONS RESEARCH PROBLEMS" (Ziyang Xiao, Dongxiang Zhang, Yangjun Wu, Lilin Xu, Yuan Wang, Xiongwei Han, Xiaojin Fu, Tao Zhong, Jia Zeng, Mingli Song, Gang Chen, 2023).
In CoE, each agent is assigned a specific role and provided with OR-related expertise. Furthermore, a conductor is introduced to organize these agents using forward-thinking construction and backward-reflection mechanisms. Additionally, to promote OR research and community development, they have constructed a benchmark dataset for complex OR problems (ComplexOR).
Experimental results show that CoE significantly outperforms existing state-of-the-art LLM-based approaches on both the LPWP and ComplexOR datasets.
The main contributions of this paper are as follows:
Proposed the first LLM-based solution for automated modeling and programming of complex OR problems: CoE enhances the reasoning capabilities of LLMs through a multi-agent collaborative framework.
Introduced agents with roles and expertise, along with a conductor to organize them: Achieves effective problem solving through forward-thinking construction and backward-reflection mechanisms.
Constructed a benchmark dataset for complex OR problems (ComplexOR): Provides a valuable resource that contributes to the development of OR research.
Demonstrated through experiments that CoE significantly outperforms existing methods: Showed effectiveness, especially for complex problems.
This paper is an important research achievement that shows how LLMs can contribute to solving complex OR problems. As frameworks like CoE develop, users without OR expertise will be able to utilize advanced optimization techniques to solve business challenges.
Future research is expected to address a wider variety of OR problems and improve the system through user interaction. This paper can be considered a pioneering study that opens up the possibilities for solving OR problems using LLMs.
Gurobi AI Modeling: A Practical Guide to Mathematical Modeling Using Generative AI
In recent years, the development of generative AI has been remarkable, and its application in various fields is expected. In particular, in the field of mathematical optimization, generative AI is attracting attention as a powerful tool to support model construction and problem solving. However, to effectively utilize generative AI, it is important to understand its capabilities and limitations and to use it in an appropriate manner.
In this regard, the "Gurobi AI Modeling" provided by Gurobi Optimization is an extremely useful practical guide for learning mathematical modeling using generative AI. This is a repository published on GitHub under the name gurobi-ai-modeling (https://gurobi-ai-modeling.readthedocs.io/).
This repository aims to help users become familiar with utilizing generative AI for mathematical modeling, and provides comprehensive content including the following:
Assisting in understanding the capabilities and limitations of state-of-the-art generative AI tools in the context of optimization
Proposing best practices for building and solving optimization models using generative AI, and explaining common pitfalls
Sharing prompt examples and expected results
Explaining tools useful for creating and solving models
Specifically, it covers a wide range of topics, starting from how to use ChatGPT as a "translator" for mathematics, to its application in various use cases and specialized fields, effective LLM prompting techniques, and example problems of typical OR problems such as facility location problems, diet problems, Sudoku, workforce problems, and portfolio optimization.
Furthermore, it introduces application examples for specific job roles such as DevOps engineers, backend engineers, and data engineers, allowing users to learn how to utilize generative AI from a more practical perspective.
Also, the "Deep Dive" section explains more advanced topics such as prompt engineering, answer engineering, question interpretation, generation of mathematical expressions, model generation, and result interpretation.
Notably, custom GPTs are provided as "Gurobi AI Modeling Assistant," "Gurobi AI Modeling Prompt Engineer," and "Gurobot". These are designed to help users utilize generative AI more easily and effectively, and their respective benefits and prompt templates are also published.
Of course, there are points to be aware of when mastering generative AI. The "Tips and Pitfalls" section provides practical advice, such as the importance of describing problems as clearly as possible, problems caused by unexpected prompts, points to note regarding variables, and the importance of providing data. Furthermore, it explains how to deal with technical issues such as wheel installation errors, code execution errors, and the correct usage of gurobipy.
In addition, common pitfalls in modeling such as typos, mixed data types, excessive use of inline data, and data preprocessing are also mentioned.
Finally, the limitations of LLM reasoning are also touched upon, pointing out issues such as "blindly assuming supply equals demand," "not knowing how to store inventory for later use," "difficulty thinking in 3D space," and "not understanding how application hosting works."
"Gurobi AI Modeling" is a highly valuable resource for anyone who wants to learn and practice mathematical modeling using generative AI. It targets a wide range of users from beginners to experienced users, and allows them to master effective modeling techniques through practical examples, tips, and pitfalls.
By utilizing this repository, you will be able to maximize the capabilities of generative AI and further expand the possibilities of mathematical optimization.
Summary and Outlook
Above, we surveyed major papers for formulating mathematical optimization with LLMs. However, the progress of LLMs is tremendous, and it feels as though the prompt engineering and multi-agent usage employed in these papers and sites have already become outdated technology. Also, the problems collected as benchmarks are simple textbook-level ones, which seem far from practical application. In the future, it will be necessary to solve practical problems by using newer technologies, and the MOAI Lab is also proceeding with the development of such methods.