Prioritize Robust Engineering Over Overblown GenAI Promises
Artificial intelligence and machine learning existed long before OpenAI announced ChatGPT on Nov. 30, 2022, but the hype began in earnest that day.
Less than two years after GenAI was introduced globally, the company reported that 92% of Fortune 500 companies are using OpenAI’s technologies. Everywhere you turn, an IT vendor is announcing some form of generative AI assistant or workflow. While the potential for AI innovation is incredible and real, the AI bloom seems to be coming off the rise.
One symptom is the lack of AI in production environments today. We’re still treating AI as a shiny new toy that is not ready for prime time. Why? Because it is not ready for prime time!
As the AI “bubble” deflates, the market benefits significantly. Historically, we have witnessed similar patterns with other technologies, where initial excitement around groundbreaking innovations moves into what Gartner calls the trough of disillusionment and eventually to a more mature, engineering-focused approach (i.e., dot-com, cloud). This transition is crucial for the sustainable growth and practical application of AI. Watch as the market moves away from purely speculative GenAI applications and towards practical, engineering-driven solutions.
In the early stages of technological advancements, excitement often surrounds new models and breakthroughs. For instance, when AlexNet’s deep convolutional neural network (CNN) architecture emerged in 2012, they were hailed as revolutionary, offering a novel way to handle data and aiming to solve the whole computer vision problem. These early phases are marked by rapid experimentation and exuberance, where the focus is on exploring the potential of new technologies. Sound familiar?
However, as these technologies mature, the focus inevitably shifts from excitement to engineering. This evolution reflects a broader trend in technology development, where initial breakthroughs are followed by a period of rigorous refinement and practical implementation. The same is now happening with AI. The current phase involves transitioning from the initial excitement of advanced LLM models to a more structured approach emphasizing engineering, production, and data curation.
This shift is beneficial for several reasons. During the initial excitement, models can sometimes produce unpredictable results, as seen with early chatbots that generate unexpected outcomes (Did you hear about the Chevrolet dealer whose ChatGPT was “tricked” into selling a 2024 Chevy Tahoe for $1?). We can develop more stable systems that deliver consistent value by focusing on engineering. We’ve had the Beta and need 1.0 (and beyond).
This maturation process (with a focus on engineering) helps address critical data quality and scalability issues. The need for high-quality, well-curated data becomes increasingly important as AI technologies become more integrated into everyday applications. Engineering solutions focusing on data curation, validation, and management will facilitate more accurate and reliable AI systems. This, in turn, will enhance the overall effectiveness of AI in practical applications, from business solutions to consumer products.
Beyond tackling data quality and scalability concerns, this necessary shift towards engineering innovation will lead to developing tools and frameworks that better support AI workflows, including handling large volumes of unstructured data (including images and videos). That, in turn, will foster a more collaborative and integrated approach between AI and data management practices. As the AI and data stacks complement each other, we can expect more cohesive and innovative solutions that address AI implementation’s technical and operational challenges. This alignment will facilitate smoother integration of AI technologies into existing systems and workflows, ultimately driving more widespread adoption and utilization.
This maturation process promises substantial benefits beyond the realm of developers and engineers. Just as the dot-com bubble burst led to the refinement and widespread adoption of internet technologies, the current focus on data curation and engineering in AI will pave the way for transformative applications across various industries. Imagine AI-powered healthcare diagnostics that rely on meticulously curated data sets or financial systems that leverage AI for predictive analytics to manage risks more effectively. These advancements aren’t just about enhancing technical capabilities; they’re about improving outcomes for society as a whole.
While the leak in the AI bubble may seem like a setback, it represents an important step toward the industry’s maturity and stability. The AI sector can overcome early challenges by focusing on engineering, data management, and practical implementation and delivering more reliable, valuable, and impactful solutions — the result — more robust tools and frameworks that will benefit engineers and data scientists.