The most famous open problem in computer science directly touches cryptography, optimisation, AI, and algorithm design — yet most engineers only know the surface. Here is the full picture, without the hand-waving.
If you have ever Googled “P vs NP,” you have almost certainly landed on an answer that sounds like this: “P is the set of easy problems; NP is the set of hard ones. We don’t know if they’re the same set.” That is technically not wrong, but it leaves out almost everything that matters. It doesn’t explain what easy and hard actually mean in a formal sense, why the question is so extraordinarily difficult to settle, or — most importantly for practising engineers — what would actually change if someone proved it either way.
So let’s fix that. By the end of this piece you will understand what P and NP really are, why NP-completeness is the conceptual workhorse of practical computer science, and why a proof of P=NP would be, as Scott Aaronson of UT Austin once put it, “the most important scientific discovery in human history.”
The formal definition of NP-completeness traces back to Stephen Cook’s landmark 1971 paper, “The Complexity of Theorem-Proving Procedures”. Richard Karp extended it the following year with 21 NP-complete problems that shape algorithm design to this day.
1. What “P” and “NP” Actually Mean
Both P and NP are complexity classes — families of decision problems (problems with a yes/no answer) grouped by how much computational resource they require. The resource we care about here is time, measured not as wall-clock seconds but as a function of input size.
The Class P
A problem is in P (Polynomial time) if there exists an algorithm that solves every instance of size n in at most c·nk steps, for some fixed constants c and k. In plain terms: the running time grows as a polynomial in the size of the input. Sorting a list of a million items, finding the shortest path between two nodes in a road network, or checking whether a number is prime — all of these are in P.
Crucially, the specific polynomial doesn’t matter much in theory. An O(n100) algorithm is technically in P, though it would be useless in practice. The class P is really about identifying a frontier: problems where the difficulty scales in a manageable way as inputs grow larger.
The Class NP
A problem is in NP (Non-deterministic Polynomial time) if, given a proposed solution, you can verify whether it is correct in polynomial time. NP is not “non-polynomial” — that is the most persistent misconception in all of computer science education.
“NP does not stand for ‘not polynomial.’ It stands for ‘non-deterministic polynomial’ — and the distinction is everything.”
Consider Boolean satisfiability (SAT): given a logical formula with hundreds of variables, is there an assignment of true/false values that makes the whole formula true? Finding such an assignment might take exponential time with the best known methods. But if someone hands you a candidate assignment, checking it is easy — just substitute the values and evaluate. That fast verification is precisely what puts SAT in NP.
Notice, therefore, that every problem in P is also in NP: if you can solve a problem in polynomial time, you can certainly verify a solution in polynomial time (just solve it again and compare). So P ⊆ NP. The question is whether the containment is strict — whether NP contains genuinely harder problems that polynomial-time solvers cannot crack.
The Complexity Landscape
2. NP-Completeness: The Most Useful Idea in Algorithms
The concept that makes P vs NP practically important — not just theoretically interesting — is NP-completeness. A problem is NP-complete if (a) it is in NP, and (b) every other problem in NP can be reduced to it in polynomial time.
That second condition is the stunning part. It means NP-complete problems are, in a precise mathematical sense, the hardest problems in NP. If you found a polynomial-time algorithm for even a single NP-complete problem, you would instantly have polynomial-time algorithms for all of them — and P would equal NP.
Stephen Cook proved in 1971 that SAT is NP-complete. Richard Karp then showed, in a 1972 paper that remains one of the most-cited in computer science, that 21 other natural problems — including the Travelling Salesman Problem, graph colouring, and the knapsack problem — are all NP-complete as well. Today, thousands of problems across optimisation, biology, economics, and engineering have been shown to be NP-complete.
| Problem | Class | Real-World Domain | Best Known Approach |
|---|---|---|---|
| Shortest path (Dijkstra) | P | Navigation, networking | Exact in O((V+E) log V) |
| Primality testing (AKS) | P | Cryptography | Exact in O(log⁶ n) |
| Linear programming (simplex/interior) | P | Operations research, ML | Exact (polynomial worst-case) |
| Boolean SAT | NP-complete | Chip design, verification | CDCL solvers (exponential worst-case) |
| Travelling Salesman (decision) | NP-complete | Logistics, circuit layout | Branch & bound + heuristics |
| Graph 3-colouring | NP-complete | Register allocation, scheduling | Exact + approximations |
| Integer factorisation | Unknown (NP ∩ co-NP?) | RSA cryptography | Sub-exponential (GNFS) |
| Halting problem | Undecidable | Static analysis, verification | No algorithm exists |
3. The Growth Problem: Why Exponential Is a Cliff Edge
To appreciate why NP-completeness matters so deeply, it helps to see the numbers. The difference between polynomial and exponential growth is not merely quantitative — it is qualitative. Below is a chart with real numbers showing how algorithm steps scale with input size, using the kinds of values that appear in real solver benchmarks.
Algorithm Steps by Input Size
For an input of just 50 items, an O(2n) algorithm requires over a quadrillion steps. At 300 items — the scale of a real RSA key — the number of steps exceeds the estimated number of atoms in the observable universe. This is not a hardware problem. No amount of Moore’s Law progress closes that gap.
This is, incidentally, exactly why your HTTPS connection is secure right now. Factoring the product of two large primes is believed (though not proven) to be hard. Your browser and the server you’re talking to exploit that hardness every single time they complete a TLS handshake.
4. If P = NP: A World Turned Upside Down
Suppose tomorrow morning a researcher posts a proof on arXiv showing that P = NP — and, crucially, that the polynomial-time algorithm is practical (a small constant, a reasonable exponent). What would actually happen? The consequences fall into three broad categories.
Cryptography: The Immediate Crisis
Almost all public-key cryptography in use today — RSA, Diffie-Hellman, elliptic-curve schemes — relies on the presumed hardness of NP problems like integer factorisation and discrete logarithm. A constructive P=NP proof with a practical algorithm would break all of it immediately. Every encrypted file, every HTTPS session, every signed software package would become readable to anyone with the algorithm.
⚠ What Would Actually Break
RSA-2048 (used by most of the internet), TLS 1.3’s key exchange, PGP email encryption, SSH private key authentication, and most blockchain consensus mechanisms all depend on computational hardness assumptions that P=NP with a fast algorithm would invalidate overnight.
It is worth noting, however, that not all cryptography is equally affected. Symmetric encryption like AES does not directly rely on NP-hard problems in the same way. And post-quantum cryptography, currently being standardised by NIST, is designed around problems believed to be hard even for quantum computers — and potentially even for a P=NP world, depending on the proof’s scope.
Optimisation and Operations Research: Enormous Gains
The flip side of the cryptographic catastrophe would be a golden age for optimisation. Virtually every hard scheduling, routing, and resource-allocation problem that engineers currently approximate with heuristics would become exactly solvable in reasonable time. Airline scheduling, supply-chain management, drug discovery, protein folding (at the sequence-to-structure level), and compiler register allocation would all improve dramatically — or be solved outright.
Machine Learning: A More Subtle Story
The relationship between P=NP and machine learning is more nuanced than popular accounts suggest. Training deep neural networks is not, formally, an NP-complete problem — gradient descent doesn’t need to solve SAT. However, many hard problems in ML are NP-complete: finding the optimal architecture for a neural network, training certain classes of networks to global optimality, and some feature-selection tasks. A P=NP result would therefore improve, but not magically resolve, all of ML.
More profoundly, P=NP would imply that creativity itself is automatable in a precise sense. Any problem where a proposed solution can be verified efficiently — including writing a proof, composing a symphony that satisfies certain criteria, or designing a molecule with given properties — could in principle be solved, not just checked, in polynomial time. The boundary between “human insight” and “mechanical computation” would shift in a fundamental way.
Industrial SAT Solver Progress (1992–2023)
That chart is worth pausing on. SAT solvers have improved by roughly five orders of magnitude since the early 1990s, thanks to algorithmic breakthroughs like Conflict-Driven Clause Learning (CDCL) and better heuristics. Modern solvers like CaDiCaL and CryptoMiniSAT routinely handle industrial verification problems with millions of variables. And yet they still have exponential worst-case complexity. Engineering progress is phenomenal; P≠NP has not budged.
5. If P ≠ NP: Confirming What We Already Assume
A proof that P ≠ NP would not change the practical landscape much — engineers already act as if P ≠ NP every day they use RSA or design approximation algorithms. But it would be enormously important for theory. It would:
- Formally justify the security assumptions underlying all of modern public-key cryptography.
- Tell algorithm designers that certain problems have provably no exact polynomial-time solution, redirecting effort toward approximations and heuristics without guilt.
- Almost certainly introduce powerful new mathematical tools — the proof technique would be as valuable as the result itself.
- Settle Ladner’s theorem implications: if P ≠ NP, there exist problems in NP that are neither in P nor NP-complete — a genuinely richer landscape than the binary picture suggests.
6. Why Has Nobody Proved It? The Genuine Obstacles
P vs NP was formally posed by Stephen Cook in 1971, building on earlier intuitions by Gödel in a 1956 letter to von Neumann. It is one of the Clay Millennium Prize Problems, carrying a $1,000,000 prize for a correct solution. After more than 50 years of effort from the world’s best mathematical minds, we remain stuck. Why?
The Relativisation Barrier
In 1975, Baker, Gill, and Solovay showed that most standard proof techniques cannot resolve P vs NP. Specifically, they proved that there exist “oracle” settings (hypothetical computational environments) where P=NP, and others where P≠NP. Any proof technique that would work regardless of the oracle cannot distinguish between these worlds — and that rules out almost all the tools complexity theorists normally use, including diagonalisation, the technique behind the proof that the halting problem is undecidable. This was a profound early signal that the problem requires genuinely new mathematics.
The Naturalisation Barrier
Razborov and Rudich’s 1994 result on “natural proofs” was equally sobering. They showed that any proof of circuit lower bounds that satisfies two very reasonable-sounding properties — being constructive and applying to a large fraction of functions — cannot work if strong pseudorandom generators exist (which most cryptographers believe they do). Since most lower-bound arguments naturally satisfy those properties, this rules out another enormous class of approaches.
The Algebrisation Barrier
Aaronson and Wigderson extended the relativisation barrier in 2009 to algebraic oracles, closing yet another potential route. Any proof that works in the “algebrised” setting — which includes most algebraic and spectral techniques — also cannot resolve P vs NP. Between relativisation, naturalisation, and algebrisation, virtually every standard mathematical toolkit has been systematically ruled out.
Scott Aaronson’s lecture notes, freely available at scottaaronson.com, give the clearest non-technical account of the barriers. The Clay Mathematics Institute problem description by Stephen Cook is the authoritative formal statement. For practical implications, Sipser’s Introduction to the Theory of Computation remains the standard graduate textbook.
7. Integer Factorisation: The Fascinating Edge Case
A common misconception is that RSA security requires P≠NP. In fact, the relationship is more delicate. Integer factorisation — given a number N, find its prime factors — is known to be in NP (verifying a factorisation takes polynomial time). But it has not been proven to be NP-complete. It sits in a curious intermediate zone: we strongly suspect it is harder than P, but we cannot rule out a polynomial-time factoring algorithm even if P≠NP.
The best classical algorithm today, the General Number Field Sieve (GNFS), has sub-exponential complexity — it runs in roughly exp(O(n1/3)) time. That is much better than exponential, but still vastly slower than polynomial for large n. Meanwhile, Shor’s quantum algorithm (1994) solves factorisation in polynomial time on a quantum computer, which is why large-scale quantum computing would break RSA even without resolving P vs NP at all.
| Algorithm | Type | Complexity (bits) | RSA-2048 Feasibility |
|---|---|---|---|
| Trial division | Classical | O(2n/2) | Utterly infeasible |
| Quadratic sieve | Classical | Sub-exponential | Infeasible (>1030 ops) |
| GNFS (best classical) | Classical | exp(O(n1/3)) | ~1027 ops — infeasible |
| Shor’s algorithm | Quantum | O(n3) — polynomial | Feasible with ~4,000 logical qubits |
8. What This Means for Everyday Engineering
P vs NP is not purely academic. It shapes the decisions engineers make in the real world, even when they don’t realise it. Here are three concrete implications worth internalising.
First, proving a problem is NP-complete is useful, not a dead end. When you identify that your scheduling or packing problem reduces to a known NP-complete problem, you are learning something important: there is no magic exact algorithm waiting to be discovered. That redirects your effort productively toward approximation algorithms (with provable guarantees), heuristics, or problem reformulations. The approximation algorithm literature for NP-hard problems is rich precisely because of this.
Second, not all NP-complete instances are hard in practice. The worst-case hardness of NP-complete problems does not mean every real-world instance is difficult. SAT solvers handle millions of variables in industrial practice because real-world instances have structure that generic worst-case analysis ignores. Understanding why your specific instances are tractable — and engineering to preserve that structure — is genuinely valuable work.
Third, P vs NP is already answered in your security model. Every time you choose RSA key lengths, set password hashing iteration counts, or evaluate whether a zero-knowledge proof scheme is sound, you are implicitly betting on specific hardness assumptions. Those assumptions are more granular than P≠NP — they concern specific computational problems with specific parameters. Knowing the landscape helps you evaluate those bets more clearly.
If you encounter a new combinatorial problem in your codebase, the first step is to check whether it is NP-complete via reduction to a known problem. Garey and Johnson’s “Computers and Intractability” (1979) remains the canonical reference — and it still lists the most complete catalogue of NP-complete problems available anywhere.
9. What We Have Learned
We started with the most famous unsolved problem in computer science and worked our way through what it actually claims. P is the class of problems solvable quickly; NP is the class verifiable quickly — and P is a subset of NP, but whether the two classes are equal remains unknown after more than 50 years of effort. NP-completeness, introduced by Cook in 1971 and extended by Karp in 1972, identifies a set of problems that are, in a precise sense, the hardest in NP: a polynomial-time solution to any one of them solves all of them.
A constructive proof of P=NP would demolish public-key cryptography, unlock enormous optimisation gains, and blur the line between verification and discovery. A proof of P≠NP would formally validate every security assumption the modern internet depends on. Neither proof is in sight, partly because the Baker–Gill–Solovay and Razborov–Rudich results have systematically closed off almost every standard proof technique. For practising engineers, the practical lesson is clear: identifying NP-completeness focuses effort productively, real-world instances are often tractable despite worst-case hardness, and the security assumptions underlying every encrypted connection you make today rest on this unresolved theoretical foundation.
