It is easy to learn optimization by absorbing a list of methods and examples that somebody else chose. It is harder, and far more valuable, \to learn how to manufacture examples on demand: examples that isolate a single phenomenon, expose a hidden hypothesis, or stress-test a theorem at its boundary.
This article is a recipe book for constructing optimization problems with specific behaviors. The goal is not to produce random instances, but to build problems that answer questions like these:
Featured Gaming CPUTop Pick for High-FPS GamingAMD Ryzen 7 7800X3D 8-Core, 16-Thread Desktop Processor
AMD Ryzen 7 7800X3D 8-Core, 16-Thread Desktop Processor
A strong centerpiece for gaming-focused AM5 builds. This card works well in CPU roundups, build guides, and upgrade pages aimed at high-FPS gaming.
- 8 cores / 16 threads
- 4.2 GHz base clock
- 96 MB L3 cache
- AM5 socket
- Integrated Radeon Graphics
Why it stands out
- Excellent gaming performance
- Strong AM5 upgrade path
- Easy fit for buyer guides and build pages
Things to know
- Needs AM5 and DDR5
- Value moves with live deal pricing
- Why does strong convexity matter for rates and stability
- What does it mean for constraints to be well-behaved
- How do nonsmooth terms change the geometry of a minimizer
- When does duality give a certificate and when does it refuse to cooperate
- How can a problem be convex and still be numerically difficult
Throughout, the constructions stay small enough to verify by hand, but general enough to scale to higher dimensions.
Choose the ambient space and the notion of geometry
Every example begins by choosing a space and an inner product. Most first-order proofs assume Euclidean geometry, but many modern arguments use a different geometry via a norm or a Bregman divergence.
- Euclidean geometry: $\|x\|_2$ and the dot product. Best for least squares and smooth convex analysis.
- One-norm geometry: $\|x\|_1$ and its dual norm $\|\cdot\|_\infty$. Best for sparse regularization and sharp corners.
- Simplex geometry: the probability simplex with the entropy divergence. Best for mirror-descent style arguments.
A quick design rule: if you want an example with many minimizers, choose a geometry that makes flat directions obvious. If you want an example with a unique minimizer but sharp behavior, use a strongly convex core plus a nonsmooth term.
Decide what you want the example to demonstrate
Good examples have a single intended lesson. The table below pairs common lessons with reliable constructions.
| Desired phenomenon | Construction pattern | Minimal example |
|—|—|—|
| Non-uniqueness of minimizers | objective flat on a subspace | $f(x,y)=x^2$ |
| Uniqueness and stability | add strong convexity | $f(x)=\tfrac12\|x\|^2$ |
| Nonsmooth geometry at optimum | add a norm penalty | $f(x)=\tfrac12(x-a)^2+\lambda|x|$ |
| Constraint qualification failure | constraints meet tangentially | $x^2\le 0$ at $x=0$ |
| Duality gap or missing multipliers | non-closed or thin feasible set | construct via limits of feasible points |
| Ill-conditioning | stretch a quadratic | $f(x)=\tfrac12 x^T Q x$ with large condition number |
The rest of the recipe is selecting parameters so the intended behavior becomes unavoidable.
Recipe: build a convex objective with the exact curvature you want
Quadratics are the cleanest way to control curvature. Let
If $Q\succeq 0$, the function is convex. If $Q\succeq \mu I$, it is $\mu$-strongly convex and has a unique minimizer.
Flat directions and non-uniqueness
To force infinitely many minimizers, choose $Q$ with a nontrivial kernel. In two dimensions, take
so $f(x,y)=\tfrac12 x^2$. Every point with $x=0$ is a minimizer. This example is the simplest way to show why many algorithms prove convergence of function values without guaranteeing convergence of iterates.
To make the flat direction occur only near the optimum, add a small nonlinear term, such as $\epsilon y^4$, which keeps convexity and introduces weak curvature.
Ill-conditioning without changing convexity
To create a convex problem that is difficult for gradient descent, choose
with $\kappa\gg 1$. The condition number controls how different directions scale. Level sets become long ellipses, and fixed step sizes that work in one direction can be too aggressive or too timid in another.
This single construction explains many practical behaviors:
- small steps that seem necessary in practice
- slow progress along shallow directions
- sensitivity to scaling and preconditioning
Recipe: add a nonsmooth term to create corners and sparsity
Nonsmooth terms are a controlled way to introduce sharp geometry. The $\ell_1$ norm is the standard example:
Even in one dimension, $g(x)=\lambda|x|$ creates a kink at zero. When combined with a quadratic data term, it produces a thresholding phenomenon.
One-dimensional soft-thresholding example
Consider
You can solve this exactly by considering the subgradient condition:
The solution is
This tiny example is a laboratory for several concepts at once:
- subgradients as optimality certificates
- how nonsmoothness can set coordinates exactly to zero
- why proximal operators appear naturally
Building higher-dimensional versions
To scale up, choose a linear map $A$ and consider
Many geometric phenomena now depend on properties of $A$, such as whether columns are nearly dependent, but the core nonsmooth geometry is still the one-dimensional kink repeated across coordinates.
Recipe: manufacture constraints that reveal what KKT really needs
Constraints are where the deepest mistakes happen, so examples are valuable.
Clean constraints: polyhedra and affine sets
If you want constraints that behave predictably, use:
- affine constraints $Ax=b$
- inequality constraints $Gx\le h$ with a nonempty interior
These settings make constraint qualifications easy to verify. They are also where duality is most reliable.
A constraint qualification failure you can see
A standard way to break KKT multipliers is to make the feasible set meet the boundary tangentially. In one dimension, consider minimizing $f(x)=x$ subject \to $x^2\le 0$. The feasible set is $\{0\}$, so $x^\star=0$. But the gradient of the constraint $g(x)=x^2$ vanishes at the feasible point, so the usual multiplier condition cannot produce the necessary stationarity in the standard way.
This example teaches two lessons:
- feasibility alone is not enough to guarantee nice multiplier behavior
- constraint qualifications exist because the geometry can degenerate
Turning a feasible set into a boundary phenomenon
To highlight boundary behavior in higher dimensions, choose a cone constraint such as $\|x\|_2\le t$ or a semidefinite constraint on a matrix variable. Many duality and sensitivity arguments become statements about normals to cones, which makes the geometry explicit.
Recipe: create a duality certificate problem
If you want an example where duality is the star, use linear programming or quadratic programming with clear geometry.
Linear programming as a corner model
A linear program is
The feasible set is a polytope or polyhedron, and optima occur at extreme points when they exist. Dual variables act as supporting hyperplanes. This example family is perfect for demonstrating:
- complementary slackness as a geometric condition
- why dual feasibility is a certificate
- why degeneracy can create multiple optimal bases
Quadratic programming as a curvature-plus-corners model
A convex quadratic program combines a quadratic objective with linear constraints. It is the simplest class where both curvature and polyhedral geometry appear together. Many proofs about primal–dual methods are easiest to understand here.
A practical workflow for building your own examples
You can build most pedagogically useful optimization examples by following a stable workflow.
- Pick a phenomenon you want to isolate.
- Choose a base class: quadratic, composite convex, constrained polyhedral.
- Add the smallest ingredient that forces the phenomenon: a kernel, a kink, a stretched direction, a tangential constraint.
- Verify the intended property directly: compute the minimizer, compute a subgradient condition, or write the dual and check the gap.
- Stress-test by adjusting one parameter at a time and watching which hypothesis breaks.
This is also how you should read optimization theorems. The fastest way to understand a theorem is to create an example that nearly violates it. The hypotheses then stop looking like decoration and start looking like load-bearing structure.
Closing perspective: examples are a proof tool
Examples in optimization are not merely illustrations. They are instruments:
- they identify the minimal assumptions needed for a claim
- they show which quantities control stability and rates
- they reveal when a certificate should exist and when it cannot
Once you can build examples deliberately, you stop arguing in the dark. You can aim your proofs at the correct structure and you can recognize immediately when a statement is too strong to be true.
Recipe: introduce stochasticity without changing the mathematical core
Many modern optimization problems are written as averages:
or as finite sums
To build an example that captures the behavior of stochastic gradient methods, keep the geometry simple and put the randomness in the data.
A clean construction is linear regression with random covariates. Let $\xi=(a,b)$ where $a\in\mathbb{R}^d$ and $b\in\mathbb{R}$, and define
Then $F(x)$ is a quadratic determined by the covariance $\mathbb{E}[aa^T]$ and the cross term $\mathbb{E}[ab]$. You can now tune the example by choosing the spectrum of $\mathbb{E}[aa^T]$:
- a wide spectrum creates ill-conditioning and slow deterministic progress
- heavy-tailed sampling creates noisy gradients and irregular progress
- nearly singular covariance creates flat directions and non-uniqueness in the population objective
This single family lets you separate three effects that are often conflated: curvature, sampling noise, and model mismatch.
Recipe: create a duality boundary case on purpose
Duality failures and pathologies are easiest to see when the feasible set is not closed. A classic pattern is to enforce a strict inequality that you can approach but never attain.
For example, consider minimizing $f(x)=x$ subject \to $x>0$. The infimum is $0$ but there is no minimizer. Any dual object that pretends there is an optimizer will necessarily misbehave, because the primal problem has no attained optimum to certify.
In higher dimensions, you can make the same phenomenon appear with a thin feasible region:
You can drive the objective toward $0$ by taking $x_2\to\infty$, but no finite point achieves the infimum. This example is a precise warning: existence theorems are not optional. Many clean duality statements assume the right closedness and coercivity properties, even when authors suppress them for readability.
Recipe: make sensitivity and conditioning visible
If you want an example that teaches why small perturbations matter, use a constrained least squares problem with an almost-dependent constraint.
Let $A\in\mathbb{R}^{n\times d}$ be well-conditioned, and impose an affine constraint $u^T x = \delta$ where $u$ is nearly orthogonal to the directions where $A$ has strong curvature. As $\delta$ varies, the minimizer can move a large distance even when the objective value changes little. This is a concrete way to see:
- why Lagrange multipliers can be large
- why scaling constraints changes numerical behavior
- why regularization can stabilize solutions
When you tune $u$ \to align with a flat direction of the quadratic, the example becomes a sensitivity amplifier, and most stability claims reveal exactly which hypothesis they need.
Books by Drew Higgins
Leave a Reply