Primer on Lagrangians

You typically have a function to optimize (minimize or maximize), subject to constraints:

• Equality constraints: , for .
• Inequality constraints: , for .

The Idea

Lagrange multipliers introduce additional variables (multipliers) to incorporate constraints into the objective function. This creates a single "augmented" function, the Lagrangian:

where are the Lagrange multipliers for the equality constraints.

Procedure

1. Construct the Lagrangian: Combine and into .

2. Stationary Points: Find critical points by solving:

   This ensures gradients of the objective and constraints are aligned.

3. Solve: Solve the system of equations to find and .

Inequality Constraints (KKT Conditions)

For inequality constraints , introduce slackness conditions:

where . The KKT conditions are:

1. ,
2. ,
3. ,
4. (complementary slackness).

Geometric Insight

The Lagrange multiplier measures the rate of change of the optimal with respect to the constraint . Think of it as a "shadow price" or sensitivity factor.