I assume the given plane has equation
x + 3y + 4z = d
for some constant d ≠ 0.
You want to maximize the function V(x, y, z) = xyz subject to the above constraint. The Lagrangian would be
L(x, y, z, λ) = xyz - λ (x + 3y + 4z - d)
Find its critical points:
∂L/∂x = yz - λ = 0
∂L/∂y = xz - 3λ = 0
∂L/∂z = xy - 4λ = 0
∂L/∂λ = x + 3y + 4z - d = 0
The first three equations tell you that λ = yz = xz/3 = xy/4. Then, for instance,
yz = xz/3 → 3yz - xz = z (3y - x) = 0 → z = 0 or 3y - x = 0
If z = 0, we get zero volume which is not useful. So 3y = x. Similarly, you would find x = 4z and 3y = 4z, so a critical point occurs when x = 3y = 4z.
In the fourth equation, we get upon substituting
x + 3y + 4z = 3x = d → x = d/3, y = d/9, z = d/12
Then the largest volume of this box would be
V(d/3, d/9, d/12) = d ³ / 324
