[THOUGHT] We need to maximize the quadratic function $F(x,y,z)$ over the polyhedron $S$ defined by nonnegative variables and four linear constraints. This is a concave quadratic maximization (since the Hessian of $-5x^2-4y^2-6z^2+4xy-2xz+3yz$ is negative semidefinite on $S$), so the maximum occurs at a vertex of $S$. We will enumerate all vertices by intersecting constraint planes, keep only those satisfying all inequalities, evaluate $F$ at each, and take the maximum. Then write $M$ as a reduced fraction $p/q$ and compute $p^2q$. [/THOUGHT] [PYTHON] import itertools

def F(x,y,z): return -5xx - 4yy - 6zz + 4xy - 2xz + 3yz + 20x + 14y + 15*z + 1

Planes: x>=0, y>=0, z>=0, x+y=2, x+2y=3, x+z=3, y+z=3

planes = [ (1,0,0, 0), # x=0 (0,1,0, 0), # y=0 (0,0,1, 0), # z=0 (1,1,0,-2), # x+y=2 (1,2,0,-3), # x+2y=3 (1,0,1,-3), # x+z=3 (0,1,1,-3), # y+z=3 ]

vertices = set() for p in itertools.combinations(planes, 3): A = [p[i][0:3] for i in range(3)] b = [p[i][3] for i in range(3)] try: # Solve Av = b for v=(x,y,z) import numpy as np A_np = np.array(A, dtype=float) b_np = np.array(b, dtype=float) v = np.linalg.solve(A_np, b_np) x,y,z = v if x<=-1e-9 or y<=-1e-9 or z<=-1e-9: continue if x+y-2 > 1e-9 or x+2y-3 > 1e-9 or x+z-3 > 1e-9 or y+z-3 > 1e-9: continue vertices.add((round(x,12), round(y,12), round(z,12))) except: continue

M = None for x,y,z in vertices: val = F(x,y,z) if M is None or val > M: M = val

Reduce M to lowest terms p/q

from fractions import Fraction frac = Fraction(M).limit_denominator() p = frac.numerator q = frac.denominator print(ppq) [/PYTHON]

[THOUGHT] The maximum value $M$ of $F$ over $S$ is $\dfrac{1}{1}$, so $p=1$, $q=1$ and $p^2q=1$. \boxed{1}