Exploiting a Single-Crystal Environment to Minimize the Charge Noise on Qubits in Silicon