The complementary function is \(y_{c}=e^{2 x}\left(c_{1} \cos 2 x+c_{2} \sin 2
x\right) .\) We assume a particular solution of the form \(y_{p}=\left(A x^{3}+B
x^{2}+C x\right) e^{2 x} \cos 2 x+\left(D x^{3}+E x^{2}+F\right) e^{2 x} \sin
2 x .\) Substituting into the differential equation and using a CAS to simplify
yields
$$\left[12 D x^{2}+(6 A+8 E) x+(2 B+4 F)\right] e^{2 x} \cos 2 x$$
$$+\left[-12 A x^{2}+(-8 B+6 D) x+(-4 C+2 E)\right] e^{2 x} \sin 2 x$$
$$=\left(2 x^{2}-3 x\right) e^{2 x} \cos 2 x+\left(10 x^{2}-x-1\right) e^{2 x}
\sin 2 x$$
This gives the system of equations
$$\begin{array}{ccc}
12 D=2, & 6 A+8 E=-3, & 2 B+4 F=0 \\
-12 A=10, & -8 B+6 D=-1, & -4 C+2 E=-1
\end{array}$$
from which we find \(A=-\frac{5}{6}, B=\frac{1}{4}, C=\frac{3}{8},
D=\frac{1}{6}, E=\frac{1}{4},\) and \(F=-\frac{1}{8} .\) Thus, a particular
solution of the differential equation is
$$y_{p}=\left(-\frac{5}{6} x^{3}+\frac{1}{4} x^{2}+\frac{3}{8} x\right) e^{2
x} \cos 2 x+\left(\frac{1}{6} x^{3}+\frac{1}{4} x^{2}-\frac{1}{8} x\right)
e^{2 x} \sin 2 x$$