解:
(1)由题意,得$\begin{cases}a + b = 1\\3a - 2b = 8\end{cases},$解得$\begin{cases}a = 2\\b = -1\end{cases}.$
(2)由
(1)知$x*y = ax + by = 2x - y,$$x\bigotimes y = ax - by = 2x + y,$
所以方程组$\begin{cases}x*y = 4 - m\\x\bigotimes y = 5m\end{cases}$可化为$\begin{cases}2x - y = 4 - m\\2x + y = 5m\end{cases},$
解得$\begin{cases}x = m + 1\\y = 3m - 2\end{cases}.$
因为$x + y = 5,$
所以$m + 1 + 3m - 2 = 5,$解得$m = \frac{3}{2}.$
(3)由题意,得$\begin{cases}x + y = 4\\x - y = 5\end{cases},$解得$\begin{cases}x = 4.5\\y = -0.5\end{cases}.$