解：(2)∵a²+b²=10a+8b-41
∴a²+b²-10a-8b+41=0
∴(a²-10a+25)+(b²-86+16)=0
∴(a-5)²+(b-4)²=0
∵(a-5)²≥0，(b-4)²≥0
∴a-5=0，b-4=0，解得a=5，b=4
∵a，b，c 是△ABC的三边长
∴5-4＜c＜5+4，即1＜c＜9
∴c 的取值范围是1＜c＜9
(3)原式=-2x²+4xy-2y²-y²-6y-9+16
=-2(x²-2xy+y²)-(y²+6y+9)+16
=-2(x-y)²-(y+3)²+16
∵-2(x-y)²≤0，-(y+3)²≤0
∴-2(x-y)²-(y+3)²+16≤16
∴多项式-2x²+4xy-3y²-6y+7的最大值是16
