After talking with Sam VK2BVS and Binu, VK2BNG I thought I'd try simply connecting both antennas to the same balun.
First contact today on 20m PSK31 got an excellent report from Adrian, VK4RV so clearly 20m works fine.
I can see that the 40m antenna has been somewhat de-tuned and now resonates below the band and is SWR 2 where I need to use it. 80m seems relatively unaffected.
This nifty graph was drawn with gnuplot and the following commands:
set terminal png
set title "20m and 40/80 dipoles"
set output "out.png"
set xlabel "Frequency MHz"
set ylabel "SWR"
plot "swr.dat" using 1:2 title 'SWR' with linespoints
Binu also pointed me to an interesting article about Off-centre-fed (OCF) dipoles by Ron Bertrand, VK2DQ that looks worth a go too.