It's not too bad to get to. A full day though, including Butchart. You'd need to rent a car. From Butchart you an drive about 2.5 hours up Island, take a look around, then head for the northern route ferry back to Vancouver. A weekday would be good. Friday not so much.
On your VICTORIA Day:
About 4 hours from Vancouver to Butchart, including the 1.5 hour ferry ride and the 1 hour wait for the ferry. Ferry ressie would be a good idea given your plans. With a ressie you need to be at the ferry terminal 60-30 min ahead of sailing to claim it, pay your fare and get in the priority boarding line.
2.5 hours from Butchart via ferry from Brentwood Bay to Mill Bay across Saanich Inlet, including the wait for the ferry and the ferry ride and up to Errington. Buy your ferry ticket from the nearby Thrifty Foods store around Butchart, as you will get a discount and no messing with cash with the ferry attendant.
http://www.bcferries.com/schedules/mainland/bbmb-current.php
Later, half an hour from Errington to the ferry at Departure Bay (Nanaimo), 1 hour wait for the ferry unless you make reservations, over to Horseshoe Bay and back into Vancouver. Just under 4 hours journey.
Map: https://goo.gl/maps/pWsUF
Or on your WHISTLER Day:
Vancouver to Errington via Horseshoe Bay > Departure Bay, about 3.5-4 hours. Errington to Whistler via the same ferry, an hour wait for the ferry, unless you reserve, about 4.5 hours. Then Whistler to Vancouver about 2 hours.
Map: https://goo.gl/maps/W6E3U
The first route plan, Vancouver > Butchart > Errington > Vancouver will provide more scenery to look at. You could consider an overnight on the Island. Leave Vancouver for Butchart about 2 pm, visit Butchart, maybe stay up Island at Chemainus, nice little town, Best Western Hotel is there, do the 39 outdoor mural walkabout. About an hour from Butchart using the Mill Bay ferry, Next day up to Errington in an hour. Look around: Cathedral Grove, Englishman River Falls, Little Qualicum Falls, Coombs Country Market. Then back to Vancouver in about 4 hours.