Definitively no more than 7nm radius.3.1.4-17 Victoria, BC:
3.1.4-18 The airspace to 2500´ (2400´ AAE) within the area bounded by a line beginning at:
N48°38'28.32" W123°15'00.24" Can/USA bdry \ thence clockwise along the arc of a circle of
7 miles radius centred on
N48°38'49.30" W123°25'32.80" (Victoria Intl, BC - AD) \ to
N48°41'41.16" W123°15'54.65" Can/USA bdry \ to
N48°38'28.32" W123°15'00.24" Can/USA bdry \ point of beginning
I found an old version (2007) of the DAH that had a very complicated CZ structure approximately (but not exactly) including the area you were referring to, but it has been tidied considerably since then and that area to the NW of YYJ is now outside the CZ. Perhaps your experience predates those changes.
As far as the chart is concerned, as you point out, there's airspace outside of the CZ whose controlling agency is the Tower, but that doesn't make it part of the Control Zone. It's either part of the TCA, or it's a Control Area Extension, or a Transition Area. (My head is spinning from too many lat/longs to check which.)
Why is the distinction important? Because there are CARs that apply only within control zones, like the VFR weather minimum of 3sm ground visibility (where reported) that doesn't apply in the area to which you refer.
Conversely, YYZ has a sliver of it's CZ controlled by Terminal and not the Tower. It's still part of the CZ.
Rookie50's point is that around CYTZ, airspace that far away from the runways is controlled by a second controller (typically Satellites East, or Satellites West) who happens to be at the Pearson ACC. At YYJ they also have a second controller for "outer" traffic, only he sits in the Tower. Same difference.