In their original paper, De Koninck & Tenenbaum take the convention that the smallest prime factor of 1 is ∞, presumably on the grounds that the minimum value of the empty set is ∞. Under this convention, the range 1..N does make sense, with the median smallest prime factor being 3 for all N ≥ 3 odd, and 2.5 for all N ≥ 4 even. So formally, the limit as N → ∞ wouldn't exist.
(They further take the convention that the second-smallest factor of a prime number is ∞, but the choice is ultimately irrelevant for the median of the second-smallest factor, since the primes have natural density 0.)
While ∞ is indeed the minimum of the (empty) set of prime factors of 1, it's still not a prime factor, and it feels wrong to include it in median computations.
(They further take the convention that the second-smallest factor of a prime number is ∞, but the choice is ultimately irrelevant for the median of the second-smallest factor, since the primes have natural density 0.)