Hume: Missing shade of blue

Hume, as an empiricist, believes that any idea in the mind is really just an impression of something that was experienced through the senses. His example of the missing shade of blue, then, bothers me because it exactly disproves that idea.. and he agrees that it is a counterexample to his theory, but he doesn't seem to care. Assume that a man has seen every shade of every color in the world, except for one particular shade of blue. Pretend that this man is shown a series of cards, each of which has a shade of blue that he has previously experienced, and the cards are presented in order from lightest to darkest, with the exception of a blank card representing the shade of blue he has never seen. Would the man be able to imagine what that shade of blue would look like? Hume says yes, and I would agree with him. The man could infer that the missing shade is just a bit lighter than the card to its right, and just a bit darker than the card to its left. What bugs me about this is that Hume admits that the man would be able to figure that out, but that's in direct opposition to his basic arguments as an empiricist. The excuse Hume gives is that the example is "so singular that it is scarcely worth our observing, and does not merit that for it alone we should alter our general maxim." I don't see why that is justified, because the missing shade of blue seems like a perfectly legitimate objection to his arguments. Hume kind of just dismisses it instead of tackling the problem.