Exactly. Gasoline, for example, is remarkably non-toxic, but it will cause instant chemical burns to your throat and lungs, possibly killing you far below the (chemically) lethal dose.

Methanol will turn you blind at a quarter of the listed dose, and those two are just from the top of my head.