Swallowtail caterpillars have an organ just behind the head called an osmeterium. When it is bothered, the caterpillar turns a sort of pocket inside out, and these orange "horns" are pushed out and then withdrawn again.
The horns emit a real stink (reminding me of marigolds). Apparently this lets birds know that the caterpillars taste bad. One of our books (Anna Comstock's Handbook of Nature Study has a unit on the Black Swallowtail) states that this protection is why they don't hide under leaves.
I've been trying to get a good photo of this phenomenon, but had no luck until today. Becca found a caterpillar outside in the parsley patch (the one that escaped during cleaning?) and we poked it a bit...for educational purposes, of course.