Jim Caldwell is rated as the 14th most aggressive head coach of fourth down.
Considering that Caldwell had an above average offense and a below average defense, he should have been among the MOST aggressive coaches in football. Instead, he rates as just about exactly league average. That's an inefficiency the team cannot afford. The better the offense, the better the chance of conversion.
I don't believe Caldwell made a mistake by calling timeout in Jacksonville, and I don't believe he made a mistake by calling timeout against the Jets. I do believe his passivity on fourth down is a plague and simply has to change.
The Colts waste too many fourth down opportunities settling for punts inside the 50 yard line.
This is the one single area in which Caldwell must improve as a coach. His other decisions might not have worked out, but they were not 'gaffes'. He had strategic justifications for them (even if not everyone agrees with those justifications). There's a big difference between a gamble and a stupid mistake.
Punting on fourth and short on the other side of the 50?
That's a stupid mistake.