Am I penalized if I don't have enough reserves?

No. If the game could have used more reserves but you didn't have any more active you are not penalized. In fact, if your reserve would have performed poorly (like an 0-3) you were actually better off not having him active since in the game resolution formula the fewer at-bats with the same number of hits improves your on-base percentage.