If the Super Bowl underwhelmed you, don't blame Stephen Colbert.