It's up to the umpires to enforce the rule and make sure that there are only 11 players on the field. Failure by the umpires as well as the opposing team to notice is the same as the failure to notice any other violation: nothing happens on the field itself, but warnings and suspensions can be handed out if done intentionally.
If the substitute has touched the ball and the umpire notices immediately or shortly afterwards, a 5-run penalty can be imposed. If they don't, then, as I said, nothing can be done about it on the field itself as the situation would be the result of an umpiring error or errors.