I think his low-grade humour has had little impact on how he is perceived. He often gets carried away with his attempt to make fun of the opposition and using names like "Shobaaz" etc., and of course the infamous blatantly racist remarks against Babar Ghauri.
Before we discuss why people are critical of him, it is important to not broadly categorise his critics as PMLN and PPP supporters. A good chunk of former PTI supporters have also lost faith in him, and they are not necessarily supporters of the two said parties.
For some reason, it has become impossible for PTI supporters to comprehend that you can be a critic of Imran, Nawaz and Zardari at the same time. Just because you do not agree with Imran does not mean that you are a support of the latter two.
The major reason why a lot of people have lost faith in him is because he has compromised his moral standing to win the election. For years, he vehemently opposed giving tickets to electables or "lotas" because these people will never be able to bring change. However, when he lost in 2013, he was prepared to compromise the manifesto of his party to gain power.
Let's not look further than Sheikh Rasheed. On a talk show, I believe Capital Talk, Imran said on record that he will not appoint someone like him as his "chaprasi", and today, he is the Railway Minister in his government. Who is going to hold him accountable for this hypocrisy?
Imran is an extremely dedicated and committed individual, and his powers of persistence are incredible. When he sets a goal for himself, he doesn't give up until he achieves it. That is a very admirable trait, but in addition to this great quality, he is also extremely power hungry to the extent that he very easily walks all over his beliefs and ideologies just to achieve the end goal.
All his morals and ethics are for show only. He is a high-achieving narcissist who suffers from Messiah complex who can be as ruthless as anyone when it comes to achieving his objective. From refusing to acknowledge his daughter born out of wedlock to taking u-turns after u-turns in politics - he has shown that he is prepared to go to any length to achieve what he wants to achieve, and he is also very skilled in taking his supporters for a ride, who will defend anything and take everything he says or does at face value.
Had Imran lived a thousand years ago, he would have started his own religion. In his mind, he is always right and and there is never a tiny possibility that he could be wrong about something. He is also very hostile and if you do not agree with what he says, you are simply just another enemy. This "you are with me or against me" attitude has rubbed onto his supporters as well, which is why whenever someone criticises PTI, they are dubbed a PMLN or a PPP supporter.
These are the reasons why people dislike Imran and his party; it is has nothing to do with his attempts to sound funny.