In 2010/11 he mastered swing bowling so there's no disagreement there. However it was a different story for most of his career.
a) From his debut in Feb 1998 until that 2004 Boxing Day Test, Yousuf averaged
30.70 against Australia, South Africa, England, India and Sri Lanka, all of whom possessed one quality swing bowler, and were our hardest opponents in that era. In contrast Inzamam averaged 46.05 against them in that timeframe hence why many Pakistan fans
at the time accused Yousuf of hiding behind Inzamam and going missing against the top teams.
b) Against pace bowling overall in that time he averaged
33.87 (opposed to 45.66 against spin). Like
@ElRaja stated one of the reasons was lazy footwork by not getting properly onto the front foot. India exposed this time and again in 2004 - their pacers accounted for 8 of Yousuf's 10 dismissals across the Tests and ODIs ! Pakistan and India fans used to joke that Yousuf was Irfan Pathan's bunny.
However post-MCG ton he was magnificent. Against the Big 5 he averaged
53.09 and his overall pace average rose to
49.00. Had he played like that his entire career he'd be the greatest Pakistani batsman without argument. But he didn't hence why it's important to balance nostalgia with what fans at the time saw and what the numbers show. He wouldn't make my Mt Rushmore of Pakistani batsmen (occupied by Miandad, Inzamam, Younis and Saeed) but he's the next man on. Just my opinion, people can disagree.