The thing is Sri Lankans have generally had good spinners, which the Pakistani batsmen haven't been good at playing. Pakistan's strength has been fast bowling (up until the last decade) which is negated in Sri Lanka due to the pitch and the kookaburra ball.
The reason Pakistan do better against India is because our pitches and the SG ball has offered something for the fast bowlers in terms of conventional and reverse swing. If it was a straight shoot out between batting vs spin, India would have generally won as we have generally had a better spin attack (not just one spinner, but a full attack) than Pakistan, and of course better batting.