You're ultimately grumbling against God, His goodness, His wisdom, His character. We're actually telling God to His face that He doesn't have the best for me. He doesn't know what He's doing, if you really think about it. We're, you know, we don't really believe that God is good and that He's looking out for us. When you think about it, grumbling, whining, complaining are all ultimately the harsh response to God, not to our circumstances.
It shows a lack of faith in God and His kingdom and focuses on us getting what we want, how we want it, when we want it. Because the grumbling is a very selfish, in my mind, sin.