L.B.
Hi J.,
I'm a teacher and can tell you... they all do this! They not only pull at their underwear but also put hands down pants to scratch too. This can go all the way through 3rd grade so she isn't alone.
While it might be driving you crazy, the more you bring attention and 'pick at her' about it, the more she will do this behavior. Your best course of action? Model appropriate behavior and ignore what she is doing.
She will eventually stop the behavior as socially in front of her peers it will become inappropriate to herself.