The House of Rolf is a line of nobility that Rolf supposedly descends from. He claims that it is well-known and has a storied history, but throughout the game, it is strongly implied that Rolf has fabricated his noble lineage and is in fact a former bandit. Specifically, Deshavi recalls him as being a member of a group called the "Brethren of the Woods", and Rolf is very familiar with the Woods of Ehlerdah which are bandit infested.
Nevertheless, granting Rolf a fief is not controversial, so it appears that many vassals more or less believe in his claim of noble birth or he didn't become a bandit until after the loss of his heritage.
Also, some bandits don't attack Rolf since he was a bandit.