Rolf supposedly descends from the House of Rolf. He claims that it is well-known and has a storied history, but throughout the game it is strongly implied that Rolf has fabricated his noble lineage and is in fact a former bandit. If you make him a vassal, he will train Sarranid troops, with him being an improverished noble.
Nevertheless, granting Rolf a fief is not controversial, so it appears that many vassals more or less believe in his claim of noble birth or he didn't become a bandit until after the loss of his heritage.
Also some bandits don't attack Rolf since he was a bandit.