UsWeekly claims that Nicole Richie teasesJoel Madden, about his former relationship with Hilary Duff

Apparently, Joel ended stuff with Hilary, because she wouldn't give up the panty.

A source said, "She will ask if he's going to watch Lizzie McGuire DVDs or listen to Hilary's albums. Joel tells her she's being out of line."