Tory Burch says she didn't want to compromise on family time even as her company demanded more of her attention. ANGELA WEISS / AFP Tory Burch says she would "never trade off" being a good mom for her career. If anything, her social life suffered — and she gave up sleep, she said. "But I will say, being a mom is the joy of my life," the fashion designer said. Tory Burch , 59, says her career success didn't come at the expense of being a mom — but it did cost her a social life. Speaking on Tuesday's episode of the " Aspire with Emma Grede " podcast, the fashion designer said she refused to compromise on family time, even as her company demanded more of her. "I mean, one thing I would never trade off was being a good mom," Burch told podcast host Emma Grede . "I never thought I could start a business if that were to have to happen." Burch shares three sons with her ex-husband, J. Christopher Burch, with whom she co-founded her namesake label in 2004. The pair divorced in 2006.…