Ed Razek, who was discussing diversity in an interview with Vogue magazine, has since said his remarks were "insensitive".
The US lingerie brand's 2018 showcase took place in New York on 8 November, and featured models including Bella Hadid and Kendall Jenner.
Mr Razek, who was part of the casting team, told Vogue he had considered putting transgender and plus-sized models in the show.
He added that he didn't because the company "did not market to the whole world".
Mr Razek went on to say: "Does the brand think about diversity? Yes.
"Do we offer larger sizes? Yes."
"It's a 42-minute entertainment special.
"That's what it is.
"It is the only one of its kind in the world."
His comments come as Victoria's Secret has experienced falling sales since 2016.
After the remarks attracted a backlash on social media, the brand tweeted an apology from Mr Razek.
It read: "My remark regarding the inclusion of transgender models in the Victoria's Secret Fashion Show came across as insensitive.
"To be clear, we absolutely would cast a transgender model for the show.
"We've had transgender models come to castings... And like many others, they didn't make it... But it was never about gender."