The band's 16th studio LP, their first to be released via the former Caroline International, entered the Official Charts Company midweeks at No.2 with 9,817 sales.
Chancellor said he was delighted to finally be working with the "UK legends".
"I've been a fan of them since, God, the early '90s," laughed Chancellor. "We'd tried to work with them in the past actually and never quite managed to get the deal to work ,or things to align. But this time there was something magical about it."
"We've been working with their management across a couple of other projects and that relationship had been going really well, so we instigated the conversation. Then it was just about getting to meet the band," explained Chancellor.
"They wanted to talk about creative ideas, what to do and who to work with, and it felt like they had the bit between their teeth, so the whole prospect was really exciting. And we heard some demos and they sounded really promising.
"They have a drive to better themselves every single time and that's quite a rarity these days, especially when you've got a decent fanbase and you could quite easily just knock the old big tunes out and then take a bow. But they've still got the hunger."
We're very bullish, we've been reaching for the stars
Jim Chancellor, Virgin Music Label & Artist Services
All The Colours Of You was recorded, in part, before the onset of the pandemic and was produced by Grammy award-winning Jacknife Lee.
"Massive props to Jacknife Lee for for producing the record in very difficult circumstances," added Chancellor. "He pretty much made the record in isolation. Tim lived around the corner, but it was quite an amazing feat. But that's why we went to him - because he's a genius."
Chancellor said he was optimistic about James' chances of strong first-week showing.
"I'd love it to go high in the charts, I think they deserve that," he said. "It's obviously been a tricky period for everybody, but we're moving into this next phase now and everybody's feeling a bit more positive, so I'd love to see them in the Top 5.
"We're very bullish, we've been reaching for the stars in going to places the band had never been, which is Radio 2, from a radio perspective, and looking to grow their socials and streaming [numbers], everything really. One goal was to try and get them a bit more fixed on the radio in the UK, and we've ticked those boxes."
James, who celebrate their 40th anniversary in 2022, have already sold in excess of 60,000 tickets for their 2021 UK arena tour.
"They've always been a brilliant, brilliant live band," said Chancellor. "But there's normally a moment in a band's career where it all comes together beautifully, then you get that bigger explosion and off they go again. That's where it feels like it is for James - there's a renewed vigour, a renewed excitement in them. And that's what we saw in the demos."