During a panel discussion run by the Conservative Environment Network, several audience members expressed support for the government's drive towards net zero but they also had concerns about how local communities would be impacted by the projects.
Mr Rowley, who as a levelling up minister has responsibility for large infrastructure projects, told the audience: "The reality is we have all got to decide what our objectives are for the country."
He said he understood every area would have its individual concerns - and pointed to his own opposition of fracking projects in North-East Derbyshire.
But he added: "The thing that this party cannot become - this party cannot say no to everything."
"If you have strong views on things that's fine... but don't say no to everything.
"That's where the Liberal Democrats get in a silly place like they did last week with housing."