A late winner by sub Chris Dodd clinched all three points for City, after Brian Cameron's first half opener was cancelled out by a Clyde strike from former international David Goodwillie.
Goodwillie also missed a penalty kick in a close encounter at Borough Briggs.
Goalkeeper Marc Waters, recalled by manager Gavin Price in one of five changes made from the Peterhead game, had two reasons to be thrilled by Elgin's success.
He said responding to the midweek mauling in Buchan was vital, and he was pleased to be back in the starting line-up.
New signing Iain Ross was injured after conceding the first two goals at Peterhead, and Waters came on to lose the next five as part of an Elgin collapse.
"It was massive to get a reaction especially after losing the goals that we did on Tuesday," he said.
"We've gone back out there and we've done the business. It's three points and this game, just like the Peterhead game has been put to bed, and we go again on Saturday at home to Cowdenbeath."
Elgin's win was also a first official game for new assistant manager Keith Gibson, unveiled as Price's number 2 at the weekend.
Gibson (36) recently bossed Broughty Athletic in the juniors and enjoyed a successful playing career with Arbroath, Montrose, Forfar, East Fife and Peterhead.