The match at the King Power Stadium was tightly contested, with Leeds looking the more dominant side in the second half, but Leicester winger Harvey Barnes struck home on 66 minutes to edge his team to victory.
Marsch, 48, of the United States, was appointed at Elland Road on Monday, signing a contract until June 2025, as a replacement for Marcelo Bielsa.
Saturday's defeat means Leeds remain in 16th place in the Premier League, just two points ahead of the relegation zone.
"Even going into the match I felt the performance was more important than the points," Marsch told the BBC.
"It's our first match together, we've had four days on the [training] pitch and I'm pleased with how they controlled the match in all phases of the game, including set pieces. They've made so much progress in a short space of a time."
Leicester striker Jamie Vardy tormented Leeds' high line early on, but the visitors managed to keep a clean sheet in the opening 45 minutes for the first time since January.
Leeds also had their chances and started the second half brightly with Leicester goalkeeper Kasper Schmeichel keeping them at bay with some fine saves.
But Leicester finally broke the deadlock against the run of play midway through the second half when Barnes played an intricate one-two pass with Kelechi Iheanacho before sliding the ball past Illan Meslier in Leeds' goal.
"It's always a tough game against Leeds. They had a new manager as well so we didn't know how they were going to set up, but they still have the same intensity, it's still end to and," Barnes told BT Sport.
"A tough afternoon but delighted with the three points... You don't want to make it end-to-end against Leeds. You want to keep the ball and we didn't do that for a lot of the afternoon, we made it a basketball game."
Information from Reuters contributed to this report.