The Matildas were heading for their first loss to the Football Ferns since 1994, trailing 1-0 at the end of regulation time, despite dominating a contest where they had 38 shots to their opponents' three.
But Van Egmond got on the end of Ellie Carpenter's cross to slam home the equaliser, before Kerr pounced on a loose ball from a corner kick on the edge of the six-yard box to head the Matildas to a dramatic win.
"We kept loyal to what we're doing," Gustavsson said after the win. "They believed in it, they believed in the way we move the ball, the amount of chances we created and then you add the heart and the spirit to that tactic ... So I wasn't that surprised but I was obviously very happy."
In the first of their two-game series, a wonderstrike by Anna Green in the 32nd minute was the difference until the final stages as the visitors made the most of their limited opportunities. It came completely against the run of play, Green slicing across a half-volley from out wide on the left following Meikayla Moore's long ball, the ball rocketing into the top corner above the outstretched hands of Lydia Williams.
It was the Ferns' only opportunity of the first half as the Matildas hounded Green's left flank through Hayley Raso and Carpenter with returning Katrina Gorry excellent in midfield.
"Her quarterback ability as a six, her positioning is phenomenal," Gustavsson said of Gorry. "When you have a six that can combine and activate our central diamond more, that gives us just a new dimension."
As the second half unfolded, the Matildas continued to attack down the right but Ferns keeper Victoria Esson was there to deny the hosts time after time, in just her sixth appearance for the national side.
Coach Jitka Klimkova said she was proud of the fight her players showed, particularly Esson who frustrated the Matildas all evening. "I am just happy about the process, not really happy about the result," she said.
"But if you look at those 90 plus two minutes, it was unbelievable performance from the Ferns and I'm a very proud coach. "She [Esson] kept us in the game many times and that's what we are expecting from our goalkeeper."
Australia enjoyed 72% of the possession, completed more than double the amount of passes and had 38 shots (16 on goal) but lacked clinical finishing.
Their shot tally was the highest ever under Gustavsson in his tenure as coach, with just under half of those shots coming in the opening term. They had three close chances in the first 10 minutes, the best falling to experienced defender Clare Polkinghorne, whose glancing header was exceptionally saved off the line by Esson.
Kerr and Kyah Simon had chances late in the opening half, but the captain scuffed her shot wide from close range and Simon's attempt didn't trouble the keeper.