United have failed to win any of the first four matches at Old Trafford in the league as manager Ole Gunnar Solskjaer lost his 100th game in charge.
Arsenal also ended their five-year wait for a Premier League away win against "Big Six" opposition. They had not enjoyed such a win in 29 games since their victory at Manchester City in January 2015.
Arsenal were by far the better team in the first half and almost took the lead after 39 minutes when Willian's curling shot hit the post.
An unmarked Bukayo Saka had a great chance to put the visitors ahead on the stroke of half-time but his header went over the bar.
Both teams missed good chances after the break when Aubameyang's curling shot missed the target while Harry Maguire's header went wide.
After 69 minutes, Pogba didn't track Bellerin's run and brought down the defender in the box. Arsenal were awarded the penalty and Aubameyang converted from the spot.
United almost equalised late on after Donny Van de Beek's cross was deflected off the post but the visitors came away with a much deserved victory.
"We're creating expectation. We want to beat [the other 'Big Six' teams] all the time and compete for the league with them. We will evolve and get better at other things so we can go to any ground and be competitive which is something already positive," Arsenal manager Mikel Arteta said.
Solskjaer was disappointed with his team's display after the excellent 5-0 win over RB Leipzig in the Champions League and did not spare Pogba from the criticism.
"He needs to stay on his feet. Paul knows it's a soft penalty to give away," said the Norwegian.
"We were too sloppy, too many passes went astray and the rotation was not there. Too many things that can go wrong in a game did for us today. Today we didn't get the response we hoped for.
"In the first half, we never turned up. The intensity, the tempo wasn't there, the shape -- they were better than us and created a couple of half-decent chances.
"In the second half, they didn't look like they were going to score, apart from the penalty. So we played better in the second half but still we have to turn up with more intensity," he added.
Pogba said that his error was the key factor in United's latest Old Trafford setback.
"I felt like I touched him [Bellerin] a bit, I knew I was in the box," Pogba admitted. "I shouldn't have given a penalty away like that.
"Maybe I was a bit out of breath, it made me do this stupid mistake."
United captain Maguire laid the blame on the entire team's performance.
"It's hard to put your finger on it," United skipper Maguire said. "It was disappointing from us. It was a cagey game, a game of few chances.
"We needed to do more, we needed to create more. It's not good enough, we've got to go work hard tomorrow at training."