Combine diced peaches, granulated sugar, brown sugar, cinnamon, nutmeg, cornstarch, lemon juice, and vanilla extract in a medium saucepan over medium heat. Stir continuously until the mixture thickens and becomes syrupy, about 5-7 minutes.

Lay an egg roll wrapper in a diamond shape, spoon 2 tablespoons of the peach filling into the center, fold the bottom corner up, tuck in the sides, and roll tightly. Use water to seal the edges.

In a deep skillet, heat about 2 inches of vegetable oil to 350°F (175°C). Test with a piece of wrapper; it should sizzle and rise.

Carefully fry the egg rolls in batches for 2-3 minutes on each side until golden brown. Transfer to a paper towel-lined plate to drain excess oil.

Dust the warm egg rolls with powdered sugar (if desired) and serve immediately, optionally with vanilla ice cream.