First things first, preheat your oven to 350°F (175°C). Prepare an 8x4 inch loaf pan by greasing it and lining it with parchment paper, leaving some overhang on the long sides. This will make it much easier to lift the loaf out later.
In a medium-sized bowl, whisk together the whole wheat pastry flour, all-purpose flour, baking powder, baking soda, salt, and all the spices. Mixing them well now ensures the spices are evenly distributed throughout the batter for consistent flavor in every bite.
In a separate large bowl, beat the brown sugar, olive oil, vanilla extract, eggs, pumpkin purée, and almond milk until everything is well combined. This wet mixture is the foundation for the loaf's moisture and rich flavor. Don't worry if your eggs are cold.
Pour the dry ingredients into the bowl with the wet ingredients. Mix until the batter is just combined and smooth, being careful not to overmix. Pour the batter into your prepared loaf pan and bake for 50-60 minutes. It's done when a toothpick inserted into the center comes out clean.
Let the loaf cool in the pan for about 10 minutes before using the parchment paper overhang to lift it out. Transfer it to a wire rack to cool completely. This step is critical; if you ice the loaf while it's still warm, the icing will melt.
While the loaf cools, make the icing. In a small saucepan, combine the brown sugar, butter, and milk. Heat over medium-high heat, whisking until it comes to a full boil. Immediately remove it from the heat and whisk in the sifted powdered sugar until smooth.
Let the icing cool and thicken for 10-15 minutes, stirring occasionally. Once it's thick enough, slowly pour it over the completely cooled loaf. Allow the icing to set for a few minutes before slicing and enjoying your delicious creation.