First, steep the Earl Grey tea and lavender in warm milk for about 15 minutes. This infuses the liquid with fragrant flavor. Mix the dry ingredients separately, then combine with wet ingredients to form a smooth batter. Divide the batter evenly into the prepared pans and bake.
Once baked, let the cake layers cool in their pans for 10 minutes. Carefully loosen the edges with a spatula and transfer to wire racks. Use a serrated knife to level the tops so stacking is easier.
While the cakes cool, prepare a Swiss meringue buttercream base. Add lavender-infused milk for flavor. Whip until light and fluffy, adjusting sweetness to your preference.
Place the first cake layer on a cake board or plate and secure it with a dab of frosting. Spread an even layer of buttercream, then repeat with remaining layers. Top layer can be placed upside down to simplify frosting.
Apply a thin layer of frosting to trap crumbs, then smooth with an offset spatula or bench scraper. Chill in the fridge for 30 minutes or freezer for 10 minutes until firm.
Apply a second, thicker layer of frosting and smooth it for a clean finish. Decorate with buttercream flowers using piping bags and tips for an elegant touch.