In a large skillet, cook bacon over medium heat until crisp. Remove to paper towels to drain; crumble bacon. Discard drippings, reserving 1 tablespoon for sauce. In a large bowl, whisk milk, eggs, 1/2 cup brown sugar, pumpkin, maple syrup and 2 teaspoons pie spice until blended.
Place half the bread cubes in a greased 2-qt. baking dish. Top with half the crumbled bacon and half the egg mixture. Repeat layers. Refrigerate, covered, overnight.
Preheat oven to 350°. Remove casserole from refrigerator while oven heats. Combine oats, pecans, flour, and the remaining 3 tablespoons brown sugar and 1/2 teaspoon pie spice. Cut in cold butter until mixture resembles coarse crumbs. Sprinkle over casserole. Bake, uncovered, until a knife inserted near the center comes out clean, 40-45 minutes. Let stand 5-10 minutes before serving. Meanwhile, for sauce, in a small saucepan, melt butter and reserved bacon grease over medium heat. Stir in brown sugar and maple syrup. Add milk and pumpkin; heat through. Serve with French toast.
