In a large saucepan, sprinkle gelatin over milk; let stand 1 minute. Heat and stir over low heat until gelatin is completely dissolved. Whisk in pumpkin, brown sugar, pie spice and salt; cook and stir over medium heat until thickened and bubbly. Reduce heat to low; cook and stir 2 minutes. Remove from heat.
In a small bowl, whisk a small amount of hot mixture into egg yolks; return all to pan, whisking constantly. Bring to a gentle boil; cook and stir 2 minutes. Immediately transfer to a clean bowl; cool 30 minutes. Press plastic wrap onto surface of mixture; refrigerate 30 minutes.
In a small bowl, beat cream until it begins to thicken. Add sugar; beat until soft peaks form. Fold into pumpkin mixture. Refrigerate, covered, until cold.
Spread scant tablespoonfuls of pumpkin mixture over wafers; arrange in a 8x8-in. baking dish, closely overlapping wafers. Repeat with remaining pumpkin mixture and wafers. Sprinkle top with wafer crumbs. Refrigerate 8 hours or overnight.