Mix the egg yolks with the sugar and vanilla sugar in a metal. bowl. Place over a hot water bath. Gradually incorporate the milk, stirring often until thickened. Gradually stir in the cognac.
Cool the mixture, then pass through a fine strainer into a bottle. Seal tightly, and keep in the refrigerator until ready to use.