Rinse swordfish steaks, pat dry and season with salt and pepper. Drizzle with lemon juice.
Heat olive oil in a large, non-stick pan over high heat. Sear swordfish steaks briefly on both sides.
Place swordfish steaks on a broiler pan or baking sheet. Finely chop olives and sprinkle over steaks along with rosemary. Bake swordfish in a preheated oven at 175°C (gas mark 2-3, fan: 160°C) (approximately 350°F), 6-8 minutes. Shortly before the end of the baking time spoon a dollop of crème fraîche onto each steak. Serve garnished with lemon wedges.