Preheat oven to 350. Arrange filets in a shallow baking pan. Brush filets on both sides with 1 T of the oil and season with pepper. Bake 8 to 10 minutes, until flaky but still moist.
Meanwhile, whisk together crème fraiche, remaining 4 T oil and lime zest. Top each filet with sauce.