Whisk the egg yolks and lime zest together in a bowl until tinted light green, about 2 minutes.
Beat in milk, then juice and set aside at room temperature till it thickens.
Preheat oven to 325 degrees.
Mix graham cracker crumbs and sugar in another bowl.
Add butter and stir with a fork until well blended.
Pour mixture into a 9-inch pie pan and press over the bottom and up the sides of the pan.
Bake on the center rack for about 15 minutes until the crust is lightly brown.
Remove and let cool to room temperature.
Pour the lime filling into crust, spread evenly, and then bake for 15 minutes until the center sets.
Remove from oven and cool to room temperature.
Refrigerate for at least three hours until well-chilled and serve.