Originally, it was mamillary from the Latin spelling of the term mamillaris. The variant mammillary arose because of the close association with the related English term mammary (relating to the breast).
Both spellings are widely used. In my textbook Anatomy & Physiology, I chose the former spelling (the "single m"). Mamillary is used in ICD-9 medical coding and in many medical textbooks) because it more closely follows the Latin word from which it is derived, mamillaris. As many of my readers have noticed, the parsing of Latin roots is a theme in our textbook that is not found in most other anatomy and physiology textbooks, so it makes sense that we'd go in this direction, given a choice of two acceptable spellings.
This brings up the interesting and important phenomenon that applies to both spelling and pronunciation: there are sometimes several acceptable alternates but we often assume that the way we learned it (or the way it appears in the teaching resources we use) is the only correct option.
Check out Mamillary or Mammillary? What's in an “m”?
It turns out that our learning of scientific and medical terminology is never complete and so we should always double-check misspellings (and mispronunciations) in case they turn out to be correct (even if odd) alternates with which we are not yet familiar.
You may also be interested in a previous post on variations in pronunciation of anatomical terms:
Graphic from Anatomography
