The name Taman sari means "fragrant garden", recalling the sweet fragrance ot the many flowers and fruit trees which once filled the gardens of the complex. The name given to the site by the Dutch colonists was "Water Gardens", reflecting the imposing two-story mansion, sunken bathing pools, secluded meditation chambers, underground passages and multitude of gardens that formerly stood in the center of a man-made lake.
Access to the retreat was gained by way of underwater tunnels with small towers protruding above the water to provide both ligth and ventilation. The whole complex is believed to have taken 11 years to complete and was abandoned shortly after the Sultan died. Its evacuation may have bee due, in part, to the problems associated with maintaining the complex hydraulic works. Neglect, the effects of damage from the 19th century Java Wars and earthquakes all took their toll, causing the structures to fall into ruin. While the central pools have been rebuilt, restoration has not extended to other parts of the structures.