It is said that the Dutch pirates who looted this temple during the 17th century had broken the granite statues and carried the Panchaloka idol of Arumuga Peruman to their ship. Suddenly, there was heavy storm and rain. Filled with fear of this sudden development, they tied a rope to the idol and dropped it in the sea. The storm was said to have stopped immediately. On finding the idol of Arumugha Nayanar missing from the temple, the devotees grew worried. Lord Muruga appeared in the dream of Vadamalaiyappa Pillai, who was an ardent devotee of the Lord Murugan and showed the place in the sea where he was lying. Immediately, Vadamalaiyappa Pillai rushed to the spot with some people and saw a lemon was floating on the waters. Garuda, the holy eagle was flying over the spot. They dived into the waters and recovered the idol. This incident took place on Friday, the 29th day of the Thai month in Kollam year 829, that is, in 1653 A.D. This fact is taken from the inscriptions.