A 10-month old baby was among three dead after a 16-storey tower block was toppled by a 6.4-magnitude earthquake in southern Taiwan Saturday, with rescuers battling to free residents still trapped inside.

Officials said four buildings collapsed in the quake which struck the island in the early hours of the morning, but rescue efforts are centering on the tower block that tumbled onto its side.

Firefighters pulled survivors from the twisted concrete, trying to access apartments through windows and scaling the rubble with ladders.

The baby, a man and a woman were pulled dead from the block, officials said, with 29 residents hospitalized.

"These three people showed no signs of life before they were sent to the hospital," said Lin Kuan-cheng, spokesman for the National Fire Agency.

"The search and rescue work continues there, home by home."

More than 100 residents have now been freed from the rubble, with local media reporting the tower contained more than 200 homes.

Officials were unable to give an estimate of how many were still trapped as they scoured the building.

At least 30 people have been freed from another residential seven-story building with reports that over 200 have been rescued across the city and more than 100 sent to hospital.

One woman told how she had fought her way out of her home in one of the collapsed blocks.

"I used a hammer to break the door of my home which was twisted and locked, and managed to climb out," she told local channel SET TV, weeping as she spoke.

The shallow quake struck at a depth of 10 kilometers (six miles) around 2000 GMT Friday, according to the US Geological Survey, 39 kilometers northeast of Kaohsiung, the second-largest city on the island and an important port.

The quake was initially reported as having a magnitude of 6.7, but was downgraded to 6.4.

The Pacific Tsunami Warning Center said a destructive Pacific-wide tsunami was not expected.

Taiwan lies near the junction of two tectonic plates and is regularly hit by earthquakes.

A strong 6.3-magnitude quake that hit central Taiwan in June 2013 killed four people and caused widespread landslides.

A 7.6-magnitude quake struck the island in September 1999 and killed around 2,400 people. (AFP)