The rights group said most of the incidents took place on AMISOM bases in the Somali capital, Mogadishu, where women come for medical care and to beg for food.
"Where this case is particularly shocking is the direct use of humanitarian assistance to lure these women in," said Laetitia Bader, one of the report's authors.
"These were displaced women coming in to get medical assistance and it's when they are in the outpatient clinics that they get approached by a Somali intermediary who says: 'Why don't you come back to the base? We'll give you medication,'" Bader told Thomson Reuters Foundation.
One woman, known as Ayanna, told HRW she was gang raped at gunpoint by six Burundian soldiers after going to their outpatient clinic to get medicine for her sick baby.
One of the three other women who were also raped at the same time was badly hurt.
"We carried the injured woman home," she told HRW. "Three of us walked out of the base carrying her… She couldn't stand."
The soldiers threw packets of porridge, cookies and $5 (£3.10) at the women as they left, she said.
Rape is rarely punished in Somalia, particularly of vulnerable women living in overcrowded Mogadishu camps housing some 350,000 people displaced by war and famine.
HRW also interviewed 14 displaced women and girls selling sex to AMISOM soldiers for around $5 a day. Sexual exploitation – the abuse of power or trust for sexual purposes – is in violation of their code of conduct.
The sex trade on AMISOM bases appears "routine and organised", HRW said.
Women who visited the bases regularly were not checked on their way in and HRW was told that some lived there, ostensibly employed as interpreters.
"Somali women having paid sex with soldiers have been able to obtain AMISOM badges allowing them easy access in and out of what should be highly secure military zones," the report said.
The African Union force deployed to Somalia in 2007 to help restore order and defeat the Islamist militant group al Shabaab. It is credited with pushing al Shabaab out of many towns in south-central Somalia, strengthening the hold of the two-year-old Somali federal government.
Al Shabaab confirmed on Saturday that its leader, Ahmed Godane, had been killed in a U.S. air strike last week.
AMISOM's 22,000 troops come from Uganda, Burundi, Kenya, Ethiopia, Sierra Leone and Djibouti. They are immune from prosecution by the Somali government, with responsibility falling on their own governments.
Only two out of the 21 women and girls interviewed filed a complaint, for fear of reprisals, HRW said, while those having sex for money did not want to lose their main source of income.
"We take all allegations against AU Troops in Somalia very seriously, especially allegations of sexual abuse," AMISOM’s spokesman Eloi Yao told Thomson Reuters Foundation in an email.
"Any AMISOM personnel found guilty of any such crimes will be dealt with appropriately."