US Olympic swimmers in Rio de Janeiro invented a story about a robbery in an effort to disguise a dispute over a damaged petrol station door, police sources have told the BBC.
One of the athletes broke the door to the bathroom and a row ensued when attendants asked the Americans to pay for the damage, they said.
After security guards were called in, the Americans reportedly paid and left.
Three of the swimmers remain in Brazil and are due to be questioned by police.
The fourth, gold medallist Ryan Lochte, returned to the US on Monday.
Before it emerged that Mr Lochte had left Brazil, a judge ordered that the four have their passports confiscated pending further police questioning, amid reports of inconsistencies in the men's accounts of the alleged robbery.
Gunnar Bentz and Jack Conger were taken off a US-bound plane at Rio de Janeiro airport on Wednesday night. Team-mate James Feigen had remained in Brazil.
Mr Lochte admitted on Wednesday to some inaccuracies in his original account of being robbed at gunpoint in the early hours of Sunday, but vehemently denied making the story up.
"I wouldn't make up a story like this nor would the others - as a matter of fact we all feel it makes us look bad," he told US TV network NBC.
The three swimmers remaining in Brazil - who have been ordered by a judge to surrender their passports - are expected to speak to investigators on Thursday.
Mr Lochte is one of the most successful swimmers in history, with 12 Olympic medals, and he once had his own reality television show in the US. In Rio, he swam in two events, winning gold in the 4x200m freestyle relay.
Mr Feigen won gold in the 4x100m freestyle relay.
Mr Bentz competed in the 4x200m preliminaries, but not the final. He still received a gold medal after the US team's win.
What do we know of the alleged robbery?
Accounts of what happened to the swimmers have been confusing from the beginning, and it appears they did not immediately alert either the Brazilian police or the Olympic authorities.
News of the incident emerged after Mr Lochte's mother told US media about it.
Mr Lochte himself gave an initial account of the events to NBC on Sunday, saying he and the other swimmers had been in a taxi returning from a club in the early hours when they were pulled over by men wearing police badges.
He said they had pulled a gun and told the swimmers to get on the ground. "I refused... and then the guy pulled out his gun, he cocked it, put it to my forehead..."
Mr Lochte has since slightly altered his account, telling NBC on Wednesday that the taxi had not been asked to pull over - they had been robbed while making a stop at a petrol station - and he said the gun had not been pointed directly at his forehead.
He called the inconsistencies a "traumatic mischaracterisation" caused by the stress of the incident.
What has been the reaction to the arrests?
US Olympic Committee (USOC) spokesman Patrick Sandusky said in a statement: "The three US Olympic swimmers [Gunnar Bentz, Jack Conger and James Feigen] are co-operating with authorities and in the process of scheduling a time and place today [Thursday] to provide further statements to the Brazilian authorities.
"All are represented by counsel and being appropriately supported by the USOC and the US consulate in Rio."
Earlier he said that Mr Bentz and Mr Conger had been "released by local authorities with the understanding that they would continue their discussions about the incident on Thursday... James Feigen is also communicating with local authorities and intends to make further statements regarding the incident on Thursday as well."
Mr Lochte's lawyer told the BBC he had returned to the US two days ago before the controversy broke.
"He was never asked to remain for further investigation or for any other purpose after he met with Brazilian authorities after he gave a statement," said Jeffrey Ostrow.
What have Brazilian authorities said?
The judge investigating the case found further inconsistencies and ordered police on Wednesday to seize the swimmers' passports to prevent them leaving Brazil.
These inconsistencies reportedly include the time the swimmers left the party, and how many alleged gunmen they were confronted by.
Police say they have not been able to track down the driver who the swimmers say drove them back to the village.
And CCTV footage of their return to the athletes' village appears to show the swimmers laughing and joking, and handing over their wallets, phones and accreditation, as they go through the security screens. The judge said they had not show signs of being affected by a robbery.