Many Latin Americans, including some key leaders, harbor bitterness over the medicine the Washington-based Fund has prescribed for the region's economic woes since the 1960s.
In a part of the world long resent of being seen as America's back yard, the IMF and World Bank's loans and accompanying fiscal policy prescriptions -- commonly derided as the "Washington consensus" -- are often criticized as just another way for the US to maintain its dominance over the hemisphere.
"It's a long love story," joked Claudio Loser, a former head of the IMF's Latin America department.
"For quite some time, there has been a political rhetoric in Latin America presenting the IMF as a tool of a new kind of imperialism."
Cuba is the most extreme case: It slammed the door in the IMF's face in 1964, withdrawing from the Fund five years after Fidel Castro's revolution.
But other Latin American countries also blast the IMF.
Argentine leaders have blamed the IMF for provoking the country's 2001 default, which still dogs Latin America's third-largest economy.
"Where was the IMF and why hasn't it managed to avert a single crisis? Where was it when the world economy was forming not just financial bubbles, but financial hot-air balloons?" railed Argentine President Cristina Kirchner in 2013.
Venezuelan President Nicolas Maduro, whose country has refused to share its accounts with the IMF since 2004, said of the Fund last July: "First they suck your blood, then when you need oxygen they take away the oxygen supply."
Bolivian President Evo Morales resorted to a different metaphor.
"It's like giving your money to the wolf, or asking him to guard the sheep. The wolf isn't going to guard the sheep, he's going to devour them," he said in 2009.
The region's largest economy, Brazil -- which got an IMF bailout from 1998 to 2005 -- found a subtler way to snub the Fund: It teamed up with other emerging powers last year to launch a rival monetary fund to the Western-led institutions.
- 'Mistakes, arrogance' -
This is the first time the annual meetings have been held in Latin America since the 1967 edition in Rio de Janeiro, Brazil.
Loser acknowledged IMF missteps in the region, but defended its track record.
"Of course there have been some mistakes and maybe some arrogance, but (Latin American) countries are more careful about their public finances," he told AFP.
The region's cold shoulder may be less about the IMF's record than the left's surge to power in many Latin American countries in the 2000s and the region's changing relationship with the United States, the Fund's largest member.
"These countries became much more independent from the United States in general, more than they have been in the prior 150 years," said Mark Weisbrot of the Center for Economic and Policy Research, a progressive think tank.
Countries in the region have built up large reserves in recent years, making financial requests to the IMF less likely than before.
Just two Latin American countries, Mexico and Colombia, are currently beneficiaries of IMF programs, in the form of precautionary credit lines.
The current head of the IMF's Latin America department, Alejandro Werner, said this was because the region's finances are stronger than in the past, not a sign of "mistrust."
"The region has changed a lot in the last 20 years and the Fund is evolving as well. The interaction with the region is much less confrontational and much more constructive," he told AFP.
He said the IMF is confident it has "mutually beneficial" assistance to offer the region.
"It's in the best interest of the country because we can provide expertise and share best practices. And it's also in our best interest because we can therefore have much more depth in our analysis for the region," he said.