The Federal Reserve said Citi, SunTrust, Ally Financial and MetLife failed to show they have enough capital to survive another serious downturn.
Citigroup is the third-largest US bank. The majority of the 19 tested passed.
All those tested are in a much stronger position than they were after the 2008 financial crisis, the Fed added.
The Fed tested the banks' ability to withstand a similar crisis that triggered a rise in unemployment to 13%, a 50% fall in share prices and a 21% drop in house prices.
Their strength is assessed by its Tier 1 capital ratio - assets held in reserve as a buffer against financial troubles.
The regulator said Citigroup had a Tier 1 capital ratio of 4.9%, with Ally Financial and SunTrust at 4.4% and 4.8% respectively.
MetLife had 6%, although it was measured slightly differently to the others.
"One problem (for Citi) is too much exposure in Europe," said Chris Lowe from FTN Financial in New York.
"It can shift its global focus, but obviously that's going to take time," he added.
The banks with the best ratio were Bank of New York Mellon, at 13.1%, State Street on 12.5% and American Express at 10.8%.
"I think the tests are quite significant," said Matthew Bishop, US business editor of the Economist.
"Even the banks that failed like Citigroup failed largely because they were planning to hand too
money back to investors... had they not handed any money back to investors, they probably would have passed."
The Federal Reserve has been conducting stress tests since 2009, but these were the first where results were made public.
The scenario was tougher than in previous years as the Fed wants to be sure US banks will be fit enough to meet new banking rules, known as Basel III, which come into force in 2014.
The results had an immediate impact on share prices, as a pass means the banks can increase shareholder dividends.
The Fed was expected to release the results of the tests on Thursday, but brought them forward after JP Morgan said on Tuesday it had passed the test.
JP Morgan's shares rose 7% after the news, while Citibank's fell 4%.
The bank had been expected to meet requirements and pay a dividend to shareholders.
The Fed wants banks to show they could not only withstand a future crisis but could keep lending, unlike the freeze of the credit crunch of 2008, which deepened economic pain.
But experts remain sceptical about how effective these tests are in preparing banks for future crises.
"My bigger worry is that the Federal Reserve and other central banks completely failed to see the last banking crisis... and I don't necessarily feel comforted that they've guessed the worst case scenario accurately," Matthew Bishop of the Economist told the BBC.
"I'm not convinced that they are any better at predicting the future than they were before."