Mr Trump tweeted that General Mattis "was a great help to me in getting allies and other countries to pay their share of military obligations".
It comes a day after the president's controversial announcement that all US troops would be withdrawn from Syria.
Mr Trump did not name a successor, but said one would be appointed shortly.