The meeting was the highest level sit-down since relations between the two countries soured last year after a series of rulings by South Korea's Supreme Court. The court ordered Japanese firms to compensate people who say they were forced to work during World War Two.
The Japanese government maintains the issue of the right to claim compensation was settled by a 1965 agreement. It says the recent rulings violate international law and is demanding that South Korea correct the situation.
In the 20-minute meeting, the two leaders agreed on the importance of exchanges at various levels, such as between lawmakers or between members of the public.
Abe stressed that the cooperation with South Korea as well as trilateral relations with the United States are important over matters including North Korea. And he again called on South Korea to correct the wartime labor issue.
Japan's deputy chief cabinet secretary, Naoki Okada, was at the meeting and told reporters that the mood was calm.
On the other hand, South Korea had high expectations to mend the relations. Prime Minister Lee handed Abe a letter from South Korean President Moon Jae-in. The South Korean government says the letter congratulates Japan on the start of the new Reiwa era and expresses hope for development in bilateral ties.
Lee told reporters on the way back to Seoul that he had expressed hope of an Abe-Moon meeting, though he didn't suggest a time or place.
Seoul is concerned that the worsening relations will affect the country's sluggish economy.
Seoul has been especially keen to discuss Tokyo's tightening of export controls on three high-tech materials. Last month, South Korea launched a WTO dispute settlement procedure over Japan's decision. It claims the controls are politically motivated, discriminatory, and in violation of WTO rules.
But he took a more conciliatory tone at a meeting with Japanese business people, Prime Minister Lee said it was important for industries in both countries to work together.
"The economies of our two countries fit together like clockwork, and we cannot bring them to a halt. We have to ensure that they keep moving," Lee said.
The next move
Abe and Moon are expected to attend international conferences in the coming days, including the East Asia Summit in Thailand and an APEC summit in Chile next month.
The time is of the essence for the two countries because there are events in the near future that could make relations worse.
South Korea has announced it will terminate an intelligence-sharing agreement between Japan and South Korea, known as GSOMIA. The deal will expire in mid-November. Japan is urging South Korea to maintain the agreement at a time of repeated North Korean ballistic missiles tests.
And the plaintiffs in the wartime labor issue are engaged in court procedures to sell assets seized from Japanese firms. Those procedures are expected to be completed by the end of the year.
Moon suffered a big blow recently as a justice minister he appointed quickly became embroiled in corruption allegations and stepped down after just a month. His approval ratings are at an all-time low. And so as he scrambles to stabilize his administration, he is unlikely to make the kind of big decision needed to resolve these issues.
At the very least, the president needs to manage the situation so it does not get any worse.