Kaoru Hasuike and his wife Yukiko were kidnapped in 1978. Yasushi Chimura and his wife Fukie were also taken, as was Hitomi Soga. In 2002, they were finally brought home following summit talks in Pyongyang.
In the years since, they have been frustrated with the lack of progress in bringing home the other 12 people kidnapped by North Korea.
The Chimuras and Hitomi Soga attended a rally in October calling for Japan's government to push for the return of the other abductees.
Pyongyang claims the other 12 are either dead or never entered the country. But Japanese officials say their explanation is contradictory and erroneous.
The North had previously presented to Japan what it claimed were the remains of Megumi Yokota. She was kidnapped at the age of 13 in 1977. But a DNA analysis in 2004 showed that the remains were not hers.
Time is running out for the abductees still in the North, and for their relatives.
In an October interview with NHK, Kaoru Hasuike, one of the five who returned, looked back on the last 15 years.
Hasuike and his then-girlfriend Yukiko were abducted by North Korean agents from a beach in Niigata Prefecture.
He says it took almost 10 years for the wounds to heal. But he has now earned a law degree and is an associate professor at a university in his home prefecture.
"Since coming back to Japan, I have been able to do many things I couldn't do in North Korea," he said. "I see the past 15 years as a period when I was able to build a new life."
The situation surrounding North Korea has changed over the years. Relatives of the other abductees are now worried that efforts to bring back their loved ones may be put on the back burner as tensions with North Korea rise.
But Hasuike thinks there is an opportunity, if Japan can take it.
"As North Korea gets pushed into a corner, it might try to start a dialogue," he says. "I think that's when Pyongyang will probably also consider its ties with Japan and South Korea, along with other issues."
Hasuike says, "The government should prepare to take advantage of such a moment to solve the abduction issue, and that means thinking of what it can offer North Korea in exchange."
He also says time is running out for the abductees still in North Korea, and for their relatives. He is urging Japan's leaders to make the issue a higher priority.
"The Japanese government shouldn't try to smooth over the problem with words," Hasuike says. "I think government officials need to tackle the issue by understanding the problem with their hearts and feeling it in their bones."
Families have spent years campaigning for the release of loved ones.
The relatives of Japanese citizens abducted by Pyongyang visited Washington in September. They asked for help in bringing their families home.
Takuya Yokota, a younger brother of Megumi Yokota, met with American officials and lawmakers, and spoke at a symposium.
"My sister was abducted on her way home from school at the young age of 13," he said. "Since then, she has been in detention for 40 years. When negotiating with North Korea, I believe that applying pressure will lead to a resolution."
Trump will meet with the families of abductees
The White House says US President Donald Trump will meet with the families of Japanese citizens abducted by North Korea during his visit to Japan that starts on November 5th.
He will begin with Japan, where he will hold talks with Prime Minister Shinzo Abe. Abe will also host the President in a meeting with the abductees' families.
Abe extended the invitation to Trump when they met in New York in September.
"When I requested that President Trump meet the families of the abductees, he agreed instantly," Abe said. "The president told me that it's a really terrible story, and he promised to do his utmost to rescue them."
Trump made a reference to abductee Megumi Yokota when he criticized North Korea during his first speech at the UN General Assembly in September.
Megumi's family has spent years campaigning for her release.
"I heard that President Trump wants to meet us. I have high hopes that the president will help the victims come home. The families simply want their children and their loved ones to return home quickly. They did nothing wrong, but were taken away from us," says Sakie Yokota, mother of Megumi.
She added it's the Japanese government's responsibility to resolve the issue.
Shigeo Iizuka, whose sister was abducted in 1978, said he hopes the meeting with President Trump will lead to the abductees' early repatriation.
Trump will be the third US president the families meet with, after George W. Bush in 2006 and Barack Obama in 2014.