"Japan's Top Inventions."

The behind-the-scenes tales of hit products and creations from Japan.

This is "Japan's Top Inventions."

On today's show: A voice known around the world.

We share the story behind singing synthesizers.

Later on the show...

Aha. Like looking at art. Beautiful.

Small silver beads.

They're actually important for making smartphones.

Just what are they? Stay tuned to find out.

Hello, welcome to "Japan's Top Inventions."

I'm your host, Jason Danielson.

In the first half of the show, we take you "Behind the Creation."

Today's topic is this: singing synthesizers.

Input the melody and lyrics into the program, and...

It sings!

Creators using this software to make music have been sharing songs online for some 15 years now.

You can even buy CD compilations of popular tracks.

People all around the world use the program.

It was developed by a major Japanese musical instrument manufacturer.

With their help, a virtual singing sensation was born.

A packed concert hall.

And on stage...

A virtual singer.

She has a special voice.

It's made with this computer software, which uses samples from a female voice actress.

The character design reflects the singing voice.

Our story begins back in the year 2000.

Electronic instruments were popular in Japan.

Many creators made music at home.

Meanwhile, in Shizuoka Prefecture, at a major instrument manufacturer...

Discussions were being held for a new project.

Synthesizing human singing voices.

The company already had electronic versions of pianos, drums, and wind instruments.

But they hadn't succeeded with human voices yet.

They'd attempted it before, but the result never sounded natural.

Development was left unfinished.

However, people at the company believed that singing synthesis would become a vital part of music production.

And so, a new project team was formed.

The team lead was chosen to be engineer Kenmochi Hideki.

Here's how he remembers things.

At the time, most instruments other than the human voice had been synthesized.

Reproducing a singing voice with a computer is extremely difficult.

We thought it would be best to restart development of a singing synthesizer from scratch.

And so, development of the singing synthesizer software began.

The team got to work straight away.

How to synthesize a singing voice?

First, the team listened to samples which the company had made previously.

Actual audio sample

They sounded very unnatural.

It was robotic, and you couldn't make out the lyrics.

A very electronic sound.

Surely this could be improved. Make it sound more human.

How could they make the voice more natural?

The team spent day after day brainstorming ideas.

But there was no clear solution.

The project was stalling.

A certain sentiment started growing at the company:

"Should we even be trying to do this?"

Does anyone actually want this?

Like the very act of synthesizing a singing voice was wrong.

There might have been some kind of visceral reaction like that at the time.

It was then that the project gained a powerful ally in Spain.

Pompeu Fabra University in Barcelona.

The team reached a joint research agreement with Professor Xavier Serra,

who specializes in music information retrieval.

Our company has longstanding ties with university professors

who specialize in sound and music.

Because of these ties, we started joint research on singing voice synthesis.

Kenmochi boarded a flight to Spain, and began working with researchers there.

It's extremely difficult to synthesize a singing voice.

When development started, people at the company doubted whether there was even any point in doing it.

But the development team was motivated,

and they had a plan for reproducing a natural singing voice.

Over in Spain, Kenmochi continued development, consulting with the local researchers.

They came up with a certain concept.

"Individual sounds are connected while singing."

"What if we focus on these connecting parts?"

For example "a-sa", which means morning in Japanese.

The "a-" and "sa" are made up of 5 parts.

From silence to the "a-".

The transition to the "sa".

Then the "sa", another transition, and silence.

The idea was to make sure these connecting parts were smooth.

I think our ears and brains kind of ignore the connecting parts of sounds,

or when a sound is just extended. That gets overlooked.

But when there's a change, those subtleties are processed in our brains.

This was the base of our concept, so we focused on the transitions.

Kenmochi returned to Japan.

He began working on gathering voice samples.

The plan was to record professional singers at the company.

But then he realized something.

What they needed were parts of sounds for the synthesizer.

It wouldn't work if they just recorded regular singing.

If you sing a song normally with lyrics, there are various nuances that get included.

It's unavoidable.

But those extra elements aren't needed. It has to be as neutral as possible.

Kenmochi had an idea for this.

He created made-up phrases from which he could extract the parts he needed,

like "a-da-a-da-i-da" or "ka-da-ka-da-ki-da."

He came up with over 500 of these.

It was time for recording.

Actual audio sample

Countless takes with nonsensical words.

The recording process was long and exhausting.

It was totally different from a typical vocal recording.

I could see the recording engineers had these kinds of puzzled looks on their faces.

The singer who was doing the phrases also looked confused.

They were able to record all 500 plus samples that were required.

But how would it sound when edited and arranged?

It was time for a test.

Actual audio sample

The pitch and quality were uneven, and it sounded unnatural.

Could they make the transitions between sounds smoother?

Kenmochi kept working at his computer to improve the programming which governed the transitions.

He continued to test how to make them smoother.

I made minor adjustments to the programming, and tested the effect it had on the synthesizer.

Certain changes sounded better.

I would change the code, run it, listen, and repeat.

After many rounds of testing...

Actual audio sample

The transitions became natural.

Then, one year later,

the team had turned their attention to the following song by a popular singer.

"LOVE LETTER"
by Makihara Noriyuki

Kenmochi was doing his best to reproduce this musical phrase.

One day...

Actual audio sample

The synthesizer was finally starting to come together.

The singing synthesizer software.

There was one more thing which the team had to figure out:

an interface for inputting the melody and lyrics.

It wasn't just about connecting sounds.

The synthesizer had to sing lyrics with a melody.

They needed a way to sequence both notes and words.

The developers began designing the interface.

They were looking to incorporate a piano keyboard.

This was the actual interface they came up with.

A keyboard is displayed on the left side of the screen.

With this, you can set the notes and durations.

You can also match lyrics to each sound.

And so, in February of 2003, 3 years after development started,

the singing synthesizer was complete.

The product reveal was to be done at a trade show in Germany.

The team brought along from Japan their software and a computer.

Kenmochi handed out flyers for the announcement in advance.

But the reaction?

I was telling them the computer sings! The computer can sing!

But they'd tell me that they could sing themselves, so they didn't need something like that.

About half of the people didn't get it.

Kenmochi did his best to attract as many people as he could.

With all eyes on him, he fired up the software.

He began by entering the melody and lyrics of a famous hymn.

It was "Amazing Grace".

A song basically everyone knows, with a slow tempo.

It was perfect for demonstrating the synthesizer.

If the result didn't sound natural to the audience, it would have all been for nothing.

Nervously, Kenmochi pressed play.

Actual audio sample

Someone in the audience spoke up: "That's pretty good!"

Some said it was futuristic, like a new instrument.

Some clapped.

We started off just trying to reproduce a human singing voice.

But there were people that thought of it as a new musical instrument.

That surprised me.

I really felt like it had been worth the effort.

That was how the value of singing synthesizers were first recognized.

Later, they collaborated with another company,

and gave birth to a hit piece of software.

Afterwards, the number of creators using the singing synthesizer software rapidly grew

and a new form of musical pop culture was born.

How do experts view this invention?

We're joined by Goto Masataka to learn more.

Welcome to the show.

Thanks for having me.

What was revolutionary about this singing synthesizer software?

You sequence the notes, type in lyrics, and you get a synthesized voice.

Those are the basics.

But the biggest thing was adding a character you could relate to.

Having the premise that this character is the one singing,

now you can write lyrics for them.

That's actually the key to its popularity.

They made a character who people want to make songs for.

That's how they succeeded.

What impact did this software have on the music industry?

People used to think that only real human voices were worth listening to.

But now people enjoy songs where the main vocalist is a singing synthesizer.

That culture was born in Japan.

Of course, you need to have people using the software for it to spread.

Video sharing sites were getting popular.

Many people were able to view creations.

And the response would feed back into new pieces.

Having a platform to spread the culture together with the synthesizer was key.

How do you think singing synthesizer software will evolve going forward?

It'll be important for the technology to enable diverse forms of expression.

Like voices that are husky, or bright.

To be able to synthesize all sorts of qualities.

There are still many voices out there, like ones that humans can't make but find favorable.

It'll be a challenge for them to make those.

Thank you for your time. Great speaking with you.

Thank you.

Top Niche Creations.

Our next segment is "Top Niche Creations."

Today, we're focusing on this:

It looks like wire at first glance, but it's actually something different.

You'll find it used in all kinds of electronics

like fridges, TVs, computers, and smartphones.

Just what is it?
We sent our reporter to learn more.

Hey everyone, Cyrus Nozomu Sethna here.

Today I'm in Adachi ward, Tokyo.

The company we're visiting is right over here.

Come and follow me!

I'm at the headquarters of a company which makes metal materials.

It employs about a thousand people.

Hello.

Nice to meet you.

My guide for today is Shimamura Masato, who works in product development.

Here we are.

I was greeted with this spread after going inside.

Something resembling wire.

Small beads.

And some kind of powder in bottles.

What exactly are all of these?

All these are solder materials that our company makes.

You can think of them as adhesive for metals.

They connect things which conduct electricity on circuit boards or electronic components.

How are they used?
I was given a demonstration.

Ah I see.

When the solder touches a hot iron,

it melts, leaving behind a bit of metal.

Electronic components can be secured onto a circuit board as it cools and hardens.

Aside from solder wire, there are several other types.
For example...

There are a lot of very fine beads in here.

Right. These are solder balls.

This type of solder is applied by machines.

They're only 0.3 millimeters in diameter.

I take a look through a loupe.

Aha look at that.
It almost looks like a bunch of crystals.

Like looking at art. Beautiful.

Here's a clip of the solder balls melting.

Each one melts into the same precise amount.

So the computers and smartphones that you use have thousands of components.

It's just not practical to solder those all with the wire type.

So the most common way now is to use these tiny solder balls to connect the components.

And, the final type.

This is even finer. It's powder?

That's correct.
It's in order of fineness.

The one on the right is the smallest.

The average particle size is under 5 microns.

The powder is mixed with chemicals to form a solder paste.

This is a sample of a circuit board with the solder paste applied.

Let's look at the paste under a microscope.

Tiny particles are visible.

After being heated inside a machine...

It becomes like this.

The paste allows for many components to be rapidly soldered.

So all these tiny components in the smartphone.

These are soldered using the paste type.

For other home appliances like TVs as well,

I think it's fair to say they basically all make use of solder paste.

The company is famous around the world for its solder.

It was founded in 1938.

By supplying solder to Japanese electronics manufacturers,

it supported the spread of Japanese electronics throughout the world.

In the 90s, it began developing a new, more environmentally friendly product.

Until then, solder manufacturers around the world used lead.

But lead can cause harm to people and the environment.

So the company collaborated with universities

and home appliance manufacturers to formulate a new kind of solder.

Today, about 90% of their solder products are lead-free.

The company has also been working to address another major issue with solder.

What am I looking at here.

This is oxidized solder. We call it dross.

This happens when solder is exposed to air.

During manufacturing, the solder is melted in a vat.

The solder on the surface is exposed to air, and oxidizes.

It becomes a waste product known as dross, which cannot be used for soldering.

In order to reduce this waste, the company changed its formula.

After some 3 years of development,

it managed to improve the solder and reduce the dross by half.

Here's a look at the new solder.

You can clearly see how much less dross there is.

It's important today for manufacturing to be environmentally friendly.

This is a great step towards that.

Yes.

Lately there's a lot of talk of self-driving cars and AI.

There's strong demand for tiny but reliable solder joints.

We hope to keep providing solder while being an environmentally friendly company.

The company is known around the world for their solder.

Today, environmental concerns like carbon neutrality have become a focus for many nations.

To help with this, the company has created a low-temperature solder.

Conventional lead-free solder melts at around 220 degrees Celsius.

The new low-temperature solder has a melting point of about 140 degrees Celsius.

By lowering the energy needed to melt the solder, the amount of carbon emissions is also reduced.

The technology behind solder continues to evolve. Who knows what's next?

That's all for this episode of "Japan's Top Inventions."

We'll leave you with more about the singing synthesizer software from the first half of the show.

See you next time! And Stay Inventive.

(The singing synthesizer was launched
over 20 years ago.)

(Songs made using the software
are still uploaded all the time.)

(Now, there are versions in languages
like English and Spanish.)

(It's beloved around the world.)

(Kenmochi Hideki developed the software.)

(He's proud of the musical pop culture
that he helped create.)

Some things can only be expressed with a synthesizer.

I was so happy when I saw how people were using something that I helped create.

I'm truly glad to have done this work.

"Japan's Top Inventions."