Computational Design

A solo dancer, moving on stage.

Beside them... A virtual dancer.

It was created by an AI that has studied human dance.

The evolution of computing has made it possible for artificial intelligence to design sounds and images: a new design frontier.

This is called computational design.

Today's guest is Daito Manabe, an artist who uses computational design to produce breathtaking images.

His works include this cutting-edge live performance by Japanese band Perfume.

He's worked with a range of artists to create visuals based on computational design.

Explore the potential of new designs that leverage AI.

Hello. Welcome to design talks plus.

I'm Andrea Pompilio.

Hi, I'm Shaula.

So, Shaula. Today, we will take a peek into the future of design.

Yeah, that's right.

We're actually going to Daito Manabe's studio right around the corner.

I'm really looking for to seeing what he is working on.

Great.

Let's go.

Andy and Shaula are visiting Manabe's personal studio.

Manabe-san.

Hello.

Hello, I'm Andy.

Hi, thank you for coming.

I'm Shaula. A pleasure to meet you.

I understand this is your personal studio, is that right?

Yes, I hole up in here to work on my projects.

It's quite big, isn't it?

Yes.

The high ceiling allows me to put speakers up there.

Oh yes!

A very versatile space.

And useful for filming too.

You have a lot of equipment. I can see a DJ booth.

I started DJ-ing at 15 or 16, so I've been doing it for around 30 years now.

That's a lot of records!

I have five or six thousand.

Five thousand records?

I feel like music is what got me into my current line of work.

Musical instruments too.

Yeah.

I often work with musicians.

There's a self-playing piano as well.

It plays itself?

Yes, it does.

I'm actually working on a project with Ryuichi Sakamoto right now.

The self-playing piano is going to be a part of that work.

An ongoing project.

That's right, yes.

Manabe shows Andy and Shaula a program that allows them to experience computational design.

Should we do something?

Move around?

You're fine as you are.

Just like this.

I'll move around a little.

Now let's see how it turns out.

That's good.

The AI analyzes the image and overlays a visual effect.

Computational design can make this happen in a snap.

Our theme for this episode is computational design.

Manabe-san, what does that term mean to you?

It's used very differently in different fields.

I personally work in the arts and entertainment.

It used to be that a person would have to draw a long series of sketches in order to create a short video sequence.

Now, software can automate the whole process.

You only need to draw one graph and you can change the size of a ball along the whole sequence, for example.

The computer handles all of the calculations involved.

But the most extraordinary changes are coming in the field of machine learning.

This is one field in a broader spectrum that most people would know as artificial intelligence, or AI.

The last decade has seen an astonishing evolution.

Manabe's collaboration with a traditional Japanese artform won acclaim.

A kyogen performance by Nomura Mansai in 2017.

He danced Sanbaso, a historic piece performed during festive occasions such as the new year.

3D images unfolded behind him, in harmony with his movements.

It seemed as though the live performer and digital images were dancing together.

The visuals were the product of a computer analyzing Nomura's dance.

Four months before the performance, he donned a special suit with markers at each of his joints.

The technology is called motion capture.

His walking speed and stride length were all digitized.

They then used 48 cameras to capture Nomura's movements from every possible angle...

...and carried out a full volumetric capture.

Together with the motion capture data, the program was able to produce a digital dancer.

Manabe's computational design visuals begin by capturing and digitizing an analog source.

It took months of work to produce visuals that would enhance the unique atmosphere of kyogen performance.

AI has also been used for other performances.

Images of the audience taken before this Perfume concert were used to show them recreating the group's signature dances.

The band also uploaded the lyrics to all of their songs and had them analyzed.

The most common words were reconfigured into an astonishing new performance.

The kyogen performance was, I believe in 2016 or '17.

That was one approach to computational design.

But nowadays you don't even need the motion-capture markers.

The AI software has evolved beyond that to recognize human poses without them.

In just five or six years?

Yes, it can simply read the movements and poses.

In just five years.

That seems incredibly fast for such improvement.

It's incredible.

It's really picked up a lot of speed since 2017.

There are some things that can only be done with a computer.

The Perfume project took all of their lyrics and laid out the most common words in order of frequency.

I guess you could get some poor intern to do all that.

But it's just so much faster to get a computer to sift through it all and figure out that "you" is the top word, then "I" is the second, "light" the third, and so on.

I think that information leads to some fascinating new places.

So when we hear 'computational design' is this what people are talking about?

A technology with unlimited potential?

I think so, yes.

But people use different terminology.

I hear 'media art' a lot as well.

Oh yes.

Or digital art.

The kinds of project that you're involved in - do you think of your work as art?

Or as design?

I suppose my perspective is that when we come up with an idea ourselves, I think of it as art.

But when we're asked to come up with an idea to help something go viral online, for example, then that's design.

In that case, we've been given a problem and we're working towards a solution.

That, for me, is design.

Posing, and solving, problems.

It's about the solution.

Exactly, yes.

For Manabe, a design solves a problem.

He showcases another shining example.

A program that lets people follow the movement of a fencing blade in real time.

It was actually used in competition?

Yes, it was.

This was employed at an international competition.

Interesting.

It makes watching so much easier!

Fencing is fascinating but the blades move so fast, and bouts often end in the blink of an eye.

I was asked to find a way to make them more entertaining.

Intriguing idea.

So I thought, what if we could follow the tips of the blades with cameras, and then leverage computation to make all of those movements visible?

That could be fun.

It's such a great idea.

I made the proposal, and at first we had to put markers on the blades, like the kyogen motion capture.

Of course that makes it easy to capture, but it's not something you can do for an actual competition bout.

All this was way back in 2013.

But I made the prediction that by the 2020 Tokyo Olympics, AI technology would have improved enough to do this without needing to use markers.

And that prediction turned out to be correct.

You made the deadline?

Yes, this was used for the fencing at the Tokyo Olympics.

How difficult is this technology?

So at the beginning, you have to gather a huge amount of images and add the location of the tip of the blade by hand.

We'd film a practice bout, break it down into thousands of pictures, and have the part-timers at the company add the location of the blade tip.

A big team?

It took twenty people months to complete over 200,000 images.

That was the training data.

Exactly.

The AI was able to learn from that data.

Then, when we showed it new, unknown images, it was able to make a prediction on the approximate location of the blade tip.

This one is more of an art project, I think.

This project asks questions: it's an artwork.

Data from the Tokyo Stock Exchange was used to show the sale of stocks in real-time through images and sound.

The result is a visual representation of movements across the entire market and intercompany connections, displayed in a three-dimensional space.

We made this at a time when the concept of artificial intelligence wasn't so mainstream.

But I knew it would make so many things easier.

I also knew it had the potential to cause problems.

So I wanted to make an artwork with AI at its heart.

This was in 2013, but one of the biggest uses for AI back then was actually to make automated stock transactions.

I see.

Now, you called this an artwork, not a design project.

I feel the border is quite fuzzy.

Here it seems like you're asking questions, but at the same time, you're presenting a solution.

You're taking the problem head on.

I feel like you have both of those elements at play in this piece.

I agree.

And of course there are some people who see this and feel that it's more design than art.

I'm sure.

As you can see, all of these moving graphs and data are making interactions on the stock exchange visible.

You can see it reflected on the share price of various brands.

It's visible.

Because of this, some people feel that this work falls more under the umbrella of design.

I think that's a valid perspective to take.

I'm happy for people to make up their own minds.

Daito Manabe drifts freely between art and design.

His personal history plays a major role in his creativity.

He was born in Tokyo in 1976.

It was a musical family - his father played bass and his mother keyboard.

He bought his first DJ turntable aged 16.

After polishing his skills, he moved to New York and began working with local artists.

At 24, he turned to the study of media art.

Manabe dug into computational design at a technology and art school in Gifu prefecture.

After graduating, he and some friends set up creative unit Rhizomatiks.

They've created countless artworks together.

Manabe first won major public attention with this work.

It's a program that turns music into electronic signals.

The signals travel to electrodes that tighten facial muscles, making the wearer's face twitch in time with the music.

Manabe's early immersion in music inspired him to find new ways to make music visible.

This inspiration was on full display during a live stream in 2020.

Computer graphics were created in real-time to the sounds of DJ KRUSH.

The result was a work people could enjoy with their eyes and ears.

This live performance marked a thousand days before the Tokyo Olympics.

The background is made up of real-time messages sent by the audience to their future selves.

Making sound visible, and interactive.

This work combines both of these elements.

An enormous installation exhibited at the National Museum of Modern Art in 2021.

Countless balls travel along rails, apparently lighting up the room.

The location of each ball is actually being tracked by cameras, and illuminated by lasers.

Multi-channel sound reacts to the location of the balls, creating a fantastical symphony of light and sound.

What is your relationship with sound?

My first love was regular music.

Songs of a fixed length - the kind of thing that would be sold on CDs, or played by DJs.

But after a while, I moved on to truly computational music.

I started to make sounds and music that was generated autonomously, following algorithmic or mathematical rules.

I actually studied math at university.

Towards the end of my course, I began to really dig into that aspect of sound production.

We've seen several works that turn music or sounds into images.

As a DJ, I'm sure you're familiar with the way that a live performance has its own unique energy that only the audience can sense.

But looking at your work, I feel like you're really able to translate that atmosphere into something visual for people who can't be there.

Maybe even give them a taste of how it feels to be in that atmosphere, together with the audience and performers.

I really feel that potential.

I do think there are certain kinds of music that become even better when accompanied by something visual.

Music alone can create an incredible sense of unity in a venue.

But when you're streaming an event online, for example, it's often better if there are visuals as well.

Right.

You know when you're in a venue, there's something happening everywhere you look.

But when you're just watching the stream, you might only get a shot of the DJ's hands.

We went into the stream with KRUSH knowing it would be streamed.

So we specifically set out to create visuals that would enhance viewers' experience of the music, that would allow them to immerse themselves, even through the stream.

So, what is your objective with these projects?

What kind of role does the audience play?

Is audience interaction something that you always focus on?

I don't necessarily think art needs to be evaluated for it to be art.

But at the same time, no artwork is truly complete without an audience.

Until someone sees it and thinks about it, it remains unfinished.

So one question is, what will the audience think?

How can we change it to react to the movement of the audience?

Making an interactive work is fascinating, because the audience moves and that changes how the visuals play out.

When the visuals change, the audience reacts to that change, which starts a feedback loop.

So it's a question of, how do I make the most effective loop?

Interesting.

That's definitely part of my thinking.

You gravitate to some really interesting projects.

How much of this is because you let yourself pursue whatever interests you?

I think that's definitely the case for much of my work, yes.

I'm interested in my own body and its responses.

And more recently, I'm also interested in my brain.

At the end of the day, I think I'm making things that I personally want to see.

So yes, I would hope that the bulk of my work continues to be about things that interest me.

Manabe's creativity continues to evolve, guided by his own taste.

In October 2018, Manabe presented a new stage piece.

An AI learned dancers' movements and created a new choreography, which it performed alongside real dancers.

It's an extraordinary new approach.

Manabe looks at it as a way to expand physical human expression through computational design.

This is his most cutting-edge project.

The user thinks of a pose in their mind, and technology known as 'brain decoding' recreates that pose on a computer.

In other words, a person lying completely still could move a virtual body just by thinking.

Manabe used this to create a dance sequence in his imagination.

It's still at the simulation stage, and this is just a prototype.

But Manabe has high hopes it could lead to new forms of expression.

That's amazing.

It is - would you mind explaining the details to us?

Is the software part of a larger research project?

Well, to start you have to have an MRI.

During the scan you might be shown images of flowers or the sea.

Or maybe chairs.

You see images.

Yes, inside the MRI.

And it captures data on your brain's reactions to seeing flowers, or on seeing a chair.

The first step is to pair up the data from your brain with the image data.

The next stage is to visualize something in your mind, without revealing what it is.

The MRI captures data from your visual cortex and a technology called 'brain decoding' predicts that you're imagining flowers, for example.

I see.

Brain decoding.

Yes, it's called brain decoding.

It's a technology that we're using in a number of different projects at the moment.

The latest one is the dance prototype.

Essentially, you visualize certain poses from inside the MRI.

Just in your mind.

That's right.

And the system actually reads what you're thinking.

If I were to visualize a whole dance, for example, it would capture data for that dance.

That's incredible.

And so you can make a virtual avatar dance.

I can see a lot of possible applications for this.

For example, you could visualize yourself doing something, then that would be turned into actual images,

and seeing those images could be a great way to help yourself overcome a mental block.

I feel like this could be super helpful for learning new skills.

There are also so many things we can't do in the physical world, but which avatars could potentially do for us in the metaverse one day.

On the other hand, there are definitely lines that mustn't be crossed in a virtual setting, and these lines need to be clearly defined.

We need to decide exactly how far computational design should go.

How much freedom devices like this should be given is a difficult question and a very important challenge.

How far do you want to take your work?

And what purpose do you want to use it for?

I think the pandemic has helped me reexamine the core of my work.

What is it that actually interests me in this space?

I'm using this opportunity to reconsider my work through that lens now.

And the answer, for me, is sound.

It's why I built the space we're in here.

Sound is the heart of it, and I'd like to go back to basics, and see what computational design can do in that field.

I'm sorry to say I don't have some overarching ambition to tell you about!

No, no, not at all!

The fact that you feel compelled to return to your roots seems to me almost like a sign that this is something not just you, but the whole world is really aching for.

It's been wonderful to have this chance to talk to you today.

Thank you so much.

Thank you.