Mike Kent, MIDI Industry Leader – 2018 DigiPen Audio Symposium | DigiPen Institute of Technology

So we’re ready to get rolling
with our next speaker. Mike Kent is an industry leader
in MIDI and musical instrument industries. He’s a member of the Technical
Standards Board of the MIDI Manufacturers Association. He’s co-author of the
USB MIDI specification, principal architect of
MIDI CI, and contributor to USB audio one, two,
and three specifications. And I know that’s a
lot of wonky stuff. But for those of you,
especially my students, you know what that means. He had a 30-year
career at Roland before starting his own
consulting business. He’s contributed components
to MIDI for Windows, for Nintendo’s developer SDK. And he’s worked closely
with Apple on MIDI and audio solutions
for many years. He contributed to an audio
processing system for a NASA event simulation center,
and to audio capture for an archiving network
for the US Coast Guard. Most of Mike’s work is now
focused on the future expansion of MIDI, working as a
consultant to Yamaha R&D. Mike owns 19 synthesizers
and eight guitars. Please welcome, Mike Kent. Thank you. I’m nervous to
know now that there are 10 people watching online. One of them probably
is my mother. OK, but before we start, I
would like to take your picture. This is proof to my wife
that sometimes people like to hear what I have to say. OK, so the topic for the next
presentation here is MIDI. And MIDI has been with us–
and we call it MIDI 1.0, because fundamentally it
was invented 35 years ago. And we still use
the same language to control musical
instruments today. How many people who use MIDI? Every hand went up. OK, how many people
are experts in MIDI. OK, half a dozen hands went up. Excellent, so that
gives me some help to know how quickly to go
through different sections here. So the history, I’d like
to talk a little bit about how we got to here. And the arrival of
digital technology in the musical instrument
business that I come from was 1977. Ralph Dyke was a
real pioneer, just across the border in
Vancouver, where I live. He worked out of Little Mountain
Sound Recording Studios. He had a studio there. And he invented a sequencer
that had a microprocessor in it. Mr. Takahashi, the founder
of Roland, met Ralph in 1977. They released the MC-8, which
was an update to his sequencer. And it was the first commercial
musical history product with a microprocessor in it. So Ralph was a real pioneer. Another pioneer right
around the same time is Dave Smith, who also
happened to building sequencers. And he was released
the Prophet-5, which was the first synthesizer with
a microprocessor for storing the patch of the
synthesizer, really a breakthrough instrument. Digital interfaces
started to happen. So instruments were
connected to each other with digital connections. And this was all breakthrough. And in 1981, Roland had
a system called DCB. Oberheim had their system from
connecting their synthesizers. And sequential circuits
had a third one. And Mr. Takahashi of Roland I
went to Tom Oberheim and said, we’re doing this thing. You’re doing the same thing. But they don’t
talk to each other. How about we both agree
to do the same thing? And Tom went to his
friend Dave Smith, and said, what do
you think of this? And Dave got really
excited about it. They went and told the whole
world, let’s do this together. You’ll see that that’s a bit
of a theme through what I have here to say here
about MIDI today. MIDI is a cooperative effort,
where even competitors together and agreed
on something, and pushed the industry forward. And talking to other people
that are in your industry is always a good thing. Belonging to the gang
network is a good thing. So out of that came MIDI 1.0. And MIDI was actually– the
language of it, the protocol, was almost basically
DCB version two. The hardware came from Dave
Smith and sequential circuits. And those two things came
together and formed MIDI. What else was I going
to tell you here? I forget, it wasn’t
important, apparently. So I’ll just go into the
background of what MIDI does. It’s a communication
language where one device can talk to another device. So I’ve got the pink
synthesizer down here. And if I play a note, it
sends out a message saying, turn on that note. And another synthesizer
will turn on the same note. And when I release
my finger, there’s a simple message that says,
stop playing that note. So these are just
note on, and note off. And this is probably the
most important MIDI message, at least for
musical instruments. In addition to the
notes on and off, there’s a couple of other types
of messages that get sent. Selector here, there’s a message
called program change, that allows you simply to
select which sound or which function
a device is doing. And then you’ve got controllers. And these are things like just
turning the volume up and down, or bending your
pitch up and down, or adding vibrato to a note. These are controllers. These form kind of the
fundamental types of controls that MIDI communicates
from one device to another. MIDI has 16 channels. So if this pink
synthesizer wants to talk to the
greens synthesizer, it can do so on channel 1. And the blue synthesizer will
be set to receiving channel 2. And so on the pink synthesizer,
when I set it to channel 2 it’ll switch from
talking to the green one to talking to the blue one. So I have multiple channels
on one cable, 16 of them. These are digital signals that
can be recorded with something called a sequencer. A sequencer is simply
a MIDI recorder. Now, sequencers can record data. They can do all kinds of
manipulation and editing on that data, and then
play it back to the device. And when that sequencer
is playing the device, it’s actually a real
time performance. The sequencer is sending,
hey, play this note now. Now, stop playing that note. It’s just playing back to
commands that is recorded or that you’ve edited. Similarly to the keyboard
talking on different channel, sequencers can send
on multiple channels. And so it can send a baseline
to one type of synthesizer, the drum notes to a different
synthesizer, and chords and melody to other synthesizer. And there are 16
channels for doing that. So these are the
core ideas of MIDI, which led to that different
types of instruments. So if I’m using a system
with a sequencer like this, maybe the blue and
green synthesizers don’t need all the keys on it. I can generate all the notes
from my pink synthesizer there. And, today, we have
plug-in synthesizers, which might be represented by
these green and blue blocks up here. So the synthesizers don’t all
have keys on them anymore. But they did originally. We’ve also been able to move
beyond just the keyboard. A MIDI does come from
the keyboard industry. And so the note on and note
off model works really well for keyboards. It becomes a little
bit more problematic. But we’ve found over the 35
years, solutions for using MIDI in other applications. So, typically, a
guitar with six strings will send on seven
MIDI channels. There’s one MIDI
channel per string. And then the seventh channel
will be master volume, and those kinds of controls. This becomes important
as compared to a keyboard to use multiple channels,
because typically on a piano, I can’t play two versions of
middle C. There’s only one key. And it’s either up or down. But on a guitar, I could have
a C coming from one string. And up a few frets on the
next string is the same C. And so I need to do that
on multiple channels. So we found over the years
ways to expand and use MIDI in new ways
to do new things beyond its original design. Now, back to
history– it’s funny that Brian talked about history
in his talk this morning. And I had the same kind of idea. MIDI expansion happened. First thing that happened
was personal computers were coming out at around the
same time that MIDI came out. And Roland had the MPU-401. It was a standard
MIDI interface that was a little box that
connected to an ISO card to go into an IBM PC or
compatible, back when we used to talk about the PC
as an IBM PC or compatible. It could plug into an
Apple or a Commodore 64, and add MIDI functions to
those personal computers. There was multimedia. Boy, did you know that
computers could show pictures? And they could even be in color? Or they could create sound. And it could do both
at one time, a picture and sound together. These were the
early days of MIDI. In the late ’80s and
early ’90s, multimedia was a big part of driving
where MIDI was used, and the MIDI specifications. And General MIDI was one
of the extensions to MIDI that came along that defined
a default set of sounds. At that time, here’s
the MP401, by the way. MP401 connected to an ISO
card, or a card for an Apple, or a little cartridge that
went into the Commodore. And beside it here
is a soundblaster. This is what you had
if you played games and you wanted sound. You had a soundblaster
sound card. And it had a Yamaha
FM chip on board, the chip that Brian was
talking about today, using the FM technology from Stanford. And there was a joystick
connector port on there. And some of the pins on
the joystick connector were the pins that were
used for MIDI interfaces. And so this is the
standard way MIDI was done was off of a joystick connector. And so MIDI advanced, and
moved into personal computer, and became part of multimedia
because of products like this. In 1997, Microsoft came
to me, and said Mike, you know that MP401 thing
that you guys set the de facto standard for, and
that Creative Labs is putting into their
soundblasters and MP401 compatible MIDI interface? That’s going to go away,
because we’re going to take away the joystick port. Other companies
like Motu and Opcode were using the parallel
ports and serial ports. They said, all of
those are going away. And USB is coming. You better think about what
to do with MIDI on USB. How’s it going to work when
the joystick port disappears? So technology forced
us to adopt USB. So I went to Roland,
and said, hey, we’re going to have to
write a USB MIDI spec. And they said, OK, why
don’t you go do that? And so we worked with
the USB implementers forum with Microsoft and Apple
at the time, and Creative Labs, and a few other
companies joined. And we wrote the USB mini spec. And it came out in 1999. It’s hard for me to believe
that I actually started this work like 20 years ago. But USB added some
things to MIDI. First of all, it allows MIDI
to run up much higher speeds. MIDI was 31.25 KBOT. We don’t measure
things in KBOT anymore. And so it allows MIDI to run
at hundreds of times the speed that it was originally
designed for. So that’s great. It also goes beyond
the 16 MIDI channels, because we have 16 virtual
tables on the one USB cable on what’s called an endpoint. And so that gives
us on one device, easily 256 mini channels
can happen on one device. And if I instantiate
another end point, I actually get another 256. I haven’t seen a device
that needed more than 256. But it’s possible. So USB added a bunch
of things to MIDI. And MIDI continued to
advance and allow us to do other things. Today, we can also run
MIDI over a network using RTP, which is real-time
transport protocol, which is built on top of UDP. So another way of running high
speed MIDI, and running it long distances in a
network environment. This is often used,
actually, with something called MIDI show control. And so you go to theme parks. A lot of things are run
by MIDI using RTP MIDI. If you’ve been to Universal
Studios in Southern California, I assume they still
have the Waterworld thing where the plane comes
down, and lands in the water. And there’s fire, and smoke. And all the systems there
are controlled by MIDI. The fountains at the
Bellagio in Las Vegas, they are controlled
by MIDI as well. So RTP really takes MIDI into
new environments as well. More recently, we have MPE,
MIDI Polyphonic Expression. If you’ve seen the ROLI
seaboard, or the Linnstrument, it’s called, from Roger
Linn, these use MPE, which allows more expressive music. So if I play a chord, I can add
to just one note of the chord, vibrato, or filter swell,
or make one note swell up in volume, as compared
to the others. So MPE adds more
expressive control. In fact, we’ve added
a lot of things to MIDI since MIDI
1.0 came out in 1983. This is a list of the
expansions we’ve made to MIDI. On this side is the
confirmation of approvals list. I don’t know. The name doesn’t
make sense to me. But these are new
message specifications, new messages in MIDI. And on the far side, we
have other specifications that are called
recommended practices. And this is taking
a set of messages, and let’s all agree to use
them in a particular way to solve a particular need,
like MIDI show control. And so we’ve done a
lot to expand MIDI. So while MIDI is still
MIDI 1.0 language, and MIDI 1.0 is what we call
it today, MIDI 1.0 of 1983 is not what we have today. We’ve done a lot. But we’ve added a lot within
kind of a walled garden. We’re kind of
pushing at the walls. There’s not a lot more that
we can do for several reasons. First of all, users expect
backwards compatibility. They want to use MIDI, the new
MIDI devices they get today or by today with
devices they bought 10, 20, and 30 years ago. And they just expect
MIDI to continue, as it’s done for 35
years, to mix old and new, and have good compatibility. And the MIDI
specification is actually fairly tightly defined, in
order to serve interoperability and compatibility. And so it’s hard when you
have a specification that is fixed to expand it sometimes. Common practices– we
just know how to use MIDI. We like the way it works. And we know how to use it. So we don’t want to
change that too much. And so as we get new ideas and
the tech environment changes, we still have our ways
of working that we like. And really fundamentally,
MIDI has no opcodes left for new messages. We’ve exhausted all the opcodes. And things that we’re doing
to expand MIDI now today, some of them are a little bit– well, I’d call them hacks. And we have to hack MIDI
in order to add things. And it’s just becoming
more difficult to expand. But today, MIDI works. In fact, it works really well. And so we’ve kind of come
to the edge of the garden. And the question
today is, are we done? I don’t think we are. But the fact is,
there’s some resistance to go any further, because,
one, it’s difficult. And, hey, it really works. Everybody in the room
here is using it. Anybody not like it? There are times I don’t like it. But, generally,
MIDI really well. So do we need to change? I’d like to take a fork in my
presentation here for a moment, and talk about the fork. The fork works really well. It’s looked like this
for about 150 years. A couple thousand years
ago, they looked like this. Now, if they still
looked like this, they’d be good for poking your
little brother under the table, but not so great for
putting food in your mouth compared to the modern fork. We’ve built other
forks along the way. Here’s one with
seven tongs on it. Now, I’m sure that’s
really good for something. But I’m not going to use
that to put food in my mouth. OK, so there’s other
types of forks. There’s other
things you can do by controlling musical instruments
that MIDI maybe doesn’t do. But for what MIDI does,
it does it really well. Do we need to change this? This has been with
us for 150 years. Could we make the handle longer? I’d have to eat up here. Make it shorter? No, it works the way it is. So do you change the fork? MIDI works. Should we change it? Well, the fact is, there
are some reasons why we might have to change it. And there are some reasons why
we might want to change it. But change it in the right way. So there’s a push for change. First of all, users’
expectations and desires, and users vote
with their dollars. You buy something
because you like it. And, therefore,
manufactures sell you more of the same thing. Whatever you buy, they’re
going to make more of that. So MIDI works well, and people
continue to buy MIDI products. And so we continue to make more. People want backwards
compatibility. And so as manufacturers,
manufacturers have to look at
that and say, well, whatever we do in the future has
got to work with the old stuff, because our customers
vote with their dollars. And they’re not
going to buy it if it breaks all their old gear. So users’ expectations
is a huge factor to weigh as we look at
change in technology. Small, one man
companies– this didn’t exist in the earliest
days of technology. Today, one man can do
so many cool things. I am prototyping new
MIDI products just on an Arduino board. There are so many great tools. And one-man
companies can produce great products, whether
it’s an app on an iPhone, or some small MIDI widget. And one-man companies
are really nimble. They can change direction
of their company very easily, very quickly. So they want to change. And a lot of innovation
comes from small companies. Big companies, on
the other hand, like Yamaha that I
worked for, and Apple, and Roland that I
used to work for, those kinds of companies,
change is difficult. Even though they’re
in the tech industry, major change is very costly. So there’s new technologies
are all around us. And we want to take
advantage of them. Or sometimes, we’re forced to. Just like when Microsoft
came to me and said, you better do something about
MIDI, because USB is coming. And the joystick
port is going away. We had no choice but to adapt. And to some extent, we’re in the
same situation with MIDI today. We almost have no choice,
because new technologies come along. And then those feed back
to user expectations. Every device is going
to be on the internet. They’re all becoming
connected devices. But your synthesizers
today are generally not connected to the internet. And that’s changing through
a new thing called Web MIDI. And so there is technology
change around us that pushes us forward,
whether we like it or not. And then there are things
that we want to do, like MP that makes more
expressive musical instruments. So we can do more
creative things. The technology moves ahead
so that new creative things can happen as well. And it’s not just technology. But we keep in mind,
fundamentally, this is about making music. MIDI is used for a
lot of other things, like the fountains of Bellagio. But, fundamentally, it’s
got to serve making music. And we can make music better
if we have a better MIDI. So one of the things we
have to consider in MIDI is the value of the
existing knowledge. We’ve done an awful lot
to learn how MIDI works. Users have invested. They’ve bought
books in the past. There used to be a best selling
book called MIDI for Musicians. They must have sold
a lot of books. And users today are
go online and read. You can watch YouTube
videos on how MIDI works. The press invested an
awful lot over the years. Magazines in the
past, today of course mostly online, but
there’s a whole mechanism of serving information. And there’s this
value of knowledge that exists in MIDI today. The dealers know about it. Manufacturers educated all
their dealers, and said, this is what MIDI does. And we have 35 years of this
almost a pedagogy, or existing knowledge, that we’ve
built up around MIDI. And we don’t want to
abandon all of that. In the early days of MIDI,
when I joined Roland, we used to ship a MIDI
cable inside every box. And so people would
buy a keyboard. And they’d say, what do
I do with this cable? I only own one synthesizer. I guess I better buy another
one so I can use this cable, and connect it up. But that was an investment
that musical instrument companies made in promoting
and educating about MIDI. And that cost
millions and millions of dollars over particularly
the first five years. it was just very expensive. So we want to use the existing
resources and knowledge. We don’t want to change
completely to something where we have to throw all of
that existing knowledge base away. We want to innovate. But there’s this thing,
backwards compatibility, backwards compatibility. This is a theme that comes up
over and over as we ask users. We’ve done surveys
over the last few years from the MMA, what do
you want out of MIDI? And what should we
do to move ahead? And people say,
hey, do cool things. But, please, don’t make it
incompatible with my old gear. And so users have time
invested and money invested in their gear. There’s comfort in the
workflow of how MIDI works, how sequencers work. All of those things, we
don’t really want to change. And there’s also–
this is different from many technologies. When a new GPS comes out,
you throw away the old one because you like the new one. And then when your
iPhone does GPS, then you probably don’t use
the GPS in my car anymore. We’re embracing new technology. But musical interests are
very personal to people. And I’ve invested a lot of time
in this musical instrument. I have an emotional
connection with it. So don’t make me
throw that away. Allow me to use that
in a new environment as part of my creative
being of who I am. So those are all
factors we consider as we look at future expansion
of MIDI, or changing the fork. So two years ago,
I had this idea. We’ve had a lot of proposals
made over the years to do new things. Some of them were
very MIDI like. And some were kind of like
the fork with the seven prongs on it. They do something really well. But it’s a little bit different. And so there have been other
proposals that come along. And some of those
ended up being in MIDI as those recommended
practice, and so on. But some things
have not worked out. I came up with an idea. And I proposed it two years ago
called MIDI capability inquiry. And I proposed it to the MIDI
Manufacturers Association. And, basically, the idea is that
MIDI is fundamentally designed as a one-way communication. I send you controls. And I control you from
sender to receiver. What if MIDI is always
a two-way communication. A USB cable, well, it
has MIDI data on it. It guarantees that two-way
communication can happen. Whereas a MIDI cable
was a one-way cable. And you had to hook up two of
them if you wanted two-way. So if we assume two
way communication, can we do new things
that we didn’t do before? And the idea is that if two MIDI
devices agreed to do something new, then do the new thing. Even if it somewhat breaks
the rules of MIDI 1.0, maybe you could
do something new. So you can agree to run at
higher speeds than the 31.25 if both units agree. You could agree to
use new messages that were not existing, and
couldn’t be added to MIDI 1.0 if you both agreed
to break the rules, and say, no, I’m going to add
messages, even though MIDI was out of messages. So this is the fundamental idea
of MIDI capability inquiry. And I took that to the MMA. And Athan Billias of Yamaha
got quite excited about this. Several other people in the
MMA thought it was a cool idea. But Adam was really
adamant on this. And just like Dave
Smith got excited about the idea of the
synthesizer companies all using one common standard,
Adam got excited about this. And he convinced Yamaha
to continue with this. I just started a
consulting company. So Yamaha hired me
to actually continue to develop these ideas. So he convinced Richie Hasegawa,
who was the head of technology at Yamaha at the
time to hire me. Richie Hasegawa also have
to be the chairman of AMEI. I’ll come back to
AMEI in a minute. But Adam went and told people. So he went and visited
[? Roley, ?] and [? Mouge, ?] Steinberg [? Arturia, ?] IK
multimedia, Abelton, Motu, and very importantly, Apple,
Microsoft, and Google, and said, we should all
go and do these things, and try to enlist
support for the idea. And it’s because these companies
came along and said, yeah, we like that idea that
we were able to advance this in our industry. Some people said, I don’t know. Dave Smith said, you know what? I really like MIDI. Now, it was his. So he was one of
the adventurers. He won a Grammy award for it. So maybe he has the right to
say, I kind of like it the way it is. But go ahead and do your thing. And so we’ve moved ahead
with MIDI capability inquiry. Yamaha, Roland,
and Corg have all built prototypes as
proof of concepts as we were writing the spec. They made prototypes
to make sure it works. And AMI is the Association of
Musical Electronics Industry in Japan. MMA is in the United States, the
MIDI Manufacturer Association. The two of them jointly develop
new MIDI specifications. And so we took the
specifications their. AMEI was on board immediately
when Richie Hasegawa took it to them. And then AMEI proposed it
back to MMA two months later. And they said, yeah,
let’s go for it. And the MMA formed
a working group. 35 companies joined
the working group to write the specification. And, finally, the
MIDI CI was adopted this January,
actually in November by AMEI, and in
January by the MMA. So it’s an official
specification. So what is it? What does it do for me? It really is laying just a
framework, or maybe a date outside of this walled garden. It’s the door to
say, here’s how we’re going to expand beyond the
garden that was MIDI 1.0. And so it doesn’t actually
do anything by itself. It just shows how to do it. So there are three P’s in the
current version of MIDI CI. There is profile configuration,
property exchange, and protocol negotiation. So I’m going to talk a little
bit about the three P’s of MIDI CI. First one is
profile negotiation. And here’s three
different instruments from three companies. Here’s a traditional
B3, which is now actually owned by Suzuki. If you buy a B2 organ today from
Hammond, it’s got MIDI on it. This is a Yamaha draw
bar organ plugin. This is a Roland
portable synthesizer with draw bars on it for
controlling its organ sound. Three different instruments
that do the same things, they have the same draw bars. And, yet, they don’t
talk to each other, because they all
use different MIDI messages for the same function. The idea of a profile is to
define that the eight foot draw bar will always be controlled
by this particular message. And so we write a
profile specification. And then MIDI CI
allows two instruments to talk to each other, and say,
do you support the draw bar organ profile? And if the other
device says, yes, I support the draw
bar organ profile, then they agree to use that. And now, when I move
the draw bars on my B3, they automatically move
on my Yamaha B3 plug-in. And I don’t have to do
mapping, which many of us have already learned how to do. Suddenly, these things
just happen automatically. I have an example that is
even worse than these three instruments. Yamaha, who I work for a lot
these days, has three organs. They have an organ
in the montage. They have that plug-in. And then they have
the re-phase YC, which is a little organ
and draw bars on it. And so Yamaha has three organs. And none of them use
the same messages. So even within one company,
they don’t talk to each other. But through a profile, devices
that have similar functions, like the nine draw bars,
the percussion on off, the three chorus, the
three different vibrato selections, on an
organ, let’s all agree to do them the same way. So this is the
idea of a profile. And so here are the
types of profiles that we expect to come. MIDI CI does not
define these profiles. It just says how profiles work. So we still need to write
these specifications. I’ve written already the
draw bar organ profile. And I’ve written the
rotary speaker one to go along with it in
the effect profiles. So we have three main categories
of profile configuration, or type of profiles. Instrument profiles are
very typical instruments. And there may be, probably I’m
guessing maybe 30 profiles that would describe the
common instruments, like orchestral strings,
brass, other than trombone, because trombones has
got the slide thing. The rest of them have
valves, generally. So there might be a brass
profile and a trombone profile. Maybe they will
be the same thing. I don’t know. Nobody’s written that one yet. Somebody who’s an
expert in that area needs to write that profile. We need Roland and
Yamaha, because they make MIDI controlled
drum sets, probably need to write the drum set profile. If you are an expert
in one of these areas, why don’t you write the
profile for that area? We need effects. There’s probably 100
effects out there, distortions, phasers,
flanders, compressors, multiband compressors. There’s just a whole
lot of profiles that can be written so
that on a reverb unit, if I turn the
reverb time on one, it also applies to another
reverb somewhere else, so that I’m using commonality of
messages and things auto map, and systems auto configure. So if all of your
devices and all your plugins support
profiles, your DAW could say, who are you? What do you do? And a Yamaha piano
could reply, I’m a piano that conforms
to piano profile. And it could automatically
pop up a piano editing window. And maybe I’ve got a plug-in. And I’ve got an orchestral
string sounds selected. And it says, the profile
is orchestral strings. It will give me things about
articulation and string sections in an editor window. And then I switch the
past to draw bar organ. And, suddenly, that
window goes away. And a draw bar organ
window comes up. So automatic
configuration is one of the core ideas
of this concept of profile configuration, making
things easier for the user. And I think that’s
really powerful, and will really help workflow. So as I said, for user
easier configuration. And for manufacturers,
profile configuration is actually quite simple,
because they’re already designing musical instruments
with all of these controls. They just happen to all
choose different messages to control these things. And so profile configuration
is not a big challenge to manufacturers to do this. And it’s all built
around SysEx messages, which we’ve used for 35 years. We know how to do this. And so this is an
incremental step. It’s not completely
changing the fork. It’s just refining the fork. And so it’s a
simple thing for us to do, and not too far away
from what we already do today. Property exchange goes one
step further than that. And property exchange
is a set of messages that can ask far more
detailed questions. Like, what parameter is
controller 15 maxed to? Or do you have an ample
amplitude envelope generator with a decay? You do, what controller do
you want me to send for that? So you can ask very detailed
questions of a device. You can also get large
data sets out of a device. And one of the ideas
with property exchanges that will probably use JSON
or some other technology not standard MIDI
system exclusive, which is really just a blob of data. But JSON becomes human readable. And it’s very easy
for us to then include things like text strings. And so we can ask for a list
of all your patch names out of a synthesizer, and get
a list of the patch names, and then some meta tags
to go along with that. So property exchange
will do that. Now, the core ideas
of property exchange are the ability to get
and set properties. And get and set is
used in other things. It’s really never been
done in MIDI before. But there’s a lot
of other things. HTTP has get and set type
mechanisms, and pings, and so on. So we’re going to
borrow some concepts from other industries,
where there’s no reason to invent new things. So that’s why one proposal would
be that the language of this would be JSON. It’s machine to machine. And so, once again,
we’re going to help the user in configuration. So devices can talk to each
other and self map, or self configure. A DAW can configure to
support very detailed features of a MIDI device, whether that’s
in hardware or in software. This does present some new
fields for manufacturers. MIDI manufacturers may not
have done JSON ever before. So there are some
new things here. But it’s not too far away from
the technology that many of us are using today. So it’s a reasonable step and
a reasonable addition to MIDI. And it’s going to do powerful
new things, and help workflow, in particular. The last area, the last
P of the three P’s, is protocol negotiation. And this allows devices
to select the protocol they’re going to use. And this, potentially, is one
of the biggest areas that’s going to allow new
creativity, not just ease of use, and better
configuration, but new creativity. And so the idea is the ability
to negotiate the language you’re going to use. You’re going to use
either the MIDI 1.0 language, or the new MIDI
protocol that is currently being defined, and we’re
simply calling next generation protocol at the moment. So devices can decide
and agree together what language they’re going
to use to talk to each other. And so, therefore, we can kind
of break the rules of MIDI 1.0, and say, well, MIDI 1.0
language said that it had to be done this way. So we can break
some of those rules, and do new things that you
could not do with MIDI 1.0. We can do them in our
next generation protocol. By the way, you do
have the ability also to switch to
something completely manufacturer specific. So if you want your
device to generally do MIDI, but every now and then
go away and do something that’s not MIDI like at all, then
you can go and do that, and then return negotiate
back to using MIDI again. And so protocol
negotiation is part of MIDI capability inquiry. It comes with a test
for compatibility. So when you switch to
the next gen protocol, the two devices that
are talking one, of them sends a test message. If the other one receives
it and everything’s good, it sends back a test message. And the second one receives it. And then the two devices
know that they’re talking successfully in
the next gen protocol. If something doesn’t work
out, after a timeout, both devices fall back to
being MIDI 1.0 language. And so this preserves
compatibility. If I take a new product and
connect it to an old product, and I plug it in, and I say,
can we use the next gen protocol by hitting a configure button
on it or something like that? It talks to the old device. The old device doesn’t
know anything about it. Then this device times
out, and says, OK, we’re going to use MIDI 1.0. And that looks after
backwards compatibility in this test mechanism. So protocol
negotiation– again, we don’t define the new protocol
in MIDI capability inquiry. We just tell you
how to get there. So let’s take a look at
what this gen protocol might look like. And one of the things
that we’ve done, as we’re writing the
next generation protocol, is it has to use the
existing semantics and mechanisms of MIDI. So remember, as I told
you, MIDI 1.0, one of the fundamental
ideas is I press a note. And it says, start
playing a note. When I stop playing a note, it
says, stop playing that note. We don’t want to
change that mechanism. We want to use the mechanisms
and architecture that MIDI has always add, and keep them. And so the next
generation protocol is really more MIDI, which
you’ll see the bottom slide. It really is MIDI,
but with more. It’s incremental change. It doesn’t change
everything wholesale to something completely new. It’s a step by step expansion. And there’s room for
more expansion later. But in it’s first version
of next generation protocol, it looks an awful
lot like MIDI 1.0. But there’s a whole lot
of room for expansion. Whereas MIDI 1.0 was
this walled garden, we’ve now moved out of that
to a much larger space that’s far more expandable. Backwards compatibility
is important. So in addition to this
negotiation compatibility that I told you about,
just translation from the new protocol to the
MIDI 1.0 protocol is important. So if I record something
into a sequence, or record a performance
into a sequencer, and later I change it so
it’s going to play back on another device
that’s a MIDI 1.0, I don’t want to have that
being completely meaningless. I’ve got to be able to translate
that and have something meaningful come out
on the other device. So translation is an important
part of, or a requirement of, this next generation protocol. So I said there’s more MIDI. So what is the more? We want to do higher resolution. MIDI is a 7-bit data language. Everything is 8-bit words. And you only get to
use seven bits of it for data, because
the first bit is a status bit on every message. So we want high resolution. And today, computers use
32-bit, or 64-bit resolution for their data. And so we want to be able to
support 32-bit resolution. 32 bits is probably
overkill for a lot of properties or parameters. But we want to be able
to support up to 32 bits. And so we want high resolution. We want more channels. And so we’re going to provide
you more channels, and more controllers. And I’ll come back
to more controllers on the next slide in a minute. Something that MIDI doesn’t
have is per note controllers. And so MP gets around that. And MP is actually a
little bit of a hack in order to get per
note controllers. And so we want to be
able to take that concept and make it native to the
next generation protocol. And so we’ve added
per note controllers, and per note pitch bend so I can
de-tune one note from the rest, or bend a pitch from
this note to this note. So per note pitch bend is
a new addition to MIDI. So those bottom two,
they’re a little bit unique, in that the top things here,
high resolution and so on, those things translate
back very well to MIDI 1.0. These bottom two probably don’t
translate back to MIDI 1.0. So not everything is
going to translate back. You can’t do something
completely new and have the old things
completely do the same thing. Otherwise they are
the same thing. So these are some new
functionality here. We have articulation control. If you’re scoring
an orchestral score, and using a strings
library, and a plug-in, you know you’ve got switching
between bode, pizzicato, and [? spitizzicato ?] on
your violins, and so on. And so we want to have
articulation control in your note messages,
and be able to do that. And so that’s part
of the protocol, so that we all start to do
those things in the same way. If I use somebody’s
string library, and use their
articulation controls, and then load up another
string library today, the articulation controls
don’t necessarily match. They’re done in
different manners. And so we want to standardize
articulation control in this next
generation protocol. Expanded tuning capabilities– MIDI 1.0 was designed around
a 12-tone equal temperament scale, the Western scale that
we generally use in this country and in our Western music. But there’s a lot
of other music. And there’s a lot
of other tunings. And so MIDI does
actually have something called MIDI scale tuning, where
you can do non-western scales. But we’re expanding on that in
the next generation protocol, and allowing far more
ways of using tuning. And I’m excited about this. Just to get musical, how
many are music students here? We’ve got a third of the
room is music students. I’m thinking of,
I’m in the key of G. And I play a D augmented chord. Where is that D
augmented chord leading? If I’m leading from
that, and I look at the top note
of that augmented, the augmented note
is an A sharp. If I tune that a little bit
sharp above equal temperament, it starts pulling even
harder up to the B that it’s kind of hinting
at moving towards. I want to be able to
do that kind of thing. Or in that same
augmented chord, the D augmented in the key of G,
if the next chord is going to be a C major 7, going
from the five augmented to the four major 7, I’m
going to take that D, and maybe make it
a little bit flat, and see if that helps pull
towards that four major 7. I want to do those
kinds of things and experiment with the
tuning of my chords. And that’s musical
creativity that comes out of expanded tuning
capabilities in the next gen protocol. And as you can tell, I’m excited
about those kinds of things. Simpler NRPN and RPN. Has anybody ever used
NRPNs or RPNs in MIDI? No, anybody know what it is? Corey Taylor probably does. Dave does. NRPN, we ran out of controllers. There was only 128 of them. And so we ran out. And we said, OK, we need more. And so we said, we’ll use
one control change message as the most significant
bit of a new address. We’ll use another controller
as the least significant bit of that address. So now, we have a
14-bit address space. And then we’ll send
controller number six that has seven
bits worth of data, the MSB data of the
actual value that I want to set at that address. That’s the MSB. And then controller number,
what is it, 37, no, 38, is the LSB I’m going
to use for that value. And then I’m going to
send a clear message at the end of that
to say, I’m finished controlling that one thing. So I send five messages
to accomplish one change. It’s really complex,
and basically has not been implemented
very well, even by companies who do know MIDI very well. It’s not been implemented
very much at all. And, therefore, none
of you are using it, because your tools don’t
allow you to use it, because even the
tool manufacturers say, man, that’s ugly. So we want to fix that. So NRPNs and RPNs become
one atomic message in the next generation protocol. It is just a message. It’s controller. And here’s the index,
or the address. And here’s the value, which
is probably 32-bit value. And it translates back perfectly
to RPNs and NRPNs in MIDI 1.0. But they’re unwieldy
to use there. They’re really easy to use in
the next generation protocol. So that’s a big improvement
over MIDI 1.0 structure at expanding its controllers. So that goes back to the
first slide, where I said, more controllers. It’s actually not
more controllers. It’s the same amount,
just where you’re taking the 32,000 of
them that were unusable, and making them really usable. We’re also making an atomic
message out of bank, select, and program change. Bank select is,
again, three messages. There’s an MSB of your bank,
and then an LSB of your bank. And then you send a program
change to select your sound, because program change,
there was only 128 of them. Most of the sizes, they
have 1,000, or two, or 10,000 sounds. And so we needed to expand. So MIDI didn’t allow for
that in program change. So we defined this
bank select idea, where we’re using a
control change message now to set an address. And so it’s a little bit
unwieldy in MIDI 1.0. It works. But it’s a little bit unwieldy. Fortunately, a lot of DAWs look
after that unwieldiness for us, and give us just list
of patches, and so on. And so that’s a lot easier. But we want to make all of
that easier, even for the DAW manufacturers. And so we have a simpler
bank and program change would be a requirement of
this next generation protocol. The next thing is
improved musical timing. And time stamps is
something that can allow us better musical timing. Just by adding higher
speed on USB MIDI, we can improve timing. So in MIDI 1.0,
running at 31.25, if I want to play 10
notes simultaneously all on the downbeat, I send
the first one, and then the next one, then the next one,
and the next one, the next one. And the last one arrives
several milliseconds after the first one. So I got his jitter. When I try to play
everything simultaneously, it doesn’t come
out simultaneously because there’s a
serial port that’s slow. If I if I run that a
lot faster, the last one arrives much closer
to the first one. And so just by running faster,
we get better timing on MIDI. But time stamps has allowed
us to capture a performance with high resolution. So if I’ve got a
great performer, Oscar Peterson on the piano. I used to work with Oscar. I was his tech support guy. He’d recorded into sequencers
his playing on the piano. And we have captures
of him playing. But it’s all got this time
skew, where he played. He had massive hands. He put his hands down
like that, and cover four octaves on the piano. And so he’d play
these massive chords. And all those notes
when he played them happened simultaneously. But the recording didn’t
necessarily get it that way, because over MIDI
some of those notes were skewed later
than the first note. So with time stamps, the piano
can say, those things all happened at the
same time, and send every one of those notes
out with a time stamp that said these things
all happen together. And so when you play it
back from the sequencer, the sequencer can say
to the rendering engine, these things all
happen together. So play them together. So we’re going to get
better musical timing out of this next generation protocol
through the addition of time stamps. I can’t really tell
you a lot more details than that, because when
it comes to writing these specifications, it’s done
within the MIDI Manufacturers Association. And these things are
actually confidential. The actual design of this
and how it’s all done is confidential until the
specification is released. If you’d like to
participate in this, you can join and participate. But then you sign a
confidentiality agreement. So this is about
all I can tell you about next generation protocol
without getting in trouble. And I know some people
watching online right now might even be saying,
I think you’re sharing too much already. So to review, MIDI
CI has the three P’s. Profile configuration
allows me to auto configure various instrument
types or effects types. Property exchange is a
get and set mechanism to dig deeper into
details, or to get large data sets of instruments. And then protocol negotiation. Now, I’d like to just
cover these three a little bit in terms of
backwards compatibility. For profile configuration,
if two devices talk, and the device
on the other and says, I don’t support any profiles,
it doesn’t break anything. MIDI 1.0 can still
continue to work just fine the way it always has. So profile is just something
that we add that devices support or don’t. And if they don’t,
MIDI continues to work the way it always has. Same with property exchange,
if a device doesn’t support it, it doesn’t support it. You just continue to use
MIDI 1.0 the way it was. Protocol negotiation is
a little bit different, in that we have a test for
the next generation protocol. So two devices agree to use it. And then you switch there. Then you do a test. If the test fails, or
doesn’t get completed within a certain
timeout, then the devices both revert back to MIDI 1.0. And they have an agreement
that they’re going to do so. And so they revert to MIDI 1.0. And you continue to use
MIDI the way you always did. And so while we advance
and step outside the gate, we have ways to say, no,
this thing belongs back inside the walled garden. And we look after
backwards compatibility. I mentioned this as well. Down at the bottom here,
you see MIDI 1.0 protocol, and the next
generation protocol. And translation between
these two has to be simple. There has to be the
same concepts of note on and note off need to exist in
order to make easy translation. So the next generation
protocol, as I said, looks a lot like MIDI 1.0. It’s just more of the same. So I asked the question
earlier, what’s next? What’s next, really,
is just more MIDI. It’s not something
completely new. It’s an incremental step. MIDI CI Allows us to
step outside the walls that we have so far, and
start to do new things. So what do we need to do? What are the actual action items
that are going on right now? First of all, profile
configuration– we’re defining
profile specifications for common instrument types. If you’re an expert in an area,
I’m talking to you right now about profiles for tuning. Maybe you should write a
tuning profile specification. Property exchange– we need
to define properties and then standard application. So we’re defining
applications for getting patched lists, all in text,
and meta tags, and so on. And we’re defining
those kinds of things in the AMEI and the MMA. And we’re defining the new
protocol specification. In fact, we’re pretty much
done with the new protocol specification. This past couple of weeks
in the MMA and AMEI, we agreed that we’re
basically done. And now we’re going to do
proof of concept prototypes. So we’re going to
build some products that implement the new protocol,
test it for the next six months before we vote, and
say, yes, it works, and then release it as a spec. Or after six months, find out
that everything we invented is terrible. And prototypes show that we
shouldn’t do it that way. So that’s where we’re going. So what can you do? Join the MIDI association. It’s free. Go to midi.org, and join
the MIDI association. And there are
discussion forums there. You can discuss
all of these ideas. If you join, you’ll get
emails every now and then telling you about the
latest things that are happening in MIDI. You’ll find out when new
profiles are released. You’ll find out when
things are happening. If you actually want to
define specifications, then you join the MIDI
Manufacturers Association, also found at midi.org. So there are things
that you could do. Everybody in the
room should at least join the MIDI association. If you use MIDI,
you should join. It’s a lot of fun. And it’s free. That’s it. Any questions? Questions? I’m going to go
to Brian Schmidt. That was great. In looking at your presentation
about the parameter sets, it seems like it’s a
very similar problem that the internet of things has. You could easily see profiles
for door locks, profiles for refrigerators
or thermostats. Are there any other
protocols that are out there kind of trying
to solve this similar problem that you’re aware of that you
might think about incorporating in this? Or is there like a secret
plan to have MIDI CI take over the internet of things? There is actually a plan for
MIDI to take over the world. AMEI did a presentation
at the NAM show where they showed
MIDI controlled robots, a MIDI
controlled toaster, which was a bit of a joke. But just the idea of appliances
being controlled by MIDI, like the fountains at Bellagio
are controlled by MIDI. MIDI is a pretty
simple language. It’s not expensive. It’s not heavy. And there are applications. And there are reasons
you might want to use it. And so one of the things that
we want to do with profiles is expand MIDI to non-musical
instrument control. So, for example, I
don’t know if you saw the Intel keynote at the
Consumer Electronics Show. They had a performer on
stage playing a piano. And then they had AI
listen to what he played, and play back some music in
the same style, and so on. And they had these
huge, like 50 feet tall, graphic performers tat were
being controlled by MIDI. And so I imagine a holographic
performer sitting at a piano. And that performer is
controlled by MIDI. That allows the
holographic performer to be on the same
data stream as music, and where the clock messages are
for the beat timing, and so on. So, in fact, our
interest in profiles is probably to take MIDI to
other things, rather than– but we don’t want
to ignore what’s happening elsewhere,
which is why we’re looking at JSON for
part of property exchanges. JSON is a perfect way to
get and set properties. How are we going to
store this stuff? Corey asked me
this earlier today. What about storing all
of this new information? And there’s a
standard MIDI file, which is a unique file
format for storing MIDI. But maybe we should
be looking at– it if we’re doing new
things that include JSON, and other new types
of messages, maybe XML is a better way to store
MIDI in the future. So we are definitely,
AMEI and MMA wants to be outward looking. And we don’t want to invent
things that already exist. It’s much cheaper to– a lot of this stuff is
just driven by dollars. It’s a lot cheaper to
borrow an idea that’s already done elsewhere. I’m really curious what do you
think about open sound control? Open sound control
is really cool. It’s a little bit like a
slightly different fork. It’s maybe the
dessert fork of forks. It’s different from
the standard fork. The problem with open sound
control, if there’s a problem, is that it’s a little
bit unlike MIDI. And, therefore, it’s
a step further away. It’s harder to
translate it to MIDI. It’s harder for manufacturers
to build products using all their existing investment. All the testing tools
that they have for MIDI don’t necessarily apply to OSC. And so OSC has been
somewhat unsuccessful, because it’s a little bit
too far away from MIDI. And also, a couple
of times I mentioned, when I wrote the USB MIDI
spec, I went and visited Motu. I went and visited Opcode. I went and visited MIDI Man,
which became M Audio, and said, hey, this USB thing is coming. You’re going to be
implementing this. And this is what
I’m doing about it. OSC was not really
well-presented to manufacturers. Same thing write MIDI BCI. It was added Adam Bilias
who went and presented it to all of the
manufacturers that count. I went with him at times,
and Ben Israel from Yamaha was active in this. And the AMEI companies
were all active in this. And various people started
to spread the news. And OSC just really wasn’t
sold well to manufacturers. And so it hasn’t caught on with
the core musical instrument companies. And it’s for two reasons. It’s a bit far away from MIDI. And it wasn’t sold well. Thanks. What is the state
of audio to MIDI? Say, with the ability of people
to play instruments, and have audio then interpret
the MIDI data, or convert into MIDI data? Sure. So audio to MIDI has been
done for a long time, for many years. Guitar controllers
do audio to MIDI. And there are newer technologies
in the last five and 10 years that have come about
that actually can do it from a polyphonic source,
or listen to a WAV file, and figure out the chords that
are being played at least. So some of that is happening. And I think the new
next generation protocol will help us in that
regard, in that it allows us to define new messages
that possibly are tied to that. One guy has proposed a
perceptual pitch profile. And so on many instruments,
they start making a sound. But at the very beginning,
it’s just noise. There’s no pitch. And as time goes on over
a number of milliseconds or whatever, a pitch
becomes more and more clear. It becomes less and less
random, until a fundamental note comes out with all of
its harmonics above that. And so he wants
to write a profile for defining that
kind of a transition from a note that has no pitch
to a note that has pitch. And I think profiles are
going to be great helpers to do that, and allow us to do
those things in a common way. It’s great for one manufacturer
to innovate and do something. And some manufacturers, their
business is built around, I’m doing. And I’m doing it my own way. And nobody else
can share with me. MIDI was successful
because Oberheim was willing to give
up their system. Sequential Circuits Roland were
willing to give up their system to say, no, let’s
use a common one. And that became the
bigger business. There’s a business model for
doing it a different way. So I’m not I’m not saying
companies shouldn’t do that. But by and large,
the business is going to happen when we agree
to do things the same way. And I think profiles will
be a big part of that audio to MIDI moving ahead, and
being more and more viable, as we do it all the same way,
and learn from each other, and compete with each
other in doing so. We’re going to take
one more question here. And then I think we’re
going to move on. Hey, my name’s Ben, great talk. You mentioned the progression
of sort of a MIDI fork from the five pin connector
to the USB connection. I wanted to know what your
thoughts are on wireless MIDI connection, since it is zeros
and ones, and the feasibility of that. Right, so RTP MIDI is
already doing wireless. You can do wireless
MIDI over RTP. And, in fact, I think Mac
OS X and iOS both support RTP MIDI wirelessly. And there’s also a
new specification that we put out a year
and a half ago now, BLE MIDI, Bluetooth MIDI. And that works
really quite well. And so you’ll find that in
a bunch of products now, wireless MIDI there. Wireless always has
a couple of problems. One is lost packets. And so you need
a protocol that’s going to look after that. And one way you look after
that is by adding latency. And wireless, generally,
is high latency. So while a five pin vin cable
can get a single message across in a third of a millisecond, on
Bluetooth to send one message, it depends on the service
interval of the Bluetooth devices you’re using. And I believe a Mac,
like my MacBook Pro, its service interval, I
think, is 11.5 milliseconds. So my average latency is
going to be 11.5 milliseconds before something happens. So wireless adds latency on
top of the latency that’s already there. If I’m playing a keyboard,
from the time I press a key to the time it
actually gets put on a bus, whether it’s the five pin, or
Bluetooth, or RTP, or whatever, there is some latency
in the keyboard to actually make that happen. And then it arrives
at a synthesizer. Or maybe it arrives at an OS. Like core MIDI gets the
message through a driver. And USB has its
latency, and so on. There’s always latency. And the wireless
latency is added on top of latencies that
already exist in the system. And, really, the wireless
technologies that exist today are really great for things
where real time performance is not critical. And where real
time performance is like playing a keyboard
over Bluetooth, we’re really just on
the verge of not really friendly and playable. But it does work. And for most people, it’s good. A great player can tell the
difference between a USB MIDI connection and a
Bluetooth connection, because the latency is there. Mike, thank you so much. Let’s have another
round of applause for Mike, fantastic talk.

Leave a Reply

Your email address will not be published. Required fields are marked *