JackTrip Foundation’s Push for Realtime Online Performance

Mike Lawson • ChoralFeatures • October 28, 2020

Share This:

The past couple of issues I have printed stories about companies rushing to fill a need that sprung forth last spring — the opportunity for musicians to perform or rehearse together remotely. The topic is fraught with barriers preventing real-time collaboration. Audio latency issues aside, there are computer needs, audio interfaces, Internet connections, microphones, webcams —a lot of things needed in order to run a remote live session using audio and video.

In addition to the hardware and software requirements are the technology skillsets by both the educator and the student musician. It is a tall order. It isn’t just the skills needed or cost to entry, it is the age-old program of the haves and have nots in districts across the country with its rampant disparity among the socioeconomic demographics schools deal with even in “normal times.” If you are in a better-advantaged district, and have or can get up to snuff quickly on the skillsets needed to create a rehearsal and even live performance opportunity for your instrumental or choral program, you’ll be happy to learn that a non-profit has formed, in partnership with Stanford University, to address the latency issues and even allow your students to participate with cloud-based devices.

I spoke with Michael Dickey, president of the JackTrip Foundation, and Chris Chafe, co-founder of the JackTrip Foundation and professor of music at Stanford University, and former director of Stanford’s Center for Computer Research in Music and Acoustics, to find out more about their plans to solve this problem for music educators and musicians.

Give us the story of how JackTrip started and what the original kind of impetus was for it. Obviously, it wasn’t under pandemic triage conditions, right?

Chris: Yeah, we’re in a very different time. This was a project that was connecting musicians, trans-continentally or city-to-city at great distances and very much oriented towards what people were trying to do. And music composed directly for this kind of new medium at the time, which did have a considerable amount of lag to it because of the geographical distance. Now we are bringing it into a smaller metropolitan-sized region. We reworked the software beginning back in March with lockdowns and music schools being up a creek. That’s when we started to switch things around so that we can focus on extremely tight latencies, kind of what I call ultra-low latency, in support of fast rhythmic performance together. That’s what we’re achieving now.

The JackTrip Foundation’s solution, as I understand it, requires a hardware device, and a wired connection. The idea is to eliminate or greatly reduce the latency from the participants to allow for group rehearsals. Have I summed that up?

Michael: Well, actually, we have multiple offerings. One of which is what we refer to as the virtual studio device, which is the hardware device that you can use to plug into your Ethernet connection on your router, and then use more like a plug and play standalone entity. The other option is software that we offer, so you can still download the JackTrip software and run that directly on the desktop or laptop computer.

The advantage of the hardware device is multi-fold. One is that the device includes a sound card that has a very low latency of about one millisecond. And the other advantage is that it’s just all pre-configured and pre-built for you. There’s a fairly easy getting started guide that steps you through the process of plugging in the different connectors on the device and registering it to a web service. Once you have that done, you’re able to connect the device through a web service to any audio servers that are available to you.

It looks to me like some of the software was still on kind of alpha-beta testing stage?

Michael: Yes, I think for the virtual studio devices in particular, and the web app virtual studio web application, that’s definitely true. JackTrip, the software itself, has been around for over 10 years. And Chris can give you a lot more background on that. But in terms of the virtual studio device and service that we’re using for our choir and band testing, that is very new. And it’s currently only available in a closed beta. But we’re hoping to make that available in open beta to the public within the next few weeks.

Chris: It’s like, we are taking what used to be a telephone call, making the audio quality professional audio level. Making it possible to do multiple channels, not just one for a telephone call, but we often hook up 16 channels studio-to-studio this way. And then bring that high-quality audio down in another way in quality to something which is ultra-low latency, for level in interactivity that we’re talking about. And that has a lot to do with what’s on the JackTrip Foundation technology page right now, if you look at that, for kind of a description of where latency gets eaten up, there’s kind of a budget. If we follow a rule that we kind of discovered in the lab 2005, this kind of tight interactive, fast rhythmic stuff depends on latency from end-to-end, one way that’s not really greater than 25 milliseconds. And to put that in context, a video frame on a television set, the sort of usual 30 frames per second, the time represented by one frame is more than 25 milliseconds is 33 milliseconds. So, we’re talking about these infinitesimal time lags that need to be transmitted geographically, like across the state of California, for example. The solutions were to actually put things into a browser, so that [it’s] running on any device. And what we’re using at Stanford right now runs on your cell phone, typical browser, you bring it up, you connect to the device in the classroom or at home.

Michael: It is the virtual video offerings we’re really intending to address. My background comes from being a board member of the Ragazzi Boys Chorus, in the Silicon Valley Bay Area, and my son’s a member of that. When the pandemic hit, I really started getting involved in trying to figure out solutions. I stumbled across JackTrip and several other projects and did a fairly extensive evaluation of the different technologies that are out there as well as the audio hardware that’s available. And I tried to look at it from the perspective of more of an end-user in their shoes, where you’re either, you know, [a] 12 to 18-year-old boy, or you’re one of their parents, and you’re trying to get this set up for them so that they can participate in their choir rehearsal. And from that perspective, I found pretty much everything out there too hard to use. And so, I started building out technology, in particular, the web application technology for the website, as well as technology on the Raspberry Pi devices to try to make it as easy as I could to get somebody up and running who has little to no technical skills.

And that’s what we’ve been rolling out at Ragazzi, as well as with several other groups, schools, courses, bands, et cetera. The number one problem that we run into is connection quality. We’re only as good as your home internet connection. And there’s nothing we can really do to make your home internet connection faster. So, there is an accessibility issue when you look at a larger population. And what we’ve tried to do is make it as accessible to as many people as possible and really trying to have as low latency as possible by using the device and that one-millisecond sound card is a huge part of that. A lot of computers that people typically have [will have] fairly high audio latencies inherent in them. And that means that the budget that Chris was talking about, they just have less available to spend when they’re trying to use a laptop to join a rehearsal. And so, the devices make it accessible to a wider audience. It’s not perfect, it’s not gonna be accessible to everyone. Because, there’s different limitations and internet availability, especially in the United States, it’s kind of trailing many of the developed countries in the world in that aspect, but we’ve come pretty far. And until last night, we had I think what’s the largest rehearsal been we’ve done so far, which is about 40 people on the same call performing.

Geographically central, though they were all in the same metro area.

Michael: Yes. All within the San Francisco Bay Area.

The geographic part is really important because of the hops involved for a person’s data packets from their house to, I assume, a server farm where your server-side application is, right?

Michael: Yes, and this particular use case, we are using a server, referred to as a hub server that does the mixing of all the audio channels together. There’s two general models, one is the audio server model, the other is the point-to-point or peer-to-peer model where everybody is connecting directly to one another. JackTrip’s software actually supports both models, but with the virtual studio stuff that we’re using for larger groups. We use the client-server model and the advantage of the client-server model primarily I think, is that you can have more people. When you’re connecting directly to somebody you run into a limit somewhere in that like six to 10 range. And of course, it just depends on internet connections. If everybody has a great fiber gigabit internet connection, you could probably handle more people. But from what I’ve seen, even just a few people playing together on p2p is fairly challenging for home internet. So, the server allows you to utilize the minimal amount of bandwidth crossing your home internet connection. And that makes it work for more people, but it also makes it work for large groups.

Let’s talk about the system requirements and also the great prevalence of Chromebooks. And how that will play into this because we talk about the haves and the have nots when it comes to bandwidth. Where does JackTrip fit in that in terms of the minimum bandwidth with a wired connection, and the minimum system requirements?

Michael: It’s kind of an easy rule of thumb to think of one megabit per channel per direction. But when you run a speed test on your home network, you’re gonna get some numbers, typically, in the USA, it’s gonna be asymmetrical, sort of larger bandwidth downs and [smaller] going back up. And the number that you read is not indicative of the actual quality that you need to achieve in order to sustain an even flow of the data packet. I think the beauty of what Mike’s talking about here, and all of the hardware-based sector solutions — they are operated from a browser. Chromebook browser is gonna be great for connecting and controlling the end-users participating in an ensemble. You can even use an iPhone or iPad or, you know, a mobile device is fine. The idea is that you don’t need a separate computer at all in order to use it (when using the JackTrip device), you just need to have some device or computer that has a browser.

How does the hardware fit in, and will it connect to a Chromebook?

Michael: No, actually it doesn’t. The devices are completely standalone computers. They’re full-blown computers themselves. And so, you don’t need to connect them to any external computing device, you just need to connect them to your Ethernet network. And then any device that’s on that network, whether it’s through Ethernet or Wi-Fi, you should be able to use its browser to manage them.

JackTrip Foundation is a non-profit. Tell us about that.

Michael: You know, we’re a very new organization. It’s a public charity, nonprofit. But we’re in the process of filing for our 501(c) (3) certification. It takes a few months, and we just formed it a couple of weeks ago.

As 501(c)(3), you’ll be on a mission, you know, and have a mission goal to fulfill. So, what exactly will that mission statement look like as a foundation and a nonprofit?

Michael: I keep getting asked, “what’s my role in all of this?” You know, “what do I want out of it?” And it’s absolutely just, let’s get music ensembles back together and lessons, working at high quality get through this period. So, it’s an urgent mission — we’ve been doing this on several fronts in order to just maximize the effect. And the feedback is great, when you get the initial response of “I’ve been waiting for this for six months,” then we’re there. And there’s another side to this, which is concert casting as well. So, ensembles, you know, being able to translate, interactive, ensemble work going on at home-to-home, be able to combine that with good video production. And then streaming add on like Vimeo live or YouTube live to audiences that’s another aspect that we’ve been working on for the six months.

Well, that’s a critical component of the music standards, the national music standards, it’s not just, you know, learning to play and read it’s actual performance of the pieces. And the huge part of the financing and support these programs get are from the parents and the grandparents and, you know, they probably attend the concerts and support the band programs and choir programs to boost donations and things of that sort. So, taking away the performance opportunity, as we’re faced with at this moment is potentially devastating. What does the foundation envision the cost of participation for students and a teacher or district to participate and how will that roll out? Will it be by the school seat license or per student?

Michael: Yeah, I think there’s multiple aspects to it. One is that we do offer free and open-source software that anyone can download and use on their laptops. They can provision their own servers they don’t have to spend anything with us to do that. Now, in order to fulfill our mission, we wanna make it easy and as easy as possible for as many people that are out there they can afford more. And in that sense, we have the device. So, the device, it’s not free. It has advantages to it in terms of getting started. And it has advantages in terms of lowering the latency. But it does have a cost, which we’re looking to make available fairly soon, at about $150 per device. And that doesn’t include the kind of cables, power supply, [or other] kind of connectors and stuff that you may need. So that could add up to an additional $50 per device. And for the service, you can certainly run your own server. If you’re a school, [or] university, you probably have a data center, and probably already have IT that you can work with to run your servers.

Even a local high school might have the ability to run a server themselves. And so that’s an option in which case, you’re just paying for the server time. If you don’t have that, we also offer fully managed servers to certain geographical locations which are, of course, leveraging cloud computing. So, there’s certain locations that that’s available in and we’ll be expanding that over time. But for that service, we’re looking to charge as little as possible, probably on the order of about $1 per hour per 10 musicians. It’s not [that] you need to be familiar with running a server. You probably have an IT person or somebody. And hopefully, like, that’s kind of what we look for within different organizations is we recommend you know who the person is who helps you with setting up your internet or setting up your local printer or something. Try to find a tech person in your community.

I went to the JackTrip page, and saw it was for older versions of Mac OS and that the new ones aren’t out yet. Is there a date coming when there’ll be a client app on there that’s ready to use?

Michael: We are definitely coming.

Chris: This is a project that kind of achieved over the decades, kind of like code maturity and sort of sat there until six months ago. And at that point, a lot of people jumped in, and a lot of you know, upgrading was going on, you know, right out of the starting date, if you wanna say like back in March. Catalina was a big issue, but it got solved, you know, within a month or so.

Michael: I would have to categorize or characterize the period that we’ve been in as the alpha testing period. So, we’re really breaking things fast and trying to learn from that and iterating on it frequently. However, you know, we do have choruses, bands, everything, who are performing together, we’re using it several times a week now. The next phase would be the traditionally the beta phase where you know that it’s not…it’s gonna have some issues, it’s gonna have bugs, there’s gonna be challenges, potentially getting up and running, in particular, if you have large groups. But we think that we’ve come far enough along that it’s pretty stable, and we’ve been very… we’ve seen a lot of success with it. And we’re gonna just keep iterating and harden it as we go.

Is there a target date for the beta, and is it possible you’ll be looking for a number of public-school music directors to participate?

Chris: Yeah, the date that was put out there was early October. But we’re charging forward to get it out there as quickly as we can.

Can we can tell our readers, “Go to the website and register to take part in the beta” ?

Chris: Yes.

More info at jacktrip.org

The Latest News and Gear in Your Inbox - Sign Up Today!