Not too long ago I made a simple little web app toy where I can put sound doodles I make. It was great fun to do and I learned bunches about the web audio api.
I wanted to have a waveform preview like soundcloud of each of the sounds. Now, I could precalculate the waveform to an image or raw json file or something but I wanted it to "adapted" based on screen size. I ended up with just calculating everything client side after the page load. It was great! But, as I added more doodles...it got slower and slower.
I thought it was a perfect excuse to dip into using webworkers to improve performance but mainly to remove the jank that happens whilst the waveforms are being loaded.
Web workers are basically js's way of adding multi threading to the browser. Normally, multi threading is a complex and ugly topic but if you read the web worker api, it isn't too bad.
The gist is to have a worker.js file which passes messages between the parent frame. Code in the worker file will run concurrently with the main thread and other workers. I'd highly recommend skimming over the examples here.
Now multi-threading is not a silver bullet for any performance problem as not everything can be done concurrently. It is particularly useful for doing a repetitive task which have a simple input output shape without side effects. A common example you may have heard of is with GPUs. They processed each pixel on your screen all at once. Each pixel needs to be updated and doesn't really care about the pixel next to it.
My use case with waveforms is a perfect example. Each sound file doesn't care about the other and just wants to build a simplified waveform from its data. Nice.
I use react and could use a useWorker hook lib but I wanted to get some experience worked directly with workers.
I had a dedicated function for building the waveform data. This is the thing which is run for every single sound doodle I have. It takes in a url and samples number and spits out an array of numbers. This is by no means a perfect but I was happy with it.
Lets take a look at it.
const buildWaveformData = async (input: { url: string; samples: number }) => {
const { url, samples = 32 } = input;
const audioContext = new AudioContext();
const data = await fetch(url)
.then((res) => res.arrayBuffer())
.then((data) => audioContext.decodeAudioData(data));
const blockSize = Math.floor(data.length / samples);
const filteredData = [];
const left = data.getChannelData(0);
const right = data.getChannelData(1);
let maxVal = 0;
for (let i = 0; i < samples; i++) {
const blockPos = i * blockSize;
let avg = 0;
for (let j = 0; j <= blockSize; j++) {
const lv = left?.[blockPos + j] ?? 0;
const rv = right?.[blockPos + j] ?? 0;
avg += lv + rv;
}
filteredData.push(avg / blockSize);
maxVal = Math.max(Math.abs(avg / blockSize), maxVal);
}
const normal = Math.pow(maxVal, -1);
return filteredData.map((val) => val * normal);
};
Ok, there is a bit going on here so lets break it down.
The first part is just fetching the data from a url and getting out the left and right channel data. I used an audioContext to decode the data.
const audioContext = new AudioContext();
const data = await fetch(url)
.then((res) => res.arrayBuffer())
.then((data) => audioContext.decodeAudioData(data));
.
.
const left = data.getChannelData(0);
const right = data.getChannelData(1);
This part is the slower part. We basically average out the left + right channels based on sample size and push the result into an array.
let maxVal = 0;
for (let i = 0; i < samples; i++) {
const blockPos = i * blockSize;
let avg = 0;
for (let j = 0; j <= blockSize; j++) {
const lv = left?.[blockPos + j] ?? 0;
const rv = right?.[blockPos + j] ?? 0;
avg += lv + rv;
}
filteredData.push(avg / blockSize);
maxVal = Math.max(Math.abs(avg / blockSize), maxVal);
}
That max val you see is for normalizing the result. This helps smooth out the result in case you have something that is loud or quiet.
const normal = Math.pow(maxVal, -1);
return filteredData.map((val) => val * normal);
Ok, but we don't really care about this. All we care about is shoving this function into a worker.
If you haven't yet looked at the worker api....the ergonomics could be better. Specifically what I want to build is something which works this way.
const result = await doSuperComplicatedWork(complicatedWorkFunc, input);
Nice. We could just toss in our existing function and call it good. That is the idea. Oh, and we want typescript typing to play nice. This is similar to how some of the useWebworker libraries look.
After a bit of Googling and reading the api it looks like we can create a web worker form a blob url which means we can stringify our function as a blob url. This should let us take any function and dynamically turn it into a worker.
There a bunch of little gotchas I ran into like forgetting that we actual had to send messages... I know. But, this what I ended up with.
export async function doWorkerTask<T, K>(
workerFunction: (input: K) => T,
input: K
) {
const workFn = workerFunction.toString().replace('"use strict";', "");
const workerString = `
self.onmessage = (e) => self.postMessage(${workFn}(e.data));`;
const workerBlob = new Blob([workerString], {
type: "application/javascript; charset=utf-8",
});
const workerBlobURL = window.URL.createObjectURL(workerBlob);
return (await new Promise((resolve, reject) => {
const worker = new Worker(workerBlobURL);
worker.onmessage = (e) => {
resolve(e.data);
};
worker.onerror = (e) => reject(e.error);
worker.postMessage(input);
})) as T;
}
I took some of this from several examples I found online. Our input is the worker function and its input. We stringify the worker func into a little wrapper around onmessage and postMessage. I think that removal of use strict; someone said was needed for firefox support...idk I but I left it in. Next we create a blob from that string and a worker from that.
Finally, we wrap the actual call with a promise. This lets us use our favorite async and await.
As for typing we take two generics K and T for the input and output. This gives us nice typing.
This is not perfect and there are likely some edge cases this may not work on but I think it is good enough.
Lets try it out.
doWorkerTask(buildWaveformData, input);
And....a bunch of errors....what the heck.
Ok. After 30 or so minutes. I found the issue. At first I thought of some transpiler issue or something but really it was that little async in buildWaveformData. You cannot stringify async the way we do since it creates a wrapper around the actual function call kinda like this.
function buildWaveformData(data) {
return _buildWaveformData.apple(args, something);
}
It kind of is a transpiler thing but it just means the work function cannot be a Promise. It could be possible to make it support async input functions but this is fine.
We use promises are try again....
Ok, fetch isn't there for web workers... That is fine. Xhr is there but honestly we can just pull this out and pass in the data right?
Pulling out the fetching isn't too bad and it does look a bit cleaner.
And...it doesn't work
Workers don't support audio contexts...well fine we can pull out that from the function as well.
This is what our final buildWaveformData looks like
export const buildWaveformData = (input: {
data: [Float32Array, Float32Array];
samples: number;
}) => {
const { data, samples = 32 } = input;
const left = data[0];
const right = data[1];
const blockSize = Math.floor(left.length / samples);
const filteredData = [];
let maxVal = 0;
for (let i = 0; i < samples; i++) {
const blockPos = i * blockSize;
let avg = 0;
for (let j = 0; j <= blockSize; j++) {
const lv = left?.[blockPos + j] ?? 0;
const rv = right?.[blockPos + j] ?? 0;
avg += lv + rv;
}
filteredData.push(avg / blockSize);
maxVal = Math.max(Math.abs(avg / blockSize), maxVal);
}
const normal = Math.pow(maxVal, -1);
return filteredData.map((val) => val * normal);
};
Awesome. It works. Typing was simple too whilst we made the changes. Nice.
I took profiler samples in chromes dev tools for the before and after. The overall load time didn't change much. This is because most of the time we are waiting for the network to load the sound data. Once it did load, we can see that there are multiple worker threads running. This almost eliminate all the blocking time from the main thread. As a result, scrolling the page on initial load is buttery smooth. No more jank. And a nice 20% or so decrease in load time.
We could load the data in the worker with an xhr request. We could skip the blob url loading and just require a file url. We could even be fancy and create a worker pool based on the number of cpu cores instead of creating potentially 100 workers and more.
It isn't the best but it got decent results. I am happy with it.
Workers are easy to use but there are many gotchas around them. For example, if I wanted to offload the animated waveform when you play a sound in the player to a worker, it may end up being slower due to copying the data between worker and main thread. There are always workarounds but they take time implement.
I see web workers primarily as a way to fix jank. It took about 3 hours to refactor everything into a worker and we have a simple worker api which makes it super easy to toss any expensive function off the main thread. If we had fetch and the audio context api, it would have been closer to 15 minutes. Compared to my time writing threaded code in Java, this was painless.