Coding a Phase Vocoder for Tempo Changing and Pitch Shifting in Audio Applications

One of my mathematics lecturers once invited us to solve a problem that has fascinated me ever since. If you create a digital sound clip by recording your own voice, how can you transform that to mimic the celebrated Norwegian weather forecaster Vidar Theisen?

theisen_n

For those of us never having heard about dear Mr. Theisen, or even wasn’t born before he passed away, you can watch one of his daily NRK weather forecasts below.

It is pretty obvious that Mr. Theisen had a very distinct way of talking. A voice of slow tempo accompanied by a monotonic pitch. Nobody, at least at that time, would be unsure who you pretended to be if you slowly and flat-voiced said something about the weather at Spitsbergen.

The lecturer also happened to be my adviser, and his name is Prof. Hans Munthe-Kaas. Hans is a very good man, and he has opened many doors for students that has knocked on his door. He is on my list of personal relationships I could not have been without.

To get the wheels rolling in his students heads, Mr. Munthe-Kaas outlined several incorrect strategies you could try to begin with: to slow things down by a factor two, try to repeat the digital samples one-by-one so that the net duration will be doubled. Or even better, try to interpolate?

It turns out that both strategies actually works in terms of doubling the duration, but has the inevitable side effect of also shifting down the pitch of audio signals. It is analogous to slowing down the rotational speed of a record player: it obviously decreases the tempo of the sound, but the tone will also be darker. If you happen to own a record player, you are probably nodding affirmative at this point. In other words, we approach Mr. Theisen in terms of tempo, but not by tone. For pure sinusoidal waves, what we want is to extend the number of periods rather than increasing the length of one period.

Red leaf

Virtually, the tempo and pitch of a tune is inseparable. If you change the tempo, the pitch will also change. The figure on the right illustrates the concept: modifications can seemingly only be done along the green diagonal line.

Until the mid 60s, this was accepted as the truth. Then suddenly a paper disturbed the community; two bright guys named Flanagan and Golden solved the problem in the context of telecommunication. Thanks to it, we can separate the two parameters. Controlling the Theisen tempo independently of the tone is now easy. And the invention’s name? The Phase Vocoder.

According to the theory, and some years of trial and error for my part, if what you want is to change the tempo of a signal by a factor Pt/Qt, and shift the pitch by a factor Pp/Qp, all you have to do is the following:

  1. Transform the signal over to the time-frequency plane using the Short-Time Fourier Transform (STFT).
  2. Linearly interpolate the magnitude of the STFT by a factor Pt/Qt*Qp/Pp across the time dimension.
  3. Go over the phase of the STFT and make sure that the partial time derivative across the time dimension is preserved.
  4. Transform the modified STFT back into the time domain
  5. Re-sample the resulting waveform by a factor Qp/Pp to get the final waveform.

We definitely need some test data to try this out. I have recorded my own voice saying the following in Norwegian:

“Som vi ser har det blitt kuldegrader over omtrent hele landet”.

Translated to English this corresponds to something like: “As you can see, we now have temperatures below zero degrees across the whole country”. You can listen to the original sound clip in the player below.

This Matlab code implements the Theisen Transform following the step-by-step Phase Vocoder recipe above:

function y = theisen(x)
%
% Geir K. Nilsen, 2006
% geir.kjetil.nilsen@gmail.com
%

Ns = length(x);
Nw = 1024;  % Window width
hop = Nw/4; % Overlap width
Pt = 2; 
Qt = 5;
Pp = 5; 
Qp = 6;
r = Pt/Qt * Qp / Pp; % Tempo-pitch shift factor

win = hann(Nw, 'periodic');

Nf = floor((Ns + hop - Nw) / hop);

fframe = zeros(Nw,Nf);
pframe = zeros(Nw,Nf);

% Step 1: STFT
c = 1;
for i = 0:hop:((Nf-1)*hop);
    fframe(:,c) = 2/3*fft(x(1+i:i+Nw).*win');
    c = c + 1;
end;

% Step 2 & 3: Linear interpolation & phase preservation
phase = angle(fframe(:,1)); 
c = 1;
x = [fframe zeros(size(fframe, 1),1)]; 
for i = 0:r:Nf-1;                        
    x1 = x(:,floor(i)+1);                
    x2 = x(:,floor(i)+2);
    
    scale = i - floor(i);
    mag = (1-scale)*abs(x1) + scale*(abs(x2)); 
       
    pframe(:,c) = mag .* exp(j*phase); 
    c = c + 1;
    
    phase_adv = angle(x2) - angle(x1); 
                                       
    % Accumulate the phase
    phase = phase + phase_adv;
end;

% Step 4: synthesize frames to get back waveform. Known as the Inverse
% Short Time Fourier Transform.
c = 1;
Nf = size(pframe,2);
Ns = Nf*hop - hop + Nw;
y = zeros(1,Ns);

for i = 0:hop:((Nf-1)*hop);
    pframe(:,c) = real(ifft(pframe(:,c)));          
    y(1+i:i+Nw) = y(1+i:i+Nw) + pframe(:,c)'.*win'; 
    c = c + 1;
end;

% Step 5: finally resample the waveform to adjust according
% to the pitch shift factor.
y = resample(y, Pp, Qp);

When applied to the original recording, the result is quite impressing. Listen to the modified sound clip below and judge yourself.

For a practical C# implementation, see my Github repository.

The Resurrection of the Computer Fridge

img_20190731_1837407130935668874283186.jpgThe computer fridge hypothesis is a seemingly brilliant idea: store and operate your computer in a fridge, and wave good bye to all heating issues.

However — in 2015, a disappointing video appeared on YouTube: Linus Tech Tips once and for all rejected the computer fridge hypothesis by an experiment showing that refrigerators always loose in thermodynamic battles against modern computers.

The heat dissipated by the personal computer will exceed the heat absorption capacity of the fridge evaporator — causing the ambient temperature in the fridge to rise rather than fall. If we consider the system as a whole, we have a serious self-heating problem, and there is nothing we can do about it.

Or is it?

Cracking The Self-Heating Problem

Pleasant ideas deserve to be treated with respect, right? They should be tried from every angle before being trashed. From first principles, I decided to challenge the rejection of the computer fridge hypothesis by proposing: the ventilated computer fridge hypothesis!

The Ventilated Computer Fridge Hypothesis

img_20190808_125029.jpg
A Ventilated Computer Fridge

The ventilated computer fridge hypothesis is based on the following reasoning: If the fridge is equipped with a balanced ventilation system, most of the heat dissipated by the computer will be transferred out of the fridge, so there will be no self-heating. When the fridge is turned off, the system will simply function as a normal computer cabinet. Now, if we in addition route the in-air stream via the evaporator, the air will be cold when it hits the computer, thus absorbing heat from the computer on the way out of the fridge. See the sketch on the right.

I decided to test the new hypothesis, so I had to build my own computer fridge.

Building a Computer Fridge

I bought a second hand mini-fridge with a glass door from TempTech on eBay as a starting point.

Installing Balanced Ventilation

bb138b1h3790128944658168010.jpgI decided to install a simple balanced ventilation system represented by four Corsair ML140 Pro cabinet fans.

While the task of installing four fans in a mini-fridge is not the most pleasant thing a human being can do in the world — it can be done by some laborious drilling on the outside, and heavy angle grinding on the inside.

My friend Øyvind Colliander created a nice template for the drilling part. See the fan hole template below. The trick is to tape a printed template to the fridge wall, and use a punch and a hammer to mark the center of the holes before drilling. For the angle grinding, put on a face mask and cut out four squares on the inside aligned with the holes on the outside.

Result: two fans on the upper back (in-air), and one fan on each side (out-air).

IMG_20190806_100327
Fan holes on the sides
IMG_20190806_100311
Fan holes on the back
holepattern
Fan hole template for Corsair ML140 Pro (A4 print-friendly PDF)

Bypassing the Thermostat & More

In order to get full control of the fridge’s cooling system, I bypassed the internal thermostat and instead installed an Arduino micro controller and an external relay board beneath the back plate on the inside of the fridge. I also permanently installed the Arduino DHT-11 sensor to get ambient temperature and humidity readings. The sensor is placed in the bottom right corner of the fridge. I drilled holes on the backside for 230 VAC power and Ethernet, and installed a power socket, as well as two AC/DC converters (36 VDC & 12 VDC) on the inside.

Lighting and a Retractable Shelf

At this point, the fridge was ready to be tested, but for fun I decided to take the concept just a few steps further… Some lighting is nice, right? And — uh, a motorized retractable shelf for the computer! The latter turned out to be more challenging.

Basically, I used FreeCad to design a rack-and-pinion based retractable shelf. The design consists of three main parts; the rack, an old electric parabola antenna motor with a pinion, and a clamp to fix it to the shelf. Thanks to Andre Böehme for helping out with the toughest part — namely modelling the pinion and a matching rack tooth profile. I used i.materialise to 3D-print the parts. Great service!

test
Retractable Shelf Design

ejectTo control the position and direction of retraction of the shelf, I combined two of the relays on the external relay board to achieve a polarity reversal switch.

The parabola antenna motor also has a built-in pulse counter circuit, which can be used to pinpoint its exact angular position. Basically, the circuit sends out short square pulses synchronized with the pinions angular movement. Position control is thus achieved by simply counting pulses with an analog input channel on the Arduino.

See below for a full (sloppy) wiring diagram of the system.

IMG_20190807_103728

Installing the Computer

img_20190731_1837407130935668874283186.jpg

I installed the computer hardware on a Lian Li test bench placed on top of the retractable shelf in the fridge.

Computer Specifications: 

  • ASUS ROG STRIX B450-E GAMING Motherboard
  • AMD Ryzen 5 2600 Wraith Stealth
  • Corsair Vengeance LPX DDR4-3200 C16 BK DC – 32GB
  • Samsung 970 EVO Plus SSD M.2 2280 – 500GB
  • ASUS GeForce RTX 2080 Ti ROG STRIX OC – 11GB GDDR6 RAM
  • Corsair 1000W PSU

Computer Fridge Code

The software I developed for the computer fridge basically consists of three parts;

  • An embedded Arduino back-end coded in C (deepfridge.ino)
  • A Python CLI frond-end to control the whole system (deepfridge.py)
Screenshot from 2019-08-09 11-47-03.png
Deep Fridge CLI
  • A set of Python test scripts with data logging functionality to conduct experiments (tests/deepfridge_logger_test*.py)

The code is available on Github: https://github.com/gknilsen/computer_fridge

The Computer Fridge Put at Test

Let’s run a series of tests to see if the computer fridge works. We will use the test scripts tests/deepfridge_logger_test*.py included in the Github repository. The test scripts log the following parameters to CSV files;

  • Fridge temperature  (by deepfridge.ino / Arduino DHT-11 sensor)
  • Fridge humidity (by deepfridge.ino / Arduino DHT-11 sensor)
  • CPU temperature (by calls to lmsensors / integrated CPU thermistor)
  • CPU utilization (by calls to cpu_usage.sh which is based on calls to top)
  • GPU temperature (by calls to nvidia-smi / integrated GPU thermistor)
  • GPU utilization (by calls to nvidia-smi)

Some of the tests put the CPU and/or GPU to maximum utilization under certain periods — this is achieved by calls to stress and gpu-burn.

Test #1 — Computer CPU & GPU at Idle with Cooling

Description: The computer CPU & GPU is left at idle. First run for 5 minutes when the compressor is OFF to get a baseline, then the compressor is turned ON, and the logging continues for 55 minutes. Total test time is 1 hour (3600 seconds).

Test script: tests/deepfridge_logger_test1.py

Log file: tests/test1.csv

Result:

Test_1

Conclusion: Running the compressor for about an hour results in a CPU & GPU temperature drop of 5 degrees Celsius, when the computer is at idle. Also the ambient fridge temperature is reduced by 5 degrees Celsius.

Test #2 — Computer CPU at Maximum Utilization without Cooling

Description: The computer CPU is left at idle the first 5 minutes, and the compressor is OFF, to get a baseline. Then the computer CPU is put on maximum utilization for 5 minutes. The computer CPU is then left at idle for 5 minutes more minutes. Total test time is 15 minutes (900 seconds).

Test script: deepfridge_logger_test2.py

Log file: tests/test2.csv

Result:

Test_2

Conclusion: Peak CPU temperature under max utilization is at 53.7 degrees Celsius.

Test #3 — Computer CPU at Maximum Utilization with Cooling

Description: The compressor is set ON, and the computer is left at idle the first 45 minutes. Then the computer CPU is put at maximum load for 5 minutes, followed by 10 more minutes at idle. Total test time is 60 minutes (3600 seconds).

Test script: deepfridge_logger_test3.py

Log file: tests/test3.csv

Result:

Test_3

Conclusion: With constant cooling for about one hour & max CPU utilization for 5 minutes, again the CPU temperature is reduced by 5 degrees Celsius. Also the ambient fridge temperature is reduced by 5 degrees Celsius.

Test #4 — Computer GPU at Maximum Utilization without Cooling

Description: The computer GPU is left at idle the first 5 minutes, and the compressor is OFF, to get a baseline. Then the computer GPU is put at maximum utilization for 5 minutes. The computer GPU is then left at idle for 5 minutes more minutes. Total test time is 15 minutes (900 seconds).

Test script: deepfridge_logger_test4.py

Resulting log file: test4.csv

Result:

Test_4

Conclusion: Peak GPU temperature under max utilization is at 66 degrees Celsius.

Test #5 — Computer GPU at Maximum Utilization with Cooling

Description: The compressor is set ON, and the computer is left at idle the first 45 minutes. Then the computer GPU is put on maximum load for 5 minutes, followed by 10 more minutes at idle. Total test time is 60 minutes (3600 seconds).

Test script: deepfridge_logger_test5.py

Log file: tests/test5.csv

Result:

Test_5

Conclusion: With constant cooling for about one hour & max GPU utilization for 5 minutes, the GPU temperature is reduced by 3 degrees Celsius. The ambient fridge temperature is reduced by 4 degrees Celsius.

Conclusion

The tests show that there are no signs of self-heating, and that the CPU/GPU temperature drops 3-5 degrees Celsius when the cooling system has been activated for about an hour.

The ventilated computer fridge hypothesis has been confirmed!

However, is a temperature drop of 3-5 degrees Celsius worth the effort? Well, it depends. If you play around with overclocking and want to squeeze the most out of your system, 3-5 degrees Celsius might be enough. But is it really worth the effort? The system complexity is overwhelming, costly and not especially effective in its current state. However, the project was a great learning experience, and it was really fun! I doubt that I will run the cooling system much in its current state on a daily basis though.

There is plenty of room for optimization. In fact, currently I have not tested to see what happens if the cooling system remains activated for a longer period than one hour. Same is true for extended periods of maximum CPU/GPU utilization.

Please leave your thoughts in the comments section below. I am happy to receive ideas which can potentially improve the cooling performance of the system.

Melted Butter Cocoons – Part II

Short story: butter coccoon dry-aging attempt #1 as described in Melted Butter Coccoons – Part I was unsuccessful. The resulting steaks came out not rotten, but cigar-close to it.

Two key take-aways are

  1. Get hold of a fridge which is stable at 1 degree Celsius maximum.
  2. Get hold of meat which is guaranteed to not have undergone any form of prior wet aging.

To be honest, I have lost a bit of interest in this project, so I am not sure if there will be a second attempt. I have another project related to my PhD program in deep learning involving the dry-aging fridge (which obviously was not suitable/cold enough for this exercise). More on the new project soon!

Melted Butter Cocoons – Part I

IMG_20181027_162834
Melted Butter Cocoons

I still remember the intense flavor and juiciness of that thick piece of Sirloin. Crusted in the name of Kona, Bone-In and Aged Dry. It was an exceptional dining experience at the Capitol Grill in Houston, Texas. It was the event that gave me the inspiration for this two-part blog post.

In this first part, I will introduce you to my little home project where I try to dry age Sirloin steaks at home. In part two, we will see the results. When? Well, in about 60 days my friend — when the meat is — say, middle — aged.

Dry Aging TL;DR

Dry aging of meat refers to the process of storing meat over a long period of time in a dry environment. This is different from traditional wet aging, where the meat is kept in a vacuum bag over a shorter period of time. Both ideas aim for increased tenderness, intensified flavor — not to mention the prolonged preservation time, which in fact is a common denominator behind many of the world’s culinary specialties. The Rakfisk of Norway is one golden example of this: trout is kept in a bin with sugar, salt and water. A controlled form of rotting process starts, and you can consume the fish over the course of months. One side effect is some very special taste and — uhm — smell properties! Same is true for many other specialties as well.

Dry aging is rooted in the same bio culinary paradigm as sour dough: you want your bio climate to be meat (or dough) friendly, and develop to the better over time. A good dry aging fridge (or room) can develop to the better over the years. It will be a unique bio climate colored by the natural microbiota in that environment.

Instead of going on to explain all the fancy sides of dry aging, I point you to Jess Pryles blog. Read on and you will learn everything you need to know. You will also notice a crucial “fact”: Dry aging is not for the home chef.

Why?

Because if you slice your meat into individual steaks before you proceed, they will basically come out as cured ham, eventually — dry and hard. Point is not to cure the meat! Original idea is to remove the outer dried layer before cooking and consumption. If you dry age individual steaks, they will be dry all the way through. Traditionally, dry aging calls for whole cuts of meat, and you will loose something like 5-10% from removing that outer dried layer after aging. So, either you go buy that large whole cut of meat, and accept the loss, or you stick to that boring wet aging, is it right?

Not quite! It turns out that Casper Stuhr Cobzcyk has developed a new dry aging technique where he first dips his (still) whole cuts of meat in melted butter to form a cocoon protecting the steak from drying out on the surface. It reminds me about cheese waxing!

If you combine the thoughts of Pryles and Stuhr Cobzcyk, I figured it would allow for dry aging of individual steaks at home, since the melted butter cocoon will protect the individual steaks from drying out at the surface, and, hence, you will not need to remove the outer layer nor will they be cured!

Butter probably acts like sort of a “let-moist-out, not-let-oxygen in”-barrier. This in turn, brings up a lot of questions Mr. Stuhr Cobzcyk does not answer. He probably has them, but why should he really share them with the world. For example, what is the optimal thickness of the butter cocoon? Unfortunately, I don’t know all the answers yet.

I decided to put strings around my two steaks, before dipping them in. This allowed me to hang them in the fridge rather than letting them sink into the rack and potentially create a butter mess.

Dry aging started: 27th of October, 2018.

We aim for 60 days, so dry aging stops: 26th of December, 2018.

A Double Thank You

I was reading at the library when I got a phone call from my boss. She asked me to immediately return to home because of a not-so-unusual & escalating disobedient kids situation.

I got onto my feet and started to pack my bag. I eyed the cord of my laptop charger, the way it snaked itself over the floor, and ended up in a socket overly close to the feet of a random lady.

I approached her and gesticulated it would require her attention to resolve the situation. She read it and started to unplug my charger but was not able to get it out. She gave up quickly, and I had to help her. And so I successfully managed to unplug it under grant of an informal intimate zone intrusion permission. Once unplugged, I said, “Thank You”, and she responded “Thank You”.

It was an interesting Double Thank You for the help situation: I thanked her for permitting me to enter her intimate zone, and I thanked her for her initial failed attempt of helping me. Then she thanked me back for being polite and helpful enough to help her when helping me.

Fat Man & Little Boy

On a sunny spring day, I craved for steak and went to see my good old barbecue on the patio. But I found it ill-conditioned. The rain in Bergen had finally taken its toll on the rusty little boy: the price of storing and using it outside during winters now had to be paid. So I questioned myself whether to buy a new one a larger and better one. Filled with ambivalence, I went to the retailer and found a big fat man. Upgrading is a bliss. Ka-ching.

Some years ago I used to work for an oil & gas company. In that context — I visited a site we had some cross-business activity with — located in Houston, Texas. Upon arrival I was told to settle in the empty cubicle of another engineer on vacation. On the cube wall I noticed a drawing showing a comparison between two versions of the site’s main product: the legacy 15k psi ram BOP (Blow-Out Preventer), and the next 20k psi generation. Let me add that the size of such equipment does not grow linearly with the pressure rating, so the next version was a fat man compared to the little boy.

Typically, when bad oil spill disasters take place, the BOP is likely to be in the spot. The BOP acts as the main security system resting on top of the very well head. They are used to seal wells under normal conditions, as well as in emergency situations. In case of the latter, the BOPs have strong hydraulic shear rams fitted with sharp steel blades. They can cut through any solid obstacles (e.g. drill pipe) situated — intentionally or not — in the bore. The aim is to seal the well and prevent oil spill, or even blow-outs on the rig deck — usually leading to fire, havoc and death.

I noticed, printed with small letters beneath each of the two drawings; “Little Boy”, “Fat Man”. It took me a couple of seconds before I realized the meaning. Call it irony, sarcasm or blasphemy — the reference was to the respective nukes detonated by US over Hiroshima and Nagasaki in 1945.

It also got me thinking about the two I now have on the patio back home.

Fat Man & Little Boy

Device Equals ANSI Dot Sys

tuxbsod3
Click here to download the above image in android wallpaper format!

As a kid I figured out how to edit my father’s config.sys and autoexec.bat to change font colors in MS-DOS. Without the help from the internet, it was such a mysterious age where small technological discoveries were made based solely on trial and error, gossip, obscure scripts, tools and white papers spread on LAN parties and floppies.

The trick was to load the binary file ansi.sys using the DEVICE= directive in config.sys, enabling escape sequence interpretation to the standard output — which included means for changing the background and foreground font color on the command prompt.

I later installed Red Hat Linux 5.2 in 1998, but after a while I ended up compiling my own “distro” based on Linux from Scratch. My favourite nostalgic piece of memory is still my only ANSI art creation to date, namely my /etc/issue (image shown above), born based on piping and overlying Tux from Welcome2L (c) LittleIgloo.Org on a classical Windows 9x blue screen of death replica!

UPDATED 17/8-18: Added Android wallpaper version of BSOD Tux.