Using sox to trim silence out of the middle of a long .mp3

So, one of the things I’ve done lately is to set up two scanners at my Mom’s house. Each scanner listens to one of the CARC Repeaters in Traverse City, MI. Each scanner feeds one side of a stereo input on a machine running Linux.

You can listen to the live-stream result over at www.radioreference.org or www.broadcastify.com if you search for the W8TCM callsign. Or use these links:
http://www.broadcastify.com/listen/feed/17816

Left Channel is fed from the scanner listening to 146.860MHz.
Right Channel is fed from the scanner listening to 442.500MHz.

There are also apps in the IOS Apple App store and Android Google Play store, which are free, that can play these streams for W8TCM (just install the app and search for W8TCM)

Ok, now that we’re done advertising, here’s the techie part.

Here’s how this is configured:

    • Ubuntu 13.10 “Saucy Salamander”
    • darkice is installed, config here:
[general]
duration        = 0        # duration of encoding, in seconds. 0 means forever
bufferSecs      = 5         # size of internal slip buffer, in seconds
reconnect       = yes       # reconnect to the server(s) if disconnected

[input]
device          = plughw:0,0
sampleRate      = 22050     # sample rate in Hz. try 11025, 22050 or 44100
bitsPerSample   = 16        # bits per sample. try 16
channel         = 2         # channels. 1 = mono, 2 = stereo

[icecast2-0]
bitrateMode     = cbr
bitrate         = 32
format          = mp3
quality         = .2            #1.0 is max.
lowpass         = 6500          #cuts frequencies above 5000 Hz
highpass        = 120           #cust freq below 120 Hz
server          = [redacted]
port            = 80
password        = [redacted]
mountPoint      = [redacted]
name            = W8TCM
description     = W8TCM Repeaters, Traverse City MI
url             = http://www.CherrylandARC.com
genre           = Scanner
public          = yes
localDumpFile   = /tmp/scanners.mp3
fileAddDate     = yes
fileDateFormat  = -%Y%m%d_%H%M%S
 
    • crontab: 0 0 * * * /etc/darkice/rotate-audio.sh 
service darkice stop
sleep 1
touch /tmp/yo
sleep 1
service darkice start
sleep 1

    • [edit: there is more to this script, which moves the file to a location other than /tmp]

This is kinda where I left it, which was fine, I was busy. I checked a month later, and there was an issue..:

330M -rw——-  1 nobody nogroup 330M Feb  3 00:00 scanners-20150202_000007.mp3
330M -rw——-  1 nobody nogroup 330M Feb  4 00:00 scanners-20150203_000005.mp3
330M -rw——-  1 nobody nogroup 330M Feb  5 00:00 scanners-20150204_000005.mp3
So, there were files and files, 330 meg each. Each mp3 a 24hour dump of audio.
Now, I get how it would be cool to try to timestamp these with subaudio somehow, so that when I do the next step, I could preserve the timestamp for when things happen.
In fact, I really would love suggestions on how to do this.
For now, however, my disk is disappearing fast. I’d like to be able to archive each year on a DVD-DL at least. so, here’s what I came up with.
  • add another entry to crontab to invoke another script for this process, as we don’t want it to interfere with the restarting darkice part. Besides, it can really be done at any time.
  • Generate a noise profile from the dump file. This is important, as there is AC Hum in varying degrees on each channel, probably due to the age and quality of the scanning receivers. Here is how I generated the noise profile that will be used by our sox command:
    • Open one of the capture files with Audacity. Yes, it takes about 20 minutes to open up a 24-hr .mp3 in Audacity whether you make a copy or not. plan accordingly.
    • Find any chunk of 4-8 minute “silence” in the file, highlight, export, as mp3.Let’s call it “NOISE.mp3” to remind ourselves with all-caps that this could be an improved process.
    • sox NOISE.mp3 -n noiseprof noise.prof
  • TO-DO: Since it’s possible that the noise profile will vary somewhat over time, and generating it is a manual process, what I’d really like to do is make a part of this script that tells sox to find a 5-minute chunk of audio in each file that has no amplitude above a threshold, export it as noise-(datestamp).mp3 and then run this noise profile routine on that silence chunk for each file processed. The result will be a higher quality de-noise process. But I digress, off to saving disk space.
  • sox scanners-20150204_000005.mp3 scanners-20150204_cut.mp3 noisered noise.prof 0.21 silence -l 1 0.1 0.2% -1 1.5 0.2% pad 1.0
    • Let’s break this down.
    • From the sox man page: 
    • silence [-l] above-periods [duration threshold[d|%]
    •               [below-periods duration threshold[d|%]]
    • sox 
      • (source file name) 
      • (target file name) 
      • (command “noisered” for noise reduction) 
      • (noise profile file) 
      • (reduction amount is 0.21) 
      • (command “silence” to use the silence function) 
      • (-l option means to keep the below-periods amount of time intact for each silence period, otherwise known as breaks between chunks of audio separated by silence) 
      • (above-preiods 1 meaning that we trim silence from the beginning until we find non-silence) 
      • (0.1 is our duration, meaning we need 0.1 seconds of non-silence before trimming the previous silence) 
      • (0.2% is our threshold from our noise floor to trigger that there is something other than silence (aka: a signal) present to preserve (not trim).) 
      • (-1 is our below-periods, which when negative means to restart the processing of the silence function, and use the absolute value of -1 (which is 1) as the new above-periods from the restart point. This is the magic sauce making removal of silence in the middle of the file, over and over, possible without using the more-confusion “reverse” function of sox) 
      • (1.5, according to the sox manpage, should be our duration for below-periods, but remember, we restarted the process with -1 which became our new above-periods value of “1”. no matter, the duration here still means that while trimming silence from the middle of the file following a restart of the silence function, we’re going to leave 1.5 sec of what we consider “silence” before we trim again.) 
      • (Threshold is still 0.2%) 
      • (by the way, we want to add a second of silence at the beginning just to make it less jarring when played back. do this with the “pad” command and 1.0 is for 1 second.)
  • On the above, things highlighted with light blue are probably going to need a LOT of tuning. Things like your noise floor, audio quality, static, squelch, interference, sound card, cable quality– all that stuff will affect what you set for these values. The ones in light orange are related to timing of the cutting and what to leave as silence padding. The rest are file-operation specific.
  • Anyway, now with this scripted, I have darkice feeding the broadcastify servers AND making a local mp3 dump. Each night this file is split using crontab, and a new one started. Another set of scripts takes over and “compacts” the file down from a 24-hour mp3 which seem to be over 300MB each to varying-length mp3 files which, so far, depending on how much the systems are used, seem to range from about 9MB to 120MB.
  • Not bad, and saving disk. average is about 25meg per file. Using the 25meg per day average, we’re at about 9.125Gig per year of audio. Is it small enough to fit on a dual-layer DVD? No, not quite. But it’s close. I think I can just schedule offloading to a normal DVD each quarter. Every 3 months. So I didn’t meet my original goal of a DVD of audio per year, but this is still much more manageable.
I need to cite some very handy references for completing the above, as the sox man page alone was great, but just didn’t quite get me there if you know what I mean:
  1. https://gist.github.com/devoncrouse/5534261
  2. http://digitalcardboard.com/blog/2009/08/25/the-sox-of-silence/
  3. http://activearchives.org/wiki/Padding_an_audio_file_with_silence_using_sox
  4. https://www.learnosity.com/blog/2009/11/removing-silence-from-audio-using-sox/
  5. http://comments.gmane.org/gmane.comp.audio.sox/2739
So, the above were extremely helpful in getting me there. I just didn’t find a good, concise, logical breakdown of what was doing what along my journey, and a google search for “using sox to trim silence out of the middle of a long mp3” just didn’t make the mark either. So there you go, for what it’s worth. 🙂
73, Joe N8CN
-jre