syncing mp3 files to the cloud!

So, one of the disappointments I have found with broadcastify / radioreference streaming is the lack of being able to listen to the archives of the stream.  (of course, by “disappointment” I mean “darn, I wish it was free.” I completely understand the need for the company to make money, and the need to offer “premium services” to achieve that. I digress.)

Since I’m generating mp3 dump files locally, and I’m trimming silence, why not pump these to the cloud into a shared folder?

Enter Copy by Barracuda to solve my issues. There’s a Linux version of the end-user client program.

 

Setting it up was trivial, I downloaded Copy.tgz and expanded it into a folder.

I made my synchronized folder “~/Copy” for simplicity.

For further simplicity, I moved the x86 binary folder to ~/.copy/ where config is kept (also a good thing to note)

Once it’s set up, invocation is simple. I’m running it live currently, under a screen session, and will probably daemonize it soon. For now, on a screen session, I invoke with:

~/.copy/CopyConsole -root=~/Copy -username=blah@blah.com

 

So, now after midnight, I changed the “dump path” of the finished file from the previous day to go to ~/Copy/Scans/2015/ where it is then detected, and synchronized to “the great big cloud in the sky.”

 

check it out here: http://tinyurl.com/W8TCM

You’ll find a 2014 and a 2015 directory. There’s also a Misc directory (empty as of now) because I had some files I didn’t datestamp.

 

Sure, this isn’t as elegant as whatever time-stamped version of the “scanner wayback machine” that broadcastify uses, but it works and it didn’t cost me anything, and I own the archives. Always handy incentives. 🙂

 

73

Joe N8CN

 

 

Using sox to trim silence out of the middle of a long .mp3

So, one of the things I’ve done lately is to set up two scanners at my Mom’s house. Each scanner listens to one of the CARC Repeaters in Traverse City, MI. Each scanner feeds one side of a stereo input on a machine running Linux.

You can listen to the live-stream result over at www.radioreference.org or www.broadcastify.com if you search for the W8TCM callsign. Or use these links:
http://www.broadcastify.com/listen/feed/17816

Left Channel is fed from the scanner listening to 146.860MHz.
Right Channel is fed from the scanner listening to 442.500MHz.

There are also apps in the IOS Apple App store and Android Google Play store, which are free, that can play these streams for W8TCM (just install the app and search for W8TCM)

Ok, now that we’re done advertising, here’s the techie part.

Here’s how this is configured:

    • Ubuntu 13.10 “Saucy Salamander”
    • darkice is installed, config here:
[general]
duration        = 0        # duration of encoding, in seconds. 0 means forever
bufferSecs      = 5         # size of internal slip buffer, in seconds
reconnect       = yes       # reconnect to the server(s) if disconnected

[input]
device          = plughw:0,0
sampleRate      = 22050     # sample rate in Hz. try 11025, 22050 or 44100
bitsPerSample   = 16        # bits per sample. try 16
channel         = 2         # channels. 1 = mono, 2 = stereo

[icecast2-0]
bitrateMode     = cbr
bitrate         = 32
format          = mp3
quality         = .2            #1.0 is max.
lowpass         = 6500          #cuts frequencies above 5000 Hz
highpass        = 120           #cust freq below 120 Hz
server          = [redacted]
port            = 80
password        = [redacted]
mountPoint      = [redacted]
name            = W8TCM
description     = W8TCM Repeaters, Traverse City MI
url             = http://www.CherrylandARC.com
genre           = Scanner
public          = yes
localDumpFile   = /tmp/scanners.mp3
fileAddDate     = yes
fileDateFormat  = -%Y%m%d_%H%M%S
 
    • crontab: 0 0 * * * /etc/darkice/rotate-audio.sh 
service darkice stop
sleep 1
touch /tmp/yo
sleep 1
service darkice start
sleep 1

    • [edit: there is more to this script, which moves the file to a location other than /tmp]

This is kinda where I left it, which was fine, I was busy. I checked a month later, and there was an issue..:

330M -rw——-  1 nobody nogroup 330M Feb  3 00:00 scanners-20150202_000007.mp3
330M -rw——-  1 nobody nogroup 330M Feb  4 00:00 scanners-20150203_000005.mp3
330M -rw——-  1 nobody nogroup 330M Feb  5 00:00 scanners-20150204_000005.mp3
So, there were files and files, 330 meg each. Each mp3 a 24hour dump of audio.
Now, I get how it would be cool to try to timestamp these with subaudio somehow, so that when I do the next step, I could preserve the timestamp for when things happen.
In fact, I really would love suggestions on how to do this.
For now, however, my disk is disappearing fast. I’d like to be able to archive each year on a DVD-DL at least. so, here’s what I came up with.
  • add another entry to crontab to invoke another script for this process, as we don’t want it to interfere with the restarting darkice part. Besides, it can really be done at any time.
  • Generate a noise profile from the dump file. This is important, as there is AC Hum in varying degrees on each channel, probably due to the age and quality of the scanning receivers. Here is how I generated the noise profile that will be used by our sox command:
    • Open one of the capture files with Audacity. Yes, it takes about 20 minutes to open up a 24-hr .mp3 in Audacity whether you make a copy or not. plan accordingly.
    • Find any chunk of 4-8 minute “silence” in the file, highlight, export, as mp3.Let’s call it “NOISE.mp3” to remind ourselves with all-caps that this could be an improved process.
    • sox NOISE.mp3 -n noiseprof noise.prof
  • TO-DO: Since it’s possible that the noise profile will vary somewhat over time, and generating it is a manual process, what I’d really like to do is make a part of this script that tells sox to find a 5-minute chunk of audio in each file that has no amplitude above a threshold, export it as noise-(datestamp).mp3 and then run this noise profile routine on that silence chunk for each file processed. The result will be a higher quality de-noise process. But I digress, off to saving disk space.
  • sox scanners-20150204_000005.mp3 scanners-20150204_cut.mp3 noisered noise.prof 0.21 silence -l 1 0.1 0.2% -1 1.5 0.2% pad 1.0
    • Let’s break this down.
    • From the sox man page: 
    • silence [-l] above-periods [duration threshold[d|%]
    •               [below-periods duration threshold[d|%]]
    • sox 
      • (source file name) 
      • (target file name) 
      • (command “noisered” for noise reduction) 
      • (noise profile file) 
      • (reduction amount is 0.21) 
      • (command “silence” to use the silence function) 
      • (-l option means to keep the below-periods amount of time intact for each silence period, otherwise known as breaks between chunks of audio separated by silence) 
      • (above-preiods 1 meaning that we trim silence from the beginning until we find non-silence) 
      • (0.1 is our duration, meaning we need 0.1 seconds of non-silence before trimming the previous silence) 
      • (0.2% is our threshold from our noise floor to trigger that there is something other than silence (aka: a signal) present to preserve (not trim).) 
      • (-1 is our below-periods, which when negative means to restart the processing of the silence function, and use the absolute value of -1 (which is 1) as the new above-periods from the restart point. This is the magic sauce making removal of silence in the middle of the file, over and over, possible without using the more-confusion “reverse” function of sox) 
      • (1.5, according to the sox manpage, should be our duration for below-periods, but remember, we restarted the process with -1 which became our new above-periods value of “1”. no matter, the duration here still means that while trimming silence from the middle of the file following a restart of the silence function, we’re going to leave 1.5 sec of what we consider “silence” before we trim again.) 
      • (Threshold is still 0.2%) 
      • (by the way, we want to add a second of silence at the beginning just to make it less jarring when played back. do this with the “pad” command and 1.0 is for 1 second.)
  • On the above, things highlighted with light blue are probably going to need a LOT of tuning. Things like your noise floor, audio quality, static, squelch, interference, sound card, cable quality– all that stuff will affect what you set for these values. The ones in light orange are related to timing of the cutting and what to leave as silence padding. The rest are file-operation specific.
  • Anyway, now with this scripted, I have darkice feeding the broadcastify servers AND making a local mp3 dump. Each night this file is split using crontab, and a new one started. Another set of scripts takes over and “compacts” the file down from a 24-hour mp3 which seem to be over 300MB each to varying-length mp3 files which, so far, depending on how much the systems are used, seem to range from about 9MB to 120MB.
  • Not bad, and saving disk. average is about 25meg per file. Using the 25meg per day average, we’re at about 9.125Gig per year of audio. Is it small enough to fit on a dual-layer DVD? No, not quite. But it’s close. I think I can just schedule offloading to a normal DVD each quarter. Every 3 months. So I didn’t meet my original goal of a DVD of audio per year, but this is still much more manageable.
I need to cite some very handy references for completing the above, as the sox man page alone was great, but just didn’t quite get me there if you know what I mean:
  1. https://gist.github.com/devoncrouse/5534261
  2. http://digitalcardboard.com/blog/2009/08/25/the-sox-of-silence/
  3. http://activearchives.org/wiki/Padding_an_audio_file_with_silence_using_sox
  4. https://www.learnosity.com/blog/2009/11/removing-silence-from-audio-using-sox/
  5. http://comments.gmane.org/gmane.comp.audio.sox/2739
So, the above were extremely helpful in getting me there. I just didn’t find a good, concise, logical breakdown of what was doing what along my journey, and a google search for “using sox to trim silence out of the middle of a long mp3” just didn’t make the mark either. So there you go, for what it’s worth. 🙂
73, Joe N8CN
-jre