Skip to content

Mac Installation / Live Audio Example #125

@seandenigris

Description

@seandenigris

Hi,

I recently got PS with GStreamer working on Mac High Sierra. It was a bit convoluted and I'd like to share my results in case it helps other users. Let me know if you think there's an appropriate place to include the following info…

Install PocketSphinx

Attempts

  1. FAILED: brew install cmu-pocketsphinx Very old version (0.8) which doesn't match docs
  2. Found tap to install HEAD on GH

GStreamer Integration

The PS docs say that leveraging GStream makes continuous mode easier, however, we couldn't get it working.

  1. Installed via brew install gstreamer gst-plugins-base gst-plugins-good (last one was for testing)
  2. Build gst-pocketsphinx plugin
  • FAILED: Reinstall PS to build the plugin; couldn't find gstreamer; posted question on both the homebrew and PS MLs
  • Cloned and built PS from scratch following the last script from this SO thread
    • Manually copied the plugin (*.so file) to /usr/local/lib/gstreamer-1.0; NB: aliasing did not work, maybe a symlink would have
  1. Test
  • Recognize sound file: gst-launch-1.0 filesrc location=~/Documents/Reference/Foreign\ Language/Speak\ From\ Day\ One/audio_book/00_about.mp3 ! mpegaudioparse ! avdec_mp3 ! audioconvert ! audioresample ! pocketsphinx ! fdsink fd=1
  • Recognize live sound
    • Tried gst-launch-1.0 autoaudiosrc ! audioconvert ! audioresample ! pocketsphinx name=asr ! fakesink but thought it wasn't working because no output; realized that fakesink means throw away output!
    • Tried gst-launch-1.0 autoaudiosrc ! audioconvert ! audioresample ! pocketsphinx name=asr ! fdsink fd=1, which returned (poorly) recognized text, but warned that autoaudiosrc0-actual-src-osxaudi: Can't record audio fast enough.
    • Added queue per the gst ML to get: gst-launch-1.0 autoaudiosrc ! queue ! audioconvert ! audioresample ! pocketsphinx name=asr ! fdsink fd=1. This thread also seemed to have some interesting ideas about how to address this.
  1. Try python bindings
  • brew install
  • Workaround: export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python3.6/site-packages (per this GH issue)
  • Link Homebrew python module target to python3: echo "/usr/local/lib/python3.6/site-packages" > /usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/Homebrew.pth per this SO thread

Usage Example - Live Audio

#!/usr/bin/env python

# Adapted from:
#   - https://github.com/GStreamer/gst-python/blob/master/examples/helloworld.py
#   - With parts from https://cmusphinx.github.io/wiki/gstreamer/

import sys
import signal

import gi
gi.require_version('Gst', '1.0')
from gi.repository import GObject, Gst

#For IPC
import socket

def on_element_message(bus, message, loop):
    """Receive element messages from the bus."""
    msgtype = message.get_structure().get_name()
    if msgtype != 'pocketsphinx':
        return

    if message.get_structure().get_value('final'):
        final_result(message.get_structure().get_value('hypothesis'),
                     message.get_structure().get_value('confidence'))

def final_result(hyp, confidence):
    #Debug - reference https://docs.python.org/3/tutorial/inputoutput.html#fancier-output-formatting
    #print('Final: {}; confidence: {}'.format(hyp, confidence))
    send_command(hyp)

def send_command(command):
    # Create a TCP/IP socket
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Connect the socket to the port where the server is listening
    server_address = ('localhost', 9999)
    #print >>sys.stderr, 'connecting to %s port %s' % server_address
    sock.connect(server_address)

    try:
        
        # Send data
        #print >>sys.stderr, 'sending "%s"' % command
        sock.sendall(command.encode()) #.encode per https://stackoverflow.com/a/37376668/424245
        
        # Look for the response
        amount_received = 0
        amount_expected = len(command)
        
        while amount_received < amount_expected:
            data = sock.recv(16)
            amount_received += len(data)
            #print >>sys.stderr, 'received "%s"' % data

    finally:
        #print >>sys.stderr, 'closing socket'
        sock.close()

def main(args):
    # Before using Python threads, or libraries using threads (GStreamer for example), you have to call GObject.threads_init(). Contrary to the naming, this function isn't provided by gobject but initializes thread support in PyGObject (it was called gobject.threads_init() in pygtk). Think of it as gi.threads_init().
    # Since PyGObject 3.10.2, calling GObject.threads_init() this is no longer needed.
    # Per https://wiki.gnome.org/Projects/PyGObject/Threading
    GObject.threads_init()
    Gst.init(None) # Cleanup via gst-deinit is normally not needed in an application as the resources will automatically be freed when the program terminates https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/gstreamer-Gst.html#gst-deinit

    # Doc: https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/gstreamer-GstParse.html#gst-parse-launch
    pipeline = Gst.parse_launch('autoaudiosrc ! queue ! audioconvert !' +
                            ' audioresample ! pocketsphinx name=asr !' +
                            ' fakesink') # to debug, use "fdsink fd=1" for last arg to write to stdout
    if not pipeline:
        sys.stderr.write("Failed to create pipeline\n")
        sys.exit(1)
    
    pocketsphinx = pipeline.get_by_name('asr')
    #From https://raspberrypi.stackexchange.com/a/56818
    pocketsphinx.set_property('kws', 'path/to/command.list')

    # create and event loop and feed gstreamer bus mesages to it
    loop = GObject.MainLoop.new(None, False)
    
    bus = pipeline.get_bus()
    bus.add_signal_watch()
    bus.connect ('message::element', on_element_message, loop)
    
    # start play back and listed to events
    pipeline.set_state(Gst.State.PLAYING)
    try:
        loop.run()
    except:
        pass

    # cleanup
    print('Cleaning up...')
    pipeline.set_state(Gst.State.NULL)

    # Block until pipeline changes state (per http://gstreamer-devel.966125.n4.nabble.com/Gstreamer-critical-on-pipeline-set-state-Gst-State-NULL-tp4678899p4678912.html), but the docs say changes to null are never async and suggest an alternate check (https://lazka.github.io/pgi-docs/Gst-1.0/classes/Element.html#Gst.Element.set_state)
    pipeline.get_state(Gst.CLOCK_TIME_NONE)
    loop.quit()
    print('Done!')

    return 0

if __name__ == '__main__':
    sys.exit(main(sys.argv))

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions