• Requirements
• MSLIB
• Speech Recognition Offline
• Unreal Engine
• Node.js Server

• Visual Studio (suggested 2017 version with Node.js Tools for Visual Studio installed).
• Node.js
• Sox
• Unreal Engine 4

• MFC and ATL Libraries, downloadable from VS installer, as dependencies for MSLIB
• Microsoft Speech Runtime & Languages
  • Microsoft Speech Platform 11 SDK
  • Microsoft Speech Platform 11 Runtime
  • Microsoft Speech Platform 11 Runtime Languages:
    • MSSpeech_TTS_en-US_ZiraPro, as output voice
    • MSSpeech_SR_en-US_TELE, for the Speech Recognition You can download more languages, but you have to update accordingly MSLIB source

After tools installation, check if sox and node work in command prompt; if not, you have to add their locations to PATH environment variable.

Sox

> sox --version
sox:      SoX v14.4.2

Node.js

> node -v
v6.9.1

MSLIB, which stands for Microsoft Speech Lib, is a simple static library which exposes the method TextToPCM. This method takes a String as an input and returns a byte array containing the TTS render of said String. MSLIB uses Microsoft Speech API (SAPI) 5.4.

NOTE: Make sure that the MSLIB project is pointing to your MicrosoftSpeech SDK install location (specifically, the Include and Lib folders), default for this project are D:\Program Files\Microsoft SDKs\Speech\v11.0\ (x64 build) and D:\Program Files %28x86%29\Microsoft SDKs\Speech\v11.0 (x86 build).

NOTE: If you change/update the library, recompiling, make sure to place the new version of the lib files and headers inside UnrealEngine_Project\RobbieTheRobot\Plugins\MicrosoftSpeechTTS\Source\MSLIB

This is a C# project to perform offline speech to text tasks with provided keyphrases.

It is used for frequently ask actions like activation phrases (e.g. Hey Robbie) or for phrases hard to recognize (e.g. Open the map confused with Open them up).

Open SpeechRecognition.sln file with Visual Studio, set Release and build it (Build -> Build Solution or CTRL+SHIFT+B).

You will find the executable file in build/Release folder.

SpeechRecognition <input>

Where <input> is the location of the file containing keyphrases; if not provided it will try to open keyphrases.txt in the same folder of executable file.

This is a Unreal Engine project named RobbieTheRobot.

It contains the third person preset and a TCP C++ class exposed to Blueprint: it allows to start two TCP servers on ports 127.0.0.1:9000 and 127.0.0.1:9001 and they are used respectively to receive speech-to-text transcription and the response from chat-bot.

ThirdPersonCharacter Blueprint is used to setup the TCP server and you can find documentation in it.

MicrosoftSpeechTTS is an Unreal Engine plugin which exposes a Blueprint Node called Spawn TTS. This node takes a Text as input and outputs a SoundWave, as simple as that. To play the output an AudioComponent is needed. Here's how the Spawn TTS node is used in this project:

You can launch the game with right click on RobbieTheRobot.uproject and click Launch game or double-click to open the project.

This is a Node.js project use to manage speech-to-text both offline and online (throught Watson SpeechToText service); to send the transcription to Watson Conversation service and to send the chat-bot response to Unreal Engine game.

This project needs the Unreal Engine game launched to run.

Run the following command to install the project dependencies.

npm install

Run the following command to execute the project.

node index.js

1. Compile the SpeechRecognition project, then copy SpeechRecognition.exe and keyphrases.txt inside the NodeJSServer folder
2. Make sure Unreal Engine is building the project correctly
3. Start the game (from the UE Editor or launch the game binary)
4. Start the nodejs server

This software use the offline speech-to-text generated from SpeechRecognition project; when it fails, the node server runs sox to get microphone stream from default audio input device, recognize the silence, convert it to .ogg format and send it to Watson.

You can change service credentials opening index.js (Watson Conversation) and src/SpeechRecognition.js (Watson SpeechToText).

keyphrases.txt contains all phrases that have to be recognized offline: if you notice that some phrases are missing you can add them into this file.

Daniel Bertocci (original author) Federico D'Ambrosio Enrico Ferro Edoardo Ferrante

• GitHub
• Telegram