Platform for Situated Intelligence is an open, extensible framework that enables the enables the development, fielding and study of situated, integrative-AI systems.
In recent years, we have seen significant progress with machine learning techniques on various perceptual and control problems. At the same time, building end-to-end, multimodal, integrative-AI systems that leverage multiple technologies and act autonomously or interact with people in the open world remains a challenging, error-prone and time-consuming engineering task. Numerous challenges stem from the sheer complexity of these systems and are amplified by the lack of appropriate infrastructure and development tools.
The Platform for Situated Intelligence project aims to address these issues and provide a basis for developing, fielding and studying integrative-AI systems. The platform consists of three layers. The Runtime layer provides a parallel programming model centered around temporal streams of data, and enables easy development of components and applications using .NET, while retaining the performance properties of natively written, carefully tuned systems. A set of Tools enable multimodal data visualization, annotations, analytics, tuning and machine learning scenarios. Finally, an open ecosystem of Components encapsulate various AI technologies and allow for quick compositing of integrative-AI applications. For more information about the goals of the project, the types of systems that you can build using it, and the various layers see Platform for Situated Intelligence Overview.
Platform for Situated Intelligence is built on the .NET Framework. Large parts of it are built on .NET Standard and therefore run both on Windows and Linux, whereas some components are specific and available only to one operating system (for instance the Kinect sensor component is available only for Windows.)
You can build applications based on Platform for Situated Intelligence either by leveraging nuget packages, or by cloning and building the code. Below are instructions:
Brief Introduction. To get started with using Platform for Situated Intelligence, the Brief Introduction page provides a guided walk-through for some of the main concepts in \psi. It shows how to create a simple program, describes the core concept of a stream, and explains how to transform, synchronize, visualize, persist to and replay data from disk. We recommend that you work through the examples in this tutorial to familiarize yourself with these core concepts.
Samples. After going through this first brief tutorial, it may be helpful to look through the set of Samples provided. While some of the samples address specialized topics such as how to leverage speech recognition components or how to bridge to ROS, reading them will give you more insight into programming with \psi.
In-depth Topics. Finally, additional information is provided in a set of In-Depth Topics that dive into ore detail on various aspects of the framework including synchronization, persistence, remoting, visualization etc.
Like the rest of the codebase, the documentation is still under construction and in various phases of completion. If you need further explanation in any of these areas, please open an issue, label it documentation, as this will help us target our documentation development efforts to the highest priority needs.
The codebase is currently in beta and various aspects of the platform are at different levels of completion and robustness. There are probably still bugs in the code and we will likely be making breaking API changes. We plan to continuously improve the framework and we encourage the community to contribute.
For additional information, we recommend you read the Known Issues section from the Release Notes document, which provides more information about important issues that are known and which we plan to address in the near future. Also, the Roadmap document provides more information about our future plans.
If you find a reproducible bug or if you would like to request a new feature or additional documentation, please file an issue on the github repo. If you do so, please make sure a corresponding issue has not already been filed. Use the bug label when filing issues that represent code defects, and provide enough information to reproduce. Use the feature request label to request new features, and use the documentation label to request additional documentation.
We hope the community can help improve and evolve Platform for Situated Intelligence. If you plan to contribute to the codebase, please read the Contributing Guidelines document. It describes how the source code is organized and things you need to know before making any source code changes.
Platform for Situated Intelligence is currently being used in a number of industry and academic research labs, including (but not limited to):
- in the Situated Interaction project, as well as other research projects at Microsoft Research.
- in the MultiComp Lab at Carnegie Mellon University.
- in the Speech Language and Interactive Machines research group at Boise State University.
- in the Qualitative Reasoning Group, Northwestern University.
If you would like to be added to this list, just add a GitHub issue and label it with the whoisusing label. Add a url for your research lab, website or project that you would like us to link to.
Platform for Situated Intelligence is available under an MIT License. See also Third Party Notices.
We would like to thank our internal and external early adopters for the feedback provided during the alpha testing period, including (but not limited to): Daniel McDuff and Kael Rowan at MSR, Chirag Raman and Louis-Phillipe Morency in the MultiComp Lab at CMU, as well as adopters in the SLIM research group at Boise State and in the Qualitative Reasoning Group at Northwestern University.