Skip to content
This repository has been archived by the owner on Oct 26, 2022. It is now read-only.

LambdaSharp/Kinesis-Paraglider-Challenge

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

24 Commits
 
 
 
 
 
 
 
 

Repository files navigation

Kinesis Paraglider Challenge Team Hackathon Challenge

Use AWS's Kinesis service to find the optimal time to go Paragliding! We will be using weather data provided by NOAA with a provided event streamer.

NOAAParagliding

Pre-requisites

The following tools and accounts are required to complete these instructions.

Level 0

  • Create a Kinesis stream in the AWS console (make sure the Region is set to US-East-1)
    • Search for Kinesis in the services dropdown
    • Click Create data steam under the Kinesis data streams section on the Amazon Kinesis Dashboard
    • Give it a name and 1 shard
  • Clone this repo to get the data streamer application on your machine
  • Update kinesis key name in src/WeatherStationsEvents/appsettings.json to match the name you just gave to your stream
Build and run 'src/WeatherStationsEvents'
  • Go to terminal
  • CD into the repo you just cloned
  • CD into 'src/WeatherStationEvents'
  • Run 'dotnet restore' and 'dotnet run'
  • Verify from logs in the terminal that events are being generated
  • NOTE: This data is historical. It starts from an arbitrary date in August 2018 and continues up to today.

Level 1

Goal - Create a lambda function to capture the streaming data from the Kinesis stream you just set up.

  • Here is a ready to go NodeJs 6.10 lambda function for reading a Kinesis Stream. Make the lambda function output the data to CloudWatch Logs
exports.handler = (event, context, callback) => {
    
    for (let i = 0; i < event.Records.length; i++) {
      const eventRecord = JSON.parse(Buffer.from(event.Records[i].kinesis.data, 'base64'));
      console.log(eventRecord); 
    }
    
    callback(null, "Hello from Lambda");
};
Make sure to set up this function to trigger off of the Kinesis Stream
  • Navigate to the AWS console for your lambda function
  • Make sure the configuration tab is selected at the top of the page
  • From the list of triggers on the left panel in the Designer, choose Kinesis
  • Scroll down to the 'Configure triggers' section
  • Select the Kinesis Stream you previously created from the dropdown
  • Make sure the 'Enable trigger' box is checked, then hit 'Add'
Check CloudWatch logs for event record output
  • Once the trigger is setup, run the streaming application from the terminal
  • A record should be pushed to the Kinesis stream every five seconds and processed by your lambda function
  • On the lambda function page, click the 'Monitoring' tab at the top and click the 'View logs in CloudWatch button on the right

Level 2 - Single Site Notification

Goal - Find the best time to go fly at Torrey Pines Gliderport. Analyze the streaming data and determine if the weather is good for paragliding.

Hints
  • Explore the event record object to find the attributes that need to be checked

Level 3 - Kinesis Data Analytics

Goal - Use Kinesis Data Analytics to add a lambda function for pre-processing records.

  • Replace NA with values of zero on field barometricPressure
  • Hook up the output of the pre processing to your original lambda function
Helpful docs
Hints
  • Be very careful with the IAM role for Data Analytics permissions
  • Make sure the data streaming application is running when using DA
  • Data is base64 encoded!

Level 4 - Multi Site Notification

Goal - Find the best time to go fly at more than one location.

  • Use this website to determine the best conditions at another location or locations
  • Update your lambda function to check the streaming data for flying conditions at multiple sites
  • Update the function to:
    • Only send one notification per site per day
    • Only notify during daylight hours - 9 AM to 6 PM
Hints
  • You will have to persist the data across lambda invocations in order to know if a notification has already been sent...

BOSS Level - Real Time Analytics with SQL

boss

Goal - Use Kinesis Data Analytics to create a new data stream to better fit the original lambda function application.
  • Within the Kinesis Data Analytics Application you created for step 3 use the SQL editor to perform real time analytics on the data
  • Create a new SQL query using the templated SQL examples. Use the source data as a guide
  • Attach the resulting stream of the real time analytics to your original lambda function

Releases

No releases published

Packages

No packages published

Languages