Skip to content

Instantly share code, notes, and snippets.

@jamgraham

jamgraham/gist:07ca2f7f2fad144e899b

Last active August 29, 2015 14:19
Show Gist options
  • Save jamgraham/07ca2f7f2fad144e899b to your computer and use it in GitHub Desktop.
Save jamgraham/07ca2f7f2fad144e899b to your computer and use it in GitHub Desktop.
Download ZIP
IoT Streaming Conference
Raw
gistfile1.md

IoT Streaming Conference

Notes

IoT Architecture

Capture

  • MQ, Kafka, etc..
  • Recommendation is kafka
  • This data gets archived into Hadoop at some point

Process

  • Likes spark streaming
  • Check out NiFi.. something the NSA has in production. They're the leaders in big data.. let's be honest about that!

Query

  • Reattime nosql
  • batch queries nosql

Store

  • Hadoop for 'forever' storage
  • NOSQL for 'real-time' storage
  • This is the source of truth

Lambda architecture

  • Came out of twitter
  • Main Principles
    • Batch layer: The batch layer precomputes results using a distributed processing system that can handle very large quantities of data. The batch layer aims at perfect accuracy by being able to process all available data when generating views. This means it can fix any errors by recomputing based on the complete data set, then updating existing views. Output is typically stored in a read-only database, with updates completely replacing existing precomputed views.
    • Speed Layer: The speed layer processes data streams in real time and without the requirements of fix-ups or completeness. This layer sacrifices throughput as it aims to minimize latency by providing real-time views into the most recent data. Essentially, the speed layer is responsible for filling the "gap" caused by the batch layer's lag in providing views based on the most recent data. This layer's views may not be as accurate or complete as the ones eventually produced by the batch layer, but they are available almost immediately after data is received, and can be replaced when the batch layer's views for the same data become available.
    • Server Layer: Output from the batch and speed layers are stored in the serving layer, which responds to ad-hoc queries by returning precomputed views or building views from the processed data.

Micro Services

  • Chad Arimura iron.io

  • Loosely coupled workers. Like Storm Topologies

  • A service-oriented architecture composed of loosely couples elements that have bounded contexts

    • bounded contexts - model where you don't need information outside of that scope or process

  • Goal:

    • zero down time deployments
    • Specific teams that own their services

  • Small loosely coupled teams

    • Never have a team that you can't feed with two pizzas at amazon Bezos

  • Stateless by design - It's about process and organization just as much as development.

    • Adrian Kakrof - "You want to treat your servers like cattle not pets" Don't feel bad if they go down. Rotate the good servers in and kill the bad ones. Don't get sad about it like your pets

  • Containerize everything - The evolution of compute has gone from a server to a virtual machine, to amazon's ami, to docker (another container), now to the event or task. It's about what's being processed by the container.

  • Polygod Persistence - Share nothing and de-normalize. Relational databases are a thing of the past. The concept of wanting to not duplicate data is aging. Create copies of that data and don't be scared to do that. Move and duplicate data synchronously between services

    • Example CNN - 50 MM alerts per second with their Messaging Service

Streaming Big Data - Analytics & Triggers

Garth Adhav - Director of Analytics, Cisco

  • Cloud -> Fog -> Mist
  • Fog -Mid layer that talks to censors it's close to
    • Concept of analyzing data as it comes in on the connected device level. That way there's less to analyze when the data meets up with Hadoop in the cloud?
  • Maslow's Hierarchy of IoT
  • The new Paradigm

Fog

Note that Cisco has started using the phrase 'Fog'.. But 'Edge' seems to make a bit more sense and may be what this layer ends up being called. A UNC professor actually coined the term 'Fog' in this context.

The term Cloud is now further localized into Fog and even Mist as the decision making intelligence moves closer to IoT hardware than to reside on a single central plane.

Security

Guy AiLee - Intel

  • "Security is foundational" as always...
  • McAfee, Wind River, Intel are leaders in the space
  • If you have identity, secure boot, and provision the device you know what's going in to it.
  • Nest is a great example of securing an iOt device.. so are pace makers.
  • Three keys to success:
    • Harden the device
    • Secure the communication
    • Manage and monitor
  • Three things you must do
    • Hardware ID - Something that's unchangeable (can't be hacked) This is your hardware root of trust
    • Secure Boot - Kernal-level device image to enable secure device provision at OS/BIOS level
    • White Listing - Secure image of allowable agents/applications for that specific device.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment