Embed Size (px)
Citation preview
REALCRAFT
Aggregate Sensor
Collection and Distribution
Erin Leigh Lukens Lead Developer
A.S. Computer Science
Nathan Sorey Geospatial Technologies [email protected]
CONTENTS
Acknowledgements .............................................................................................................................. 3
Introduction ........................................................................................................................................... 4
The Purpose of This Document .................................................................................................... 5
An Overview .................................................................................................................................... 6
Applications for Data Collection ................................................................................................... 7
Realcraft in Education ........................................................................................................................ 13
The Maker Movement ................................................................................................................. 13
Data Analytics .............................................................................................................................. 14
Geospatial Analysis ..................................................................................................................... 15
Application Development ............................................................................................................ 16
Style Guide .......................................................................................................................................... 17
Typography and Color ................................................................................................................. 18
Business Card .............................................................................................................................. 19
Letterhead .................................................................................................................................... 20
The Logo ....................................................................................................................................... 20
Website Design ............................................................................................................................ 21
Specifications ..................................................................................................................................... 22
Main Server .................................................................................................................................. 22
Development Server .................................................................................................................... 22
Software Specifications .............................................................................................................. 23
API Specifications............................................................................................................................... 24
JSON Specification ...................................................................................................................... 24
RESTful API .................................................................................................................................. 25
References .......................................................................................................................................... 26
REALCRAFT
FRIDAY, MAY 13, 2016 3
ACKNOWLEDGEMENTS
I would like to dedicate this project to my husband and all the times he interrupted my
train of thought. Without him, I would have a seriously boring life.
Nathan Sorey for inspiring this project with his insight input into the missing pieces in
the Geo-Spatial sciences and pushing my previous weather station project to the next
level.
MPAX and NorthWest Arkansas Community College for providing the opportunity and
classes available to develop a project in an environment that encourages community
integration and the promotion of education
REALCRAFT
FRIDAY, MAY 13, 2016 4
INTRODUCTION
Realcraft is an aggregate server for collecting and distributing sensor data for research
and education. Clients and organizations will be able to retrieve, contribute, analyze, and
share numerous types of sensor data in real-time across the world.
Realcraft uses open source software such as MySQL, MQTT, Python, and Flask to make
usage and operational overhead low. Utilizing simple protocols and interfaces, the
ability to contribute and analyze data can be brought to a wide range of devices and
platforms. The provided API allows for simple formatted JSON and RESTful calls to the
server that even the most basic of devices can perform.
REALCRAFT
FRIDAY, MAY 13, 2016 5
THE PURPOSE OF THIS DOCUMENT
This document is intended for research and information regarding the Realcraft project
and the work done by Erin Leigh Lukens and Nathan Sorey respectively. All of the
information contained herein is for educational and non-profit usage only. Realcraft is
listed under MIT licensing.
Copyright (c) 2016 Realcraft.io
PERMISSION IS HEREBY GRANTED, FREE OF CHARGE, TO ANY PERSON OBTAINING A COPY
OF THIS SOFTWARE AND ASSOCIATED DOCUMENTATION FILES (THE "SOFTWARE"), TO
DEAL IN THE SOFTWARE WITHOUT RESTRICTION, INCLUDING WITHOUT LIMITATION THE
RIGHTS TO USE, COPY, MODIFY, MERGE, PUBLISH, DISTRIBUTE, SUBLICENSE, AND/OR SELL
COPIES OF THE SOFTWARE, AND TO PERMIT PERSONS TO WHOM THE SOFTWARE IS
FURNISHED TO DO SO, SUBJECT TO THE FOLLOWING CONDITIONS:
THE ABOVE COPYRIGHT NOTICE AND THIS PERMISSION NOTICE SHALL BE INCLUDED IN
ALL COPIES OR SUBSTANTIAL PORTIONS OF THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
REALCRAFT
FRIDAY, MAY 13, 2016 6
AN OVERVIEW
A quick overview of how data traverses to and from the Realcraft servers is shown
below. Two of the simplest interfaces are shown (MQTT and Restful), which allow for
quick and simple recording of data.
REALCRAFT
FRIDAY, MAY 13, 2016 7
APPLICATIONS FOR DATA COLLECTION
Realcraft utilizes many current pieces of technology to allow clients to collect data from
a nearly limitless variety of sensing devices for use in geological, Meteorological,
geospatial, and other physical sciences.
Agricultural monitoring of crops for watering and pest management is becoming more
and more crucial as our populations continue to increase, having accurate and
comprehensive data is imperative.
Traffic and crowd data collected can improve future construction decisions and provide
the information that is important to making utilities and public works optimal for the
public.
Power management and conservation is a tremendous topic and having real-time
analytics can reduce air pollution while drastically increasing production from
renewable non-renewable sources.
REALCRAFT
FRIDAY, MAY 13, 2016 8
Water Quality Monitoring
Image: USGS / Amphibian Research and Monitoring Initiative (ARMI)
http://ga.water.usgs.gov/projects/armi/
Since 2000, water-quality data have been collected
at selected amphibian sampling locations to
support understanding of the occurrence and long-term
changes in amphibian populations. Data were collected in
vernal pools, streams, springs, and subterranean pools
and streams. Field parameters included field
measurements of water temperature, dissolved oxygen,
pH, specific conductance, and turbidity and laboratory
analyses of major ions,
nutrients, and metals.
REALCRAFT
FRIDAY, MAY 13, 2016 9
Air Quality Monitoring
Image: Turn-Key Ambient Air Quality Monitoring System
http://cas-en.com/
The rumble of the train passing underfoot is a
familiar sound to any New Yorker. But few realize
that each time the train whizzes by, a plume of polluted
air comes up through the street vents, causing a spike
in fine particulate matter, or PM 2.5, a contaminant
associated with asthma, heart
attacks and other health problems.
REALCRAFT
FRIDAY, MAY 13, 2016 10
Land Survey Monitoring
Image: Steckbeck engineering & Surveying, Inc http://www.steckbeck.net/
Geographic Information Systems (GIS) Specialists
GIS specialists use ongoing developments in digital technology to enhance the ways that information can be used. Integrating these databases in real time with GPS positions or other devices opens up a new world of location-based services – one of the many uses of GIS.
REALCRAFT
FRIDAY, MAY 13, 2016 11
Urban Environmental Monitoring
Image: Urban Sensing – Sensors in Urban Environments
http://www.citizensense.net/projects/urban-sensing/urban-sensing/
Urban sensing projects are by now a well-established
feature of urban design. From smart cities to
augmented urban ecologies, numerous projects are now
located at the intersection of wireless sensor landscapes
with urban processes. This project area focuses on a
particular aspect of urban sensing projects, namely those
actual and speculative proposals that suggest wireless
sensing technologies are a way to achieve
more sustainable and efficient cities.
REALCRAFT
FRIDAY, MAY 13, 2016 12
Traffic and Crowd Metrics
Image: DOT Radar Detector
http://ops.fhwa.dot.gov/publications/fhwahop06006/
The traffic signal impacts virtually everyone
every day. Even on uncongested routes, stops at
traffic signals punctuate an urban or suburban area trip.
School children obediently wait for a traffic signal to
interrupt traffic so they can cross a busy thoroughfare.
Drivers confidently place their own and their
passengers' physical safety in a
signal's allocation of right-of-way.
REALCRAFT
FRIDAY, MAY 13, 2016 13
REALCRAFT IN EDUCATION
Realcraft provides many opportunities for students, teachers, and all manners of
educators a space to collaborate, design, and analyze data from already existing tools
and from project kits utilizing hardware like Arduino and Raspberry Pi.
THE MAKER MOVEMENT
Access to the Realcraft API simplifies the tedious tasks of programming complex
applications into a small package where students can get more from the total
experience than spending long amounts of time debugging.
Above: NodeMCU vE-12
Right: Arduino Nano, ESP8266
Both microcontrollers have an analog temperature and light sensor that send
information back to the Realcraft server every 15 seconds.
REALCRAFT
FRIDAY, MAY 13, 2016 14
DATA ANALYTICS
Collect and aggregate missing data from other sources using the Realcraft Toolkit to
use in applications to build striking, informative, and useful charts.
Image: Chartist.js for R, powered by htmlwidgets
https://recordnotfound.com/chartist-yutannihilation-38428
REALCRAFT
FRIDAY, MAY 13, 2016 15
GEOSPATIAL ANALYSIS
Realcraft incorporates the ability to interpolate data over geological maps provided by a
tile server. The tile server is based on OpenStreetMaps and OpenLayers technology that
pulls in the efforts of the community to build and maintain beautifully crafted maps for
use in education and non-commercial use.
Image: OpenStreetMaps (Beaver Lake, Arkansas)
http://www.openstreetmap.org/
REALCRAFT
FRIDAY, MAY 13, 2016 16
APPLICATION DEVELOPMENT
Realcraft is open-source and available to download and run on your own. While this is
enticing to run your own server, be advised that running high-volume services require a
lot of resources such as bandwidth and processing power. MySQL and OpenStreetMaps
require a lot of server overhead to operate and these requirements should be
considered before running your own.
Since Realcraft is open-source, we encourage everyone to download the code, improve
upon it, and share it with the rest of the world but, by all means, you are more than
welcome to create your own server.
Realcraft is hosted on GitLab and its current state is only open to a select few
developers until a stable and branded version is ready. If you are interested in working
with the initial code before being released to the public, please contact Erin Leigh
Lukens at [email protected] for more information.
GITLAB INC.
GitLab Inc. is a company based on the GitLab open-source project.
GitLab is an application to code, test, and deploy code together. It
provides Git repository management with fine grained access
controls, code reviews, issue tracking, activity feeds, wikis, and continuous integration.
REALCRAFT
FRIDAY, MAY 13, 2016 17
STYLE GUIDE
Style is important. Even in education! The following guides help to demonstrate easy
usage of Realcraft assets and designs.
I always say I have a Socratic approach to most
things that I do. I pummel people with questions,
because I need to know what they're thinking, what they're
trying to achieve, what they believe
the final outcome is going to be.
- Tim Gunn
REALCRAFT
FRIDAY, MAY 13, 2016 18
TYPOGRAPHY AND COLOR
Roboto
Christian Robertson
Apache License, version 2.0
Roboto has a dual nature. It has a mechanical skeleton and the forms are largely
geometric. At the same time, the font features friendly and open curves. While some
grotesks distort their letterforms to force a rigid rhythm, Roboto doesn’t compromise,
allowing letters to be settled into their natural width. This makes for a more natural
reading rhythm more commonly found in humanist and serif types.
Build-It (Realcraft Color Scheme) P
AN
TO
NE
6
55
C
PA
NT
ON
E
66
0 C
PA
NT
ON
E
54
67
C
PA
NT
ON
E
63
2 C
PA
NT
ON
E
75
41
C
PA
NT
ON
E
Bla
ck
6 C
REALCRAFT
FRIDAY, MAY 13, 2016 19
BUSINESS CARD
The basic business card handout is a vertical layout utilizing the typography and color
configuration standard across the platform to maximize recognition and remain
continuity.
Each member of the team has their own personal information placed ½ down the media
including:
Name
Position
Up to two (2) phone numbers
[email protected] email address
Mailing address (optional). City and state are required.
REALCRAFT
FRIDAY, MAY 13, 2016 20
LETTERHEAD
This is a standard Realcraft letterhead using the same guidelines as the business card.
It is important to present the same look-and-feel across all your media to generate
familiarity across the variety of products.
THE LOGO
The Realcraft Brand logo was created by P. Sanchez Rowe
using Adobe Illustrator. The logo resembles the foundation of
Realcraft and the ideas that constructed it. Idealized as a
server to store weather data to simulate inside of Minecraft,
the concept quickly became a lot larger. The blocks represent
all those ideas coming together to build much, much more.
REALCRAFT
FRIDAY, MAY 13, 2016 21
WEBSITE DESIGN
The foundational styling is brought in from Twitter Bootstrap, which provides the
interactive elements of the website with minimal changes to the standard style set.
The Realcraft logo is set to overflow the Brand page element and the default font set
has been changed to Roboto.
All modern browsers, within reason, will be able to view the website in most resolutions
and provide responsive designing for small displays such as phones, tablets, and
embedded devices.
Styling, formatting, and responsiveness is tested for Mozilla Firefox, Google Chrome,
and Chromium browser in both desktop and mobile platforms on Android and iOS.
REALCRAFT
FRIDAY, MAY 13, 2016 22
SPECIFICATIONS
MAIN SERVER
CPU 1 vCore
RAM 1 GB Guaranteed
HDD 50 GB SSD
Traffic Unlimited
Operating System Linux
Management Software Parallels® Plesk 12 unlimited with Application Pool
Bandwidth 100 Mbit/s
DEVELOPMENT SERVER
CPU Pentium Core 2 Duo
RAM 8 GB SDRAM
HDD 550 GB 7200 RPM
Operating System Ubuntu Server 14.04 LTS
Management Software Webmin
REALCRAFT
FRIDAY, MAY 13, 2016 23
SOFTWARE SPECIFICATIONS
Development
Python
Flask
GIT
PyCharm
nodeMCU Arduino
Twitter Bootstrap
Javascript
JSON
RESTful
TwistedMatrix
SQLAlchemy
D3js
jQuery
Jinja2
Android Studio
Server
Apache
RabbitMQ
MySQL
PostgreSQL
Webmin
GITLab
Postfix
Dovecot
Hardware
nodeMCU E-12
ESP8266 v1
REALCRAFT
FRIDAY, MAY 13, 2016 24
API SPECIFICATIONS
JSON SPECIFICATION
Realcraft accepts JSON formatted strings from devices to the dedicated RabbitMQ
messaging broker.
Making a post request:
The JSON string can be submitted by either MQTT or via RESTful web server posts.
Payload Format
{
“type”: “sensor”,
“timestamp”: 148526998,
“client_id”: “Ag5rg9sdlfo349dla0eaz”,
“api_key”: “343jc83kd9ldkd9ckqwasd”,
“data”: {
<JSON containing sensor data>
}
}
REALCRAFT
FRIDAY, MAY 13, 2016 25
RESTFUL API
You can form a full RESTful URL with the PUT method parameters set to:
https://realcraft.io/api/v0.1/sensor/<client_id>/<sensor_id>/<routing>/
While Realcraft does not enable SSL (https) by default, it is recommended that all
servers and clients enable this function in a production environment. You can do so via
your web server configuration, which Realcraft will then detect and use.
Payload Format
{
“timestamp”: 145688474,
“data”: <JSON String Containing Data>
}
REALCRAFT
FRIDAY, MAY 13, 2016 26
REFERENCES
"ABOUT CAS - CAS - CONSOLIDATED ANALYTICAL SYSTEMS." CAS CONSOLIDATED ANALYTICAL SYSTEMS. N.P.,
N.D. WEB. 12 MAY 2016. <HTTP://CAS-EN.COM/ABOUT-CAS/>.
"AMPHIBIAN RESEARCH AND MONITORING INITIATIVE (ARMI) - SOUTHEAST REGION WATER QUALITY." :
USGS SOUTH ATLANTIC WATER SCIENCE CENTER. N.P., N.D. WEB. 12 MAY 2016.
<HTTP://GA.WATER.USGS.GOV/PROJECTS/ARMI/>.
"EPA SCIENTIST: TOOLS THAT LET EVERYONE MEASURE AIR POLLUTION ARE A “GAME-CHANGER”." – NEXT
CITY. N.P., N.D. WEB. 12 MAY 2016. <HTTPS://NEXTCITY.ORG/DAILY/ENTRY/TOOLS-MEASURE-AIR-
POLLUTION-AIRBEAM-AIRCASTER-EPA>.
"TRAFFIC CONTROL SYSTEMS HANDBOOK: 1. INTRODUCTION." TRAFFIC CONTROL SYSTEMS HANDBOOK:
CHAPTER 1 INTRODUCTION. N.P., N.D. WEB. 12 MAY 2016.
<HTTP://OPS.FHWA.DOT.GOV/PUBLICATIONS/FHWAHOP06006/CHAPTER_1.HTM#1-3>.
"WHAT SURVEYORS AND GEOSPATIAL ENGINEERS DO." SCHOOL OF CIVIL AND ENVIRONMENTAL
ENGINEERING. N.P., 17 OCT. 2013. WEB. 12 MAY 2016.
<HTTPS://WWW.ENGINEERING.UNSW.EDU.AU/CIVIL-ENGINEERING/WHAT-IS-ENGINEERING/CIVIL-
AND-ENVIRONMENTAL-ENGINEERING-CAREERS/WHAT-SURVEYORS-AND-GEOSPATIAL#GEO>.
"CHARTIST.JS FOR R, POWERED BY HTMLWIDGETS" BY YUTANNIHILATION. N.P., N.D. WEB. 12 MAY 2016.
<HTTPS://RECORDNOTFOUND.COM/CHARTIST-YUTANNIHILATION-38428>.
"TAKE BACK CONTROL OF YOUR MAPS." SWITCH2OSM. N.P., N.D. WEB. 12 MAY 2016.
<HTTPS://SWITCH2OSM.ORG/>.
"A HIGH-PERFORMANCE, FEATURE-PACKED LIBRARY FOR ALL YOUR MAPPING NEEDS." OPENLAYERS 3. N.P.,
N.D. WEB. 12 MAY 2016. <HTTP://OPENLAYERS.ORG/>.
"THE MIT LICENSE (MIT)." NEWS. N.P., N.D. WEB. 12 MAY 2016.
<HTTPS://OPENSOURCE.ORG/LICENSES/MIT>.
REALCRAFT
FRIDAY, MAY 13, 2016 27
"BOOTSTRAP · THE WORLD'S MOST POPULAR MOBILE-FIRST AND RESPONSIVE FRONT-END FRAMEWORK."
BOOTSTRAP · THE WORLD'S MOST POPULAR MOBILE-FIRST AND RESPONSIVE FRONT-END
FRAMEWORK. N.P., N.D. WEB. 12 MAY 2016. <HTTP://GETBOOTSTRAP.COM/>.
"FLASK - WEB DEVELOPMENT, ONE DROP AT A TIME." WELCOME. N.P., N.D. WEB. 12 MAY 2016.
<HTTP://FLASK.POCOO.ORG/>.
"WOOZELWORKZ - ILLUSTRATION AND DESIGN." HOME. N.P., N.D. WEB. 12 MAY 2016.
<HTTP://WOOZELWORKZ.COM/>.
"NAKED GUNN - METRO WEEKLY." METRO WEEKLY. N.P., 26 SEPT. 2007. WEB. 12 MAY 2016.
<HTTP://WWW.METROWEEKLY.COM/2007/09/NAKED-GUNN/>.
"WELCOME TO PYTHON.ORG." PYTHON.ORG. N.P., N.D. WEB. 12 MAY 2016.
<HTTPS://WWW.PYTHON.ORG/ABOUT/>.
"GITLAB INC." GITLAB. N.P., N.D. WEB. 13 MAY 2016. <HTTPS://ABOUT.GITLAB.COM/ABOUT/>.
