Embed Size (px)
Citation preview
Introduction to Embedded Systems Research:Course Review
Robert Dick
[email protected] of Electrical Engineering and Computer Science
University of Michigan
Fovea
Lens
Cornea
Variable Resolution
and Position Sampling
Change-Adaptive
Signalling
Spatial State
Cache
(Occipital
Place Area)
Analasys, e.g.,
Classification
Adequate
decision confidence?
Sampling Guidance
Y
N
Long-Term Memory
Decision /
Result
(b) Iterative, multi-round human vision system.
Image Signal
Processor
Application Processor
(CPU and/or GPU)
CloudDecision /
Result
(a) Conventional machine vision pipeline.
Image Sensor: typically
homogeneous RGGB or RCCC.
Demosaicing, binning,
denoising, gamma
correction, and compression.
Hardware Feature
Extraction Accelerator
or
Feature extraction on
raw captured data.
Runs CNN, LSTM or
other analysis algorithm.
May drop computation on less
important data, but already payed
Image Signal Processor transfer cost.
May render decision or (at high
energy cost) do feature extraction
and defer decision to cloud.
Minimalistic
Image Pre-Processor
Application Processor
(CPU and/or GPU)
CloudDecision /
Result
Image Sensor: capture only the
most important
data for decision accuracy.
Efficient gamma
correction
and binning.
Decide based on features.
Very high energy cost for
wireless data transfer.
Scene Cache
Capture ControllerHardware Feature
Extraction Accelerator
or
Adequate
decision confidence?
or
Y
N
Captures most relevant
and rapidly changing data.
Learns important sample locations
from prior rounds.
Maintains state built from
prior still-relevant samples.
Determine and
transmit
relevant data.
Decide based on features.
Very high energy cost for
wireless data transfer.
Issue commands to
capture important data.
(c) Goal: multi-round, energy-efficient, low-latency
continuous learning machine vision.
Feature extraction on sparse captured
data with similar distribution to processed data.
Continuously learn features and important
data based on prior captures.
1.1040
2.1041
385
2.1039
704
1.1039
36
2.1040
1734
0.1039
1
3.1040
4
4.1039
642
3.1039
4.1040
409
3.1041
396665
5.1042
108612774
5.1040
337
4.1041
10644
6.1044
117
6.1039
609
6.1045
164
6.1040
841434 723938
6.1042
209
5.1041
4 12 164
5.1045
529417
5.1044
140105 88
5.1039
154 10551 90677
7.1039
1248
7.1047
2106
7.1040
29773362 241966 22903106
6.1041
3119 1936 40
6.1047
4128 253
8.1050
4
8.1042
9.1050
24784
8.1039
89632
8.1044
2840
8.1040
1088
9.1039
10.1040
4
11.1039
957
10.1050
144
11.1040
156
10.1041
32 16
10.1047
87780
10.1045
2152
10.1039
1165
12.1040
145
13.1052
2404
12.1042
24
12.1045
33008
12.1044
8217
12.1041
8
12.1039
135427
14.1049
113
14.1040
229
13.1050
132
13.1042
74433
13.1041
17187
13.1040
2715
13.1039
170059 90
15.1040
16
15.1050
242 1225
14.1050
237
14.1042
6200
14.1041
4
14.1044
720
14.1052
20
14.1039
84 36939
16.1040
88
15.1049
6
15.1039
56
17.1054
2919
16.1050
129
16.1041
36
16.1047
49154
16.1045
2632
16.1049
27
16.1052
16
16.1039
222734
18.1048
133
18.1039
3439
18.1040
241
17.1050
16
18.1049
36 339224
17.1042
172832
17.1041
49620 448
17.1040
1376493 2883
17.1045
72
17.1044
1073
17.1049
55441826
17.1048
124 3648290
17.1052
3547
17.1039
2484110477445
17.1047
24
19.1040
72
19.1039
6 88 60631617
18.1041
28 76
18.1047
3305
18.1054
13744
20.1040
109
20.1039
269
19.1052
8
19.1047
11712
19.1049
10
19.1054
7520
21.1039
82
20.1049
5
20.1047
4896
20.1054
864
22.1040
4
22.1050
23.1040
4
22.1039
144
24.1058
3389
23.1050
76
24.1040
4
23.1042
17528
23.1041
4
23.1054
24
23.1044
6234
23.1058
261
23.1049
4
23.1052
2944
23.1039
3069658
25.1040
80
24.1050
4
24.1039
58
25.1039
26.1040
4
27.1039
489
26.1050
4
27.1040
4
26.1047
3808
26.1045
2248
26.1058
113 80
26.1049
11
26.1039
66
26.1054
840
28.1055
84266 1542
27.1042
1229
27.1041
29619
27.1058
12
27.1049
3984
27.1048
35337
29.1040
262
29.1056
164
29.1039
742
28.1050
4 2464
28.1042
2137912
29.1055
1128
28.1041
4 2633 4
28.1040
2716132 691
28.1058
24 84 32
28.1049
36176 3
28.1048
1192 48
28.1039
365 110957 24475
29.105029.104229.104129.104729.104529.105829.104929.105229.1054
2
3
4
5
6
7
8
0 1 2 3 4 5 6 7 8
Pow
er
(mW
)
Time (s)
35 40 45 50 55 60 65 70 75 80 85 90
-8 -6 -4 -2 0 2 4 6 8
-8
-6
-4
-2
0
2
4
6
8
35 40 45 50 55 60 65 70 75 80 85 90
Temperature (°C)
Position (mm)
Temperature (°C)
ReviewOverviewBreadthDepth
Where to go from here?
Overview
Previous lecture
Energy-efficient sampling for computer vision applications.
Today’s lecture
Overview of home automation standards.
Review of main concepts in course.
Adminstrative details related to final exam.
Ongoing study: sources of information.
Next lecture: project presentations.
2 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Outline
1. Review
2. Overview
3. Breadth
4. Depth
5. Where to go from here?
3 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Understanding embedded system design flow
4 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Main IoT challenges in context of energy-efficient machinelearning
Device cost, thus performance, constraints.
Wireless communication time and energy consumption.
Computatation energy consumption.
Large attack surface.
Varied and sometimes harsh operating environments.
Of these, energy and performance concerns are central for machine learning.
5 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Trade offs between computation and computationefficiency
Can transmit data before intense (high time or energy complexity in terms ofinput size) computation.
Can reduce transmissions by doing local inference, size-reducing featureextraction, or using other compression approaches.
6 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Neural network perspectives
Biological system emulation.
Function approximation
Successive weighted sums (multiply-accumulates or MACs) followed byusually simple, non-linear fuctions.
Back-propagation training.
Goal: Learning and approximating complex functions.
7 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Approaches to efficient machine learning I
Devices enabling low-level hardware structures reflecting the vector/matrixoperations used in the nueral network.
Computer architectures enabling massive parallelism
Communicatio network architectures enabling many-element tomany-element communication as required by many neural network structures.
Network architecture optimization via pruning and merging.
Operand bit width changes.
Error-tolerant computation to improve efficiency.
Using a hierarchy of increasingly computationally intensive algorithms withincreasing accuracies and low false negative rates.
8 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Approaches to efficient machine learning II
Sampling to convert from sparse, large potential input datasets to small,dense datasets.
Another view of this is to eliminate data and associated computationirrelevant to inference accuracy.
9 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Outline
1. Review
2. Overview
3. Breadth
4. Depth
5. Where to go from here?
10 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
What I am attempting to rate
Do you have a broad understanding of research topics and ideas connectedto embedded system analysis, design, and implementation?
Do you have a deep understanding of the challenges facing IoT systems andhow they relate to using machine learning techniques for analysis anddecision making?
11 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Placing out
I may conclude that there is sufficient evidence to assign grades to somestudents without a final exam.
If you are one of these students, I will contact you. You will have the optionto accept the proposed grade or take the final, in which case the final will beweighted in appropriately.
12 R. Dick EECS 507
How to study
Lectures
Review all the lecture notes.
Review video sections on topics you don’t remember well.
Papers
Do you understand the main new ideas in the paper well enough toquickly find important details?
Read your summaries of all papers.
Skim the summaries of two other students.
Use Piazza to discuss ambiguous concepts.
I will check Piazza frequently until the exam.
Projects
Goal, challenges, ideas, and results (did it work?).
Read your notes on student presentations, or read their slides.
Watch videos for topics that aren’t clearly explained in the slides.
ReviewOverviewBreadthDepth
Where to go from here?
Outline
1. Review
2. Overview
3. Breadth
4. Depth
5. Where to go from here?
14 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Topics I
Application trends.
Technology trends.
Costs and constraints.
Specification languages and models.
Allocation, assignment, and scheduling.
Memory hierarchies.
Embedded and real-time operating systems.
Sensors and actuators.
Wireless power transfer applications.
15 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Topics II
Cyberphysical systems.
Energy- and temperature-aware low-power design and power modeling.
Wireless communication and its impact on power consumption.
Reliability-aware design and formal methods.
A little bit of material on testing.
Embedded system security.
Embedded vision applications.
Smartphones.
Wireless sensor networks.
16 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Topics III
Basic understanding of architectures used in application domains.
Wearables.
17 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Outline
1. Review
2. Overview
3. Breadth
4. Depth
5. Where to go from here?
18 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Topics I
Communication, machine learning, and energy efficiency in theInternet-of-Things.
Energy-efficient machine learning algorithms: pruning, BNNs, weightcompression, etc.
Energy-efficient machine learning hardware.
LPWANs.
Reliability, security, and privacy in the Internet-of-Things.
19 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Student projects
Sensing human presence in indoor environments.
Sensing pollinators.
Efficient gesture-based communication.
20 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Papers I
S. A. Edwards, “Design and verification languages,” Columbia University,Tech. Rep., Nov. 2004.
L. Yang, R. P. Dick, H. Lekatsas, and S. Chakradhar, “High-performanceoperating system controlled on-line memory compression,” ACM Trans.Embedded Computing Systems, vol. 9, no. 4, pp. 30:1–30:28, Mar. 2010.
P. Huang, P. Kumar, G. Giannopoulou, and L. Thiele, “Energy efficientDVFS scheduling for mixed-criticality systems,” in Proc. Int. Conf.Embedded Software, Oct. 2014.
E. A. Lee, “The past, present and future of cyber-physical systems: A focuson models,” Sensors, Feb. 2015.
21 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Papers II
L. Zhang, B. Tiwana, Z. Qian, Z. Wang, R. P. Dick, Z. M. Mao, andL. Yang, “Accurate online power estimation and automatic battery behaviorbased power model generation for smartphones,” in Proc. Int. Conf.Hardware/Software Codesign and System Synthesis, Oct. 2010, pp. 105–114.
T. Trippel, O. Weisse, W. Xu, P. Honeyman, and K. Fu, “WALNUT: Wagingdoubt on the integrity of MEMS accelerometers with acoustic injectionattacks,” in Proc. European Symp. on Security and Privacy, Apr. 2017.
J. Polastre, R. Szewczyk, A. Mainwaring, D. Culler, and J. Anderson,“Analysis of wireless sensor networks for habitat monitoring,” in WirelessSensor Networks, C. S. Raghavendra, K. M. Sivalingam, and T. Znati, Eds.Springer US, 2004, ch. 18, pp. 399–423.
22 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Papers III
W. R. Heinzelman, A. Chandrakasan, and H. Balakrishnan, “Energy-efficientcommunication protocol for wireless microsensor networks,” in Proc. HawiiInt. Conf. on System Sciences, 2000.
A. Arcuri, M. Z. Iqbal, and L. Briand, “Black-box system testing of real-timeembedded systems using random and search-based testing,” in Proc. Int.Conf. on Testing Software and Systems, Nov. 2010.
Y. Zhu, A. Samajdar, M. Mattina, and P. Whatmough, “Euphrates:Algorithm-SoC co-design for low-power mobile continuous vision,” arXiv,Tech. Rep., Apr. 2018.
R. P. Dick, L. Shang, M. Wolf, and S.-W. Yang, “Embedded Intelligence inthe Internet-of-Things,” IEEE Design & Test of Computers, Dec. 2020.
23 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Papers IV
B. Widrow and M. A. Lehr, “30 years of adaptive neural networks:Perceptron, madaline, and backpropagation,” Proc. IEEE, vol. 78, no. 9,Sept. 1990.
A. S. Cassidy et al., “Real-time scalable cortical computing at 46giga-synaptic OPS/Watt with 100x speedup in time-to-solution and100,000x reduction in energy-to-solution,” in Proc. Int. Conf. HighPerformance Computing, Networking, Storage and Analysis, Nov. 2014.
Y. Chen, N. Chiotellis, L.-X. Chuo, C. Pfeiffer, Y. Shi, R. G. Dreslinski,A. Grbic, T. Mudge, D. D. Wentzloff, D. Blaauw, and H. S. Kim,“Energy-autonomous wireless communication for millimeter-scaleInternet-of-Things sensor nodes,” IEEE J. on Selected Areas inCommunications, vol. 34, no. 12, Dec. 2016.
24 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Papers V
P. Coussy, C. Chavet, H. Wouafo, and L. Conde-Canecia, “Fully binaryneural network model and optimized hardware architectures for associativememories,” ACM J. on Emerging Technologies in Computing Systems,vol. 11, no. 4, Apr. 2015.
U. Raza, P. Kulkarni, and M. Sooriyabandara, “Low power wide areanetworks: An overview,” IEEE Communications Surveys and Tutorials,vol. 19, no. 2, May 2017.
E. Ronen, A. Shamir, A.-O. Weingarten, and C. O’Flynn, “IoT goes nuclear:Creating a ZigBee chain reaction,” in Proc. Symp. on Security and Privacy,May 2017.
S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. A. Horowitz, and W. J. Dally,“EIE: Efficient inference engine on compressed deep neural network,” inProc. Int. Symp. Computer Architecture, June 2016.
25 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Papers VI
C. Farabet, B. Martini, B. Corda, P. Akselrod, E. Culurciello, and Y. LeCun,“NeuFlow: A runtime reconfigurable dataflow processor for vision,” in Proc.Conf. Computer Vision and Pattern Recognition, June 2011.
C. Gomez and J. Paradells, “Wireless home automation networks: a surveyof architectures and technologies,” IEEE Communications Magazine, June2010.
26 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Outline
1. Review
2. Overview
3. Breadth
4. Depth
5. Where to go from here?
27 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Where to go from here?
Advising offer
Research, product design, or career options.
Projects
Ideas.
What sort of evidence would it take to pass peer review?
Which conferences and journals would be most appropriate?
28 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Status of research area
Moving target.
Need to refresh periodically.
29 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Information sources I
General embedded systems conferences
Embedded Systems Week conferences.
ACM Transactions on Embedded Computing Systems.
IEEE Design and Test of Computers.
CPS-IoT week.
Design automation related
Design Automation Conference.
Design, Automation, and Test in Europe Conference.
IEEE Transactions on Very Large Scale Integration Systems.
IEEE Transactions on Computer-Aided Design of Integrated Circuitsand Systems.
30 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Information sources II
Architecture related
International Conference on Architectural Support for ProgrammingLanguages and Operating Systems.
Wireless sensor networks
ACM Conference on Embedded Networked Sensor Systems.
ACM/IEEE Conference on Information Processing in Sensor Networks.
Mobile and pervasive computing
IEEE Transactions on Mobile Computing.
ACM International Conference on Mobile Systems, Applications, andServices.
IEEE International Conference on Pervasive Computing andCommunications.
31 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Information sources III
Popular media, blogs, funding, jobs, and other related websites
DeepChip.
The Embedded Muse.
Slashdot.
Angel List.
Techstars.
embedded.com.
EmbeddedRelated.com.
Google Scholar.
32 R. Dick EECS 507
ReviewOverviewBreadthDepth
Where to go from here?
Information sources IV
Heuristics
See which papers have recently cited relevant work, to find more recentwork in the field.
Read all the titles, many of the abstracts, and a few of the papers inrelevant conferences and journals.
Attend relevant conferences.
Review the recent publications of important researchers in the area ofinterest; find these based on authorship of high-quality papers.
33 R. Dick EECS 507
