Upload
phammien
View
221
Download
1
Embed Size (px)
Citation preview
Feiner, COMS W4170, Fall 2017
1
COMS W4170Programming by Demonstration 3Steven FeinerDepartment of Computer ScienceColumbia University New York, NY 10027
November 28, 2017
2
Chimera D. Kurlander 1988–92
Graphical history can be used to select past actions
7:00-14:30https://www.youtube.com/watch?v=JbrJQW25ekI
Feiner, COMS W4170, Fall 2017
3
Chimera D. Kurlander 1988–92
Macro creation
Confirmation of inferences Macro Invocation
4
Chimera D. Kurlander 1988–92
Testing
Feiner, COMS W4170, Fall 2017
5
Chimera D. Kurlander 1988–92
Debugging
6
Chimera D. Kurlander 1988–92
Testing again
Feiner, COMS W4170, Fall 2017
14
Generating Photo Manipulation Tutorials F. Grabler et al. SIGGRAPH 2009
15
Generating Photo Manipulation Tutorials F. Grabler et al. SIGGRAPH 2009
Implemented in GimpCapture UI state and app state
Prune changes that don’t make a difference
Use existing vision-based recognition/labeling techniques to classify parts of images
E.g., faces, landscapesUse heuristics to generate related text
Relative location, “nose”, “eyes” (and parts: “left pupil” “right iris”, “sky”)
Feiner, COMS W4170, Fall 2017
17
Generating Photo Manipulation Tutorials F. Grabler et al. SIGGRAPH 2009
Tutorial GenerationOne step per major image processing op
Aggregate related and repeated actions (e.g., parm tweaking)
Annotate screenshots with arrows and highlights corresponding to user actions
Crop to include only “relevant” portionsGenerate text descriptions
Template-based “fill in the blanks” approachLay out screenshots and text in grid
Use text (on left)-screenshot (on right) pairsUse before-after screenshot pairs to show consequencesUse screenshots of multiple actions when necessary
Stack pairs in columns, laid out left-to-rightInclude before-after screenshot pair at left for entire task
Berthouzoz et al., ACM Trans. on Graphics 2011 generalizes to create parameterized macros
18
Generating Photo Manipulation Tutorials F. Grabler et al. SIGGRAPH 2009
Follow-on work: Adobe Tutorial Builder for Photoshop
http://labs.adobe.com/technologies/tutorialbuilder/
Feiner, COMS W4170, Fall 2017
19
Excel 2013 (, 2016,…) “Flash Fill”S. Gulwani
Infers iterations over cells based on one or a small number of examples
Type and press <enter> to accept suggestionsOtherwise press <ESC>Or use ctrl-E to execute
http://office.microsoft.com/en-us/excel-help/split-a-column-of-data-based-on-what-you-type-HA102809794.aspx?CTT=5&origin=HA102809308
https://www.microsoft.com/en-us/research/project/flash-fill-excel-feature-office-2013/
0:10−
20
Apparition W. Lasecki et al., CHI 2015
Crowd-poweredrefinement of UI prototypeCrowd-powered “wizard-of-Oz”animation of prototype inresponse to designer’s spec
Crowd-powered refinement of prototypehttps://www.youtube.com/watch?v=tBCB6P7FwWY
Feiner, COMS W4170, Fall 2017
21
SketchExpress S. Lee et al., UIST 2017(built on Apparition)
Crowd-powered creation, editing, and saving of animated behaviors for UI prototypes by workers“Demonstrate–remix–replay” approach
Worker candemonstratebehavior that isrecorded,remix it to modifyparameters, andreplay it whenneededRecordedbehaviorsare saved
https://www.youtube.com/watch?v=A_Pngz1mbDs