T.C.

MARMARA UNIVERSITY FACULTY of ENGINEERING

COMPUTER ENGINEERING DEPARTMENT

CSE497 Engineering Project Project Specification Document

“INTELLIGENT WALL CONSTRUCTION BY MEANS OF A ROBOTIC ARM”

Group Members 150112072 Filiz DALIP

150112922 Moustafa MEMET

Supervised by Asst. Prof. Peter Schüller

Co-Advisor

Assoc. Prof. Borahan TÜMER

1. Problem Statement

Nowadays arm robots are used almost everywhere like welding, material handling,

painting and assembling products. When we look at these, we can spot a huge gap in

construction sector. In the very near future we might need the help of some arm robot to

construct us small buildings or miniature houses using wall materials like bricks. We study

this topic to build skills in this important area.

2. Problem Description and Motivation

Description

Robotic arms have become the keystones of industrial production facilities. Robots

are widely used in heavy industrial cutting and automotive industry. However we noticed that

the robotic arms are still not well developed in construction sector. With this project we are

going to realize construction automation and it’s going to be a major step in creating huge

construction facilities. We will create both hardware and software for autonomous wall

building. Because autonomy is important, so that humans do not need to specify each small

movement of the robot.

The working principle of our robotic arm is like this: The arm will reach the location

where the lego bricks are given as input by moving on the conveyor belt. As soon as it

reaches there system will measure the size and the dimensions of the input brick. Then the

gripper will grab the piece and it will place it to best available position. This way, by

minimizing the supply usage we optimize the construction.

Motivation

Robotic arms which are used in industrial sectors becoming brick constructing and

wall building robots is our biggest motivation. Along with that, by using image processing

technology, building a wall with lego pieces and lego statues with robotic arms can have a

remarkable contribution to lego sector. Also with today’s technology we can make anything

we want with unlimited Lego pieces using Lego Digital Designer. On top of that they are

making games and films with Lego for different platforms.

1

3. Aims of the Project

★ Construct a jointed arm robot by using Lego pieces which is similar to Figure 1. ★ Move the robotic arm right, left, up and down stably. ★ Be able to hold the lego piece with the gripper. ★ Mount the robotic arm to conveyor belt (Figure 2), and remove the bottom rotation

motor. ★ Stabilize the robotic arm and conveyor belt so that the vibration minimizes. ★ Place a webcam onto gripper. ★ Perceive the size and dimensions of a lego piece using image processing with a

webcam. ★ Compute the best available position to place the lego piece on the wall. ★ Place the lego piece that we picked up in our grip to the best available position. ★ Build a wall with the lego pieces by placing them into right positions. ★ The main purpose of this robot will be to build a wall by using Lego pieces

autonomously.

Figure 1: Example Jointed Arm Robot

Figure 2: Conveyor Belt Assembly

2

4. Related Work

The International Organization for Standardization (ISO) gives a definition of an industrial robot like this: "an automatically controlled, reprogrammable, multipurpose, manipulator programmable in three or more axes, which may be either fixed in place or mobile for use in industrial automation applications."

Quoted from Wikipedia. See [1] for more.

What we are going is almost the same with this definition of industrial robot. It will be indeed reprogrammable and automatically controlled but it will be working stationary in 2 axes.

Industrial robots usually consist of a jointed arm (multi-linked manipulator) and an end effector that is attached to a fixed surface. One of the most common type of end effector is a gripper assembly.

Quoted from Wikipedia. See [1] for more.

As described in the quote above we are going to make the most common type of end effector. Our gripper will be used to hold the lego pieces and release them to correct positions.

There are different types of industrial robots:

Cartesian Robot: Used for pick and place work, application of sealant, assembly operations, handling machine tools and arc welding. It’s a robot whose arm has three prismatic joints, whose axes are coincident with a Cartesian coordinator.

Cylindrical Robot: Used for assembly operations, handling at machine tools, spot welding, and handling at diecasting machines. It’s a robot whose axes form a cylindrical coordinate system.

Spherical Robot: Used for handling machine tools, spot welding, diecasting, fettling machines, gas welding and arc welding. It’s a robot whose axes form a polar coordinate system.

Scara Robot: Used for pick and place work, application of sealant, assembly operations and handling machine tools. This robot features two parallel rotary joints to provide compliance in plane.

Jointed Arm Robot: Used for assembly operations, diecasting, fettling machine, gas welding, arc welding and spray painting. It’s a robot whose arm has at least three rotary joints.

Quoted from Wikipedia. See [2] for more.

We are going to build our robot on a conveyor belt which will allow us to move it in

x-axis. The arm part which will be placed on top of the conveyor belt will make movements

in y-axis. Since its moving in only 2 axis, our robot will belong to cartesian robot category.

3

Minimization of human power, increasing the production by reducing the error

margin, makes a great improvement for today’s technology. Along with that the robotic arm

which was created with high expenses produces the desired product flawlessly by

assembling the lego pieces together. You can find the article about this robotic arm in

references section [3].

Another project accomplishes almost the same mission as the mega robot above.

This one is created using LEGO Mindstorms NXT kit [4] and its working principle is slightly

different than the other one. It’s created using the 3D printer logic. The desired piece is

placed to correct place stably by using 3 motors. Design and the related video can be found

in references section [5].

5. Scope of the Project

Our scope begins with the well assembly of the robotic arm and the conveyor belt.

We used the Mindstorms EV3 45544 set [6] to build a cartesian robot. After making some

motor and durability tests we made some changes on the original arm. By changing some

bricks we rotated the gripper part of the arm to suit our needs. We are going to integrate a

conveyor belt which is made with LEGO Technic 42028 Set. This belt will be placed

underneath robotic arm to move it linearly to right and left.

As soon as we arrange the arm system onto belt we are going to mount a small

webcam onto gripper to read the image of the lego pieces which are given as input. Here we

are going to analyze the image using MATLAB’s image processing functions. We are going

to measure the dimensions and the size of the lego piece and use an algorithm to find the

best possible position to place the brick.

Using these combined we are going to build a single wall which is made from lego

bricks. After successfully building a wall if there is time left to make a house using these

walls.

4

6. Success Factors and Benefits

We can conclude that our project has become successful only if the following

statements are satisfied:

★ Robotic arm grabbing the lego piece from their place

★ Robotic arm moving freely and flawlessly on the conveyor belt

★ Understanding the grabbed piece correctly using image processing

★ Placing the lego piece to correct place by using the correct algorithms and

calculations

After achieving the above we are going to arrange the software of the robotic arm to

build us a wall autonomously. We are going to make an input location for lego bricks on the

table. Robot will always come back to that location to obtain new bricks. According to given

bricks the robot will place them to the best available position. The only human interaction in

this process is going to be giving the input bricks.

The benefits of the project can be listed as follows:

★ The most important benefit might be the reduction of the workload for the workers

because of its use in construction industry.

★ If the application of the project is done correctly it might provide a great profit to the

owner. Because robotizing one job improves the quality of products and reduces the

expenses drastically.

★ It can make contribution to economy and technology can be sold to foreign countries.

★ The gap in the construction sector might be closed with the autonomous builders like

this robotic arm.

★ With the replacement of the gripper part, the arm can also be used to spill cement

between the bricks.

Along with these a complete schematic of the total project can be taught to robot.

Getting this done can give you a complete building with the help of worker robots.

5

7. Methodology and Technical Approach

Figure 3: Flowchart of the tasks

Our project consists of three parts. These are hardware and robot assembly, robot

movement algorithms and software and image processing software.

The things that we need for the robot assembly:

❖ Lego Mindstorm EV3 Set 45544 [6]

❖ Robot Arm

❖ Programmable Brick

❖ Motors

❖ Touch Sensor

❖ Conveyor Belt

❖ Camera

❖ Lego Pieces

Lego Mindstorm EV3 Set 45544 [6]: This set consists of all the following parts that we need

for our project.

Robot Arm: Using this set we can make number of different designs. What we are going to

build is the robotic arm with some modifications.

6

Programmable Brick: The EV3 Brick serves as the control center and power station for your

robot.

Large Motor: Lets you program precise and powerful robotic action.

Medium Motor: Maintains precision, while trading some power for compact size and faster

responses.

Touch Sensor: Makes your robot respond to touch, recognizes three conditions: touched,

bumped and released.

Conveyor Belt: We will use Lego Technic 42028 Sets to make suitable conveyor belt to

mount the robot arm.

Camera: A small webcam will be used to read the dimensions and the size of lego bricks.

This webcam will be mounted onto gripper.

Lego Pieces: We are going to need a lot of pieces to make tests.

The things that we need for the robot movement software:

❖ MATLAB [7]

❖ LEGO Mindstorms EV3 library for MATLAB [8]

❖ MathWorks Image Processing Toolbox [9]

MATLAB [7]: MATLAB (matrix laboratory) is a multi-paradigm numerical computing

environment and fourth-generation programming language. A proprietary programming

language developed by MathWorks, MATLAB allows matrix manipulations, plotting of

functions and data, implementation of algorithms, creation of user interfaces, and interfacing

with programs written in other languages, including C, C++, C#, Java, Fortran and Python.

LEGO Mindstorms EV3 library for MATLAB [8]: Mathworks provides free libraries for LEGO

Mindstorms EV3 robot control. We are going to use these libraries to move our robot.

MathWorks Image Processing Toolbox [9]: This toolbox consists many image processing

functions. We are going to use these libraries to understand the dimensions of the lego

pieces that we grab.

7

Image processing software and algorithms:

Image Processing is processing of images using mathematical operations by using any form of signal processing for which the input is an image, a series of images, or a video, such as a photograph or video frame; the output of image processing may be either an image or a set of characteristics or parameters related to the image. Most image-processing techniques involve treating the image as a two-dimensional signal and applying standard signal-processing techniques to it. Images are also processed as three-dimensional signals where the third-dimension being time or the z-axis.

Quoted from Wikipedia. See [10] for more.

In our case, the images that we are going to process are going to be photographs

which are taken by our webcam. These images will be 3-dimensional signals, each

representing a color space (Red, Green, Blue). These will be analyzed with the help of

MATLAB’s image processing toolbox.

We are going to integrate robot movement and image processing together using the

MATLAB software. MATLAB can support the Mindstorms EV3 controls using its add-on

extensions [8].

We will connect the EV3 brick to our computer along with the webcam that we are

going to mount onto gripper. Webcam will receive the image and convert it to JPEG format.

We will analyze these images using MATLAB and understand the dimensions of the given

lego piece. Mathworks also provides a huge library [9] for image analysis and segmentation.

Using these libraries we are going to develop our software on MATLAB.

8

8. Professional Considerations

Methodological Considerations

In the figure below you can find the block diagram of the process that our robotic arm

will realize.

Figure 4: Block Diagram of the process

9

We put the input bricks to predefined input place consecutively. Our gripper will take

the piece from there and move it in front of the camera. The camera will capture the picture

of the lego brick and will try to understand the size of it. Then it will run some algorithms to

compute the best available position for the new brick. After that we will move the robot to

proper position and leave the brick to best available position. Repeating this process builds a

wall.

Societal and Ethical Considerations

Societal and ethical considerations has been taken into consideration while preparing

this document.

One social impact could be fewer building jobs. Since the robotic arms are going to

take the place of building workers. This also might increase the need of the programmers for

the robots.

Economically this project will help to reduce the human power on some specific jobs.

It’s going to increase the efficiency and improve the construction process by reducing the

time required to complete it.

The realization of this project definitely has no physical and biological impacts. Along

with these, this project will be designed with a good care of environmental conditions. There

will be no harm to natural environment.

Legal Considerations

The usage of MATLAB and its libraries in our project may need licenses and usage

permissions if this project ever comes to market.

Please note that we do not apply the project in reality and the robotic arm that we are

going to build is small so it’s not dangerous. If it was a real size robot, it would need ethical,

work safety, engineering and many more permissions.

10

9. Management Plan

The steps to take and the deadlines to follow in two terms is as follows:

1. Literature research, reading the reviews of the related work and finding ideas to apply

them to our project. (completed)

2. Arrangement of required equipments and additional parts. (completed)

3. Assembly of the parts for prototype.(completed)

4. Preparation of the Project Specification Document. (in progress)

5. Placement of the robotic arm onto conveyor belt. Redesign of prototype.

6. Running tests for motors, sensors and Brick controller. Hardware stability control.

7. MATLAB movement tests.

8. Achieving the movement in 3 axis.

9. Grabbing the lego pieces with the gripper and placing them into positions manually.

10. Webcam installation and and capturing the images.

11. Processing of the images, storing the brick sizes and dimensions.

12. Integration of image processing and robot movement.

13. Development and improvement of algorithms.

14. Placement of the sequentially given bricks to best available position.

15. Running tests with different brick sizes.

Division of responsibilities among teammates

Filiz Dalıp Tasks: 6, 7, 8, 9, 11, 12, 13, 14, 15

Moustafa Memet Tasks: 5, 8, 9, 10, 11, 12, 13, 14, 15

We have successfully completed up to 4th task. Within the following weeks we are

going to complete the hardware design of the project. Up to 6th phase we are working on our

hardware design. We will gently try to make our arm more stable that the original. Because

the precision is going to be really important in the following phases. Running some stability

tests may take some longer time.

After completing the hardware, we are going to run some movement tests with the

help of the software. Here we will try and see our design’s capabilities. We are going to

move the robot manually and achieve the movement in 3 axis. All possible movements that

we are going to need later will be realized in phases 7-9.

11

Once we can be sure that our robot has full movement capabilities in 3 axis, we are

going to install a camera and start working on image processing algorithms. Our aim at first

is to be able to read the brick correctly.

As soon as we accomplish reading the images we are going to work on placement

algorithms. We are going to keep the current state of the bricks in our memory and find the

best possible place for the new brick. Until the end of the April we will be working on these

algorithms.

Below you can see the timeline with milestones:

Figure 5: Gantt chart for timeline with milestones

Risk Management

One of the major risks can be the lack of stabilization on our system. We might not

be able to build an arm from legos with full precision. In this case we are going to try our best

to make them stable and strong for our job.

Another risk could be the compatibility of EV3 software with MATLAB. If they are not

fully compatible, we might need to try another way to control the motors.

Reading the bricks from one perspective may not be enough to understand the size

and dimensions of bricks. In this case we might reduce the amount of possible bricks we

use.

12

References [1]- Wikipedia, Robots [Online] Available: https://en.wikipedia.org/wiki/Robot (Date of Access: 24 October 2016) [2]- Wikipedia, Robotic Arms [Online] Available: https://en.wikipedia.org/wiki/Robotic_arm (Date of Access: 24 October 2016) [3]- Singularityhub, Motoman robot impresses with lego building prowess (by Aaron SAENZ) on Jan 08,2010 [Online] Available: http://singularityhub.com/2010/01/08/motoman-robot-impresses-with-lego-building-prowess-video/ (Date of Access: 25 October 2016) [4]- Wikipedia, Lego Mindstorms NXT [Online] Available: https://en.wikipedia.org/wiki/Lego_Mindstorms_NXT (Date of Access: 26 October 2016) [5]- Wired,Machine made of lego builds anything you want out of lego (by Priya GANAPATI) on October 20,2010 [Online] Available: https://www.wired.com/2010/10/legobot/ (Date of Access: 26 October 2016) [6]- Lego Education, [Online] Available: https://education.lego.com/en-au/lego-education-product-database/mindstorms-ev3/45544-lego-mindstorms-education-ev3-core-set (Date of Access: 27 October 2016) [7]-Wikipedia, Matlab [Online] Available: https://en.wikipedia.org/wiki/MATLAB (Date of Access: 28 October 2016) [8]- Mathworks, Lego Mindstorms EV3 support from Matlab [Online] Available: https://www.mathworks.com/hardware-support/lego-mindstorms-ev3-matlab.html (Date of Access: 29 October 2016) [9]- Mathworks, Perform image processing, analysis, and algorithm development [Online] Available: https://www.mathworks.com/products/image/ (Date of Access: 29 October 2016) [10]-Wikipedia, Image Processing [Online] Available: https://en.wikipedia.org/wiki/Image_processing (Date of Access: 30 October 2016)

13