Approaches to Application Modernization

Don Denoncourt

dondenoncourt@gmail.com

Outline

• What is App Modernization and Why Bother?• Modernizing Coders• Code Moves to Data, Modernizing the Database• Gradational versus Global Approach• Fear Factor• Grokking RPG• Save the Program, Save the Platform

"Modernization is our jobs. It’s what IT does. We improve access to information for the businesses we work for."

• Trevor Perry

Traits of a Modern App

• It has maintainable and testable code• It is modular and component-based• It provides a rich user interface• It uses the Model-View-Controller (MVC)

design pattern – Separate presentation, flow control, and business

logic

• It is responsive to new business requirements

Why Bother?

• Modernizing applications is costly – Not modernizing is more costly

• "60 to 80 percent of all software development is enhancements to or maintenance of existing projects."

– Gartner Group report

• Amounts to a 10% increase in legacy code per year

Escalating Costs

• Effort required to enhance and maintain legacy apps escalates with time

• Fact:– Code is read far more often than it is written.

• So:– "What sense does it make to bloat unreadable

code at a rate of 10 percent per year?"

• Reducing maintenance costs may be the biggest reason to modernize

Modular Applications:

• Allow you to quickly respond to business requirements

• Improves interoperability:– Other applications– Languages– Technologies

• Including SOA and REST

Modernizing Coders

Three Contexts of Modernization

1. User Interface

2. Code

3. Coders

Modern UI

• If you're not rewriting your entire app in a new language– Need to modularize RPG

• before adding a Web front-end

• What about… HATS and WebFacing?– WDHT is the Botox of UI modernization

• Quick and easy

• But behind that puffed-up face– There will always be the same old person

Application Make-over

• To make over a person (or an application)– You need to start from the inside

• That means– Attitude– Knowledge– Organization

Attitude, Knowledge, and Organization

• Attitude should come from personal fulfillment– But it can also be coerced from effective management

• Knowledge is acquired from experience or education

• Organization– micro-level

• when restructuring a program

– macro-level • when planning an attack for modernization.

Training

• Formal education – Most common strategy

• But the best training is mentoring

• Conferences and formal classes – are worthwhile

• But mentoring virtually guarantees – New techniques can transform from knowledge

to functioning business code

Mentor

• If you don’t have internal expertise – Get a consultant

• one with mentoring skills

• To substantially decrease costs:– Consider:

• WebEx, GoToMeeting, and alternative web conferencing tools

– Consultant may be off-site and part-time

• Maximize your training budget's ROI

– With 2 hours mentoring a day

Master Programming Toolsets

• Promote use of a modern development tools – WDSc/RDi– Eclipse, NetBeans, Visual Studio

A good IDE provides tools that are necessary in the modernization process.

Grokking the Code with WDSc

Application Diagram

Search and Replace

Outline View

Integrated Help

Streamlined edit-compile-debug

Code Moves to Data

Modernize Your Database

• 20-year-old RPG

– Code manages what should be done with db• Table relationship management• Foreign key constraints.

• Refactoring that code out– Reduces requirement for code

Does your database really need modernization?

• If your files are defined with DDS• If you are not using constraints and triggers• If you are planning to implement Business Intelligence • If you are planning to provide a web front-end to your

data• If new developers need to look at code to understand

your database– Then. Yes. You really need to modernize your database.

Significant Performance Improvements

• DDL-defined table page size much larger that DDS-defined files – DDL page size 64K– DDS page size 8-32K

• Encoded Vector Indexes (EVI)

• SQL query engine (SQE) runs below the MI– Versus the Classic Query Engine (CQE)

http://systeminetwork.com/article/performance-comparison-dds-defined-files-and-sql-defined-files

New Column/Field Capabilities

• BLOBs and CLOBs• Constraints• User-Defined Data types (UDT)• Formula columns• Encryption and decryption• Complex view definitions (logical files)

– Expressions

– Aggregation

– Complex selection

New Column/Field Capabilities

• Long table and view names – 128 characters

• Identity and RowId columns

• Sequences

• Auto-increment of keys

• Column-level triggers

• Column-level privileges (authorities)

Object-relation Mapping Tools

• Generators for ORM engines– Such as

• Java Persistence API

• Hibernate

• OJB

– Work best with a well-crafted database • And standard SQL DDL constructs

Object Traversal• Enabled with 1 database operation via joins

– Versus a read for each row/record – Java:

• event.getVenue().getAddress().getStreet();

– Grails:• event.venue.addresss.street

More reasons to modernize•  SQL is a widely used standard.

– Openness• more and better options for third-party tools

– Books and training readily available

• Performance: – IBM is investing money on improving database access

through SQL, not elsewhere.

• Availability of skills– Easier find Java and SQL skills over RPG and DDS

knowledge.

• Functionality– Some new functions require SQL.

“Code moves to data”

• Business rules – As much as possible

• belong in DDL

• DDL-based Business Rules– Constraints

• Primary and foreign keys• Column-level contraints

– Triggers• Operations to perform on

– Add, update, or delete operations.

Benefits of DDL Business Rules

• Removes duplicate code– Decreasing complexity

– Improving maintainability

• Self-describing• Enforces database integrity

– Rules are always followed• Whether .Net, PHP, or Java web front-end applications

• Or other RPG

SQL's Data Definition Language (DDL)5-minute tour

create table item ( itemid varchar(10) not null, productid varchar(10) not null, listprice decimal(10,2) not null, unitcost decimal(10,2) not null, supplier int not null, status varchar(2) not null, attr1 varchar(80) not null, attr2 varchar(80) not null, attr3 varchar(80) not null, attr4 varchar(80) not null, attr5 varchar(80) not null, constraint pk_item primary key (itemid), constraint fk_item_1 foreign key (productid) references product (productid), constraint fk_item_2 foreign key (supplier) references supplier (suppid));

create index itemProd on item (productid);

• PetStore DB– Used in the ubiquitous

sample web application

create table product ( productid varchar(10) not null, category varchar(10) not null, name varchar(80) not null, descn varchar(255) not null, constraint pk_product primary key (productid), constraint fk_product_1 foreign key (category) references category (catid)); create table category ( catid varchar(10) not null, name varchar(80) not null, descn varchar(255) not null, constraint pk_category primary key (catid));

create table supplier ( suppid int not null, name varchar(80) not null, status varchar(2) not null, addr1 varchar(80) not null, addr2 varchar(80) not null, city varchar(80) not null, state varchar(80) not null, zip varchar(5) not null, phone varchar(80) not null, constraint pk_supplier primary key (suppid));

PetStore on Squirrel SQL Client

PetStore Database onWDSc's Data Perspective

Field-level Constraints

CREATE TABLE orders ( CustNum CHAR(4), OrderNum CHAR(4), Quantity INTEGER CHECK (Quantity BETWEEN 1 AND 50)

);

On Delete and On UpdateCREATE TABLE CORPDATA.EMPLOYEE (EMPNO CHAR(6) NOT NULL PRIMARY KEY, FIRSTNME VARCHAR(12) NOT NULL, MIDINIT CHAR(1) NOT NULL, LASTNAME VARCHAR(15) NOT NULL, WORKDEPT CHAR(3) CONSTRAINT WORKDEPT_EXISTS REFERENCES CORPDATA.DEPARTMENT (DEPTNO) ON DELETE SET NULL ON UPDATE RESTRICT, PHONENO CHAR(4), HIREDATE DATE, JOB CHAR(8), EDLEVEL SMALLINT NOT NULL, SEX CHAR(1), BIRTHDATE DATE, SALARY DECIMAL(9,2), BONUS DECIMAL(9,2), COMM DECIMAL(9,2), CONSTRAINT UNIQUE_LNAME_IN_DEPT UNIQUE (WORKDEPT, LASTNAME) );

Database Reverse Engineering Tooling

• Navigator • WDSc • System i System API• Third-party tools

Navigator has a suite of SQL tools

Note: IBM is phasing out database development with Navigator

WDSc's Database Perspective

Generate DDLon a PF called ALLTYPES

System i API

• Generate Data Definition Language (QSQGNDDL)– Carsten Fleming wrote an iSeries command:GENSQLDDL DBOBJ(yourFile) +

DBLIB(QSYS) +

TYPE(*PF) +

SRCFILE(QGPL/QDDLSRC) +

MBR(yourFile) +

MBROPT(*REPLACE)NAMING(*SQL)        

http://systeminetwork.com/article/apis-example-reverse-engineering-database-files-and-objects-sql-ddl-statements

Gradational versus Global Approach

Two Strategies: Globally or Gradually

• Global modernization – Management backs the project with bucks– Bring in consultants – Purchase a modernization tool-set

• such as Databorough’s X-Analysis.

• Gradational modernization– Refactor code on an ongoing basis during

normal business development

Global Realities

• Management often will not back such a concentrated effort

– Even though, in the long run, • It is the most cost effective

• There are two alternatives to this 1. Do nothing

• And let your code swell beyond recognition

2. Improve code with the gradational approach.

Gradational Philosophy

"Regard every bug fix or enhancement as an opportunity to improve code."

Gradational Realities

• Consider a change request – Estimated as a 10- to 20-hour project

• Quick-and-dirty status-quo

• Opportunity to modernize

• First time realities– Potentially take 80–140 hours

• Time will appreciably decrease on each new request

Gradational Benefits

• Soon you will be more receptive change requests

• Along with – Increased adaptability– Greater interoperability– Reduced code bloat

Fear Factor

Fear Factor

• Strategy a bit too radical?

• Scared to make far sweeping changes to a program?

• Frightened because you can’t deduce the impact of those changes?

Fear is Why Legacy Code Bloats

• Instead of correcting existing code…– We add:

• Global variables

• Subroutines

• Or copy the program and modify the new on

• We do this so we know our new code won’t affect old code

We Need Confidence

• Conviction that changes will work in production

"The only way to be confident about code changes is automated testing."

"If you can’t figure out how to test code, don’t bother writing it. "

Refactoring

• Well known software engineering practice– Changing code to simplify or improve

readability without changing the result.

• Automated unit testing ensures– Code modifications cannot cause the

application to fail

RPG Unit Test Example

/free

// test encryption encrypted = IMAXgetIFSFileAsBase64('/rxs/contract/irm/dontdelete.xml'); assertEquals(IMAXgetIFSFile('/rxs/contract/irm/dontdelete.xml'): IMAXdecryptBase64(encrypted));

// test uploadFiles web service with 2 good files, and 2 bogus recipients

paths(1) = '/rxs/contract/irm/US0106091235211t.TXT'; paths(2) = '/rxs/contract/irm/duplicate.txt'; recipients(1) = 'Sam'; recipients(2) = 'Will'; uploadRsp = IMAXuploadFiles('TRACM01908':'rpdp01': paths: recipients: 'RXS test request'); assertEquals('500': %char(uploadRsp.statusCode)); kar = uploadRsp.statusMsg; assertEquals('Internal System Error - Invalid users:SAM,WILL': kar);

// test uploadFile with the same 2 good files and 1 valid recipient reset recipients; recipients(1) = 'TRACMTEST'; uploadRsp = IMAXuploadFiles('TRACM01908':'rpdp01': paths: recipients: 'RXS test request'); assertEquals('500': %char(uploadRsp.statusCode)); paths(1) = '/bogus/bogus.TXT';

Gradational 4 Steps1. Write a wide range of unit tests over existing code

– Run them before any modifications are made.

2. Make modernization changes• Without adding bug fixes or enhancements

• Rerun the tests.

3. Add tests for the change request• Run the tests

• which will fail

4. Make the change • Iteratively rerun tests and tweak the code

• Until the tests succeed

And consider fear no longer a factor.

Confidence with Source Control

• Source control make you bold with code changes– Without the ability to revert to a prior version

of code• Coders are reluctant to make extensive changes

• Use a source-control product – Don't rely on manual methods.

• Use open-source CVS and Subversion tools. – or one of the various commercial products

Grokking RPG

Grokking the Code

• When a change request comes in– Don't hack the change– Grok the code and modernize

• Grok means:– "Understand profoundly and intuitively"

Grokking RPG• Rip the program apart with both your brain and

WDSc tooling until you fully comprehend it. • If you can't Grok RPG because of a poorly named

variable or routine– Rename it to something more meaningful– Run unit tests to prove that your refactoring didn’t

break the code

• Discover business entities – Those not already in record formats– Define external structures for them.

• Refactor the code to use the external entity structure

• Recompile and retest

Separating UI from Biz Logic

• First step to modernizing

• Separating business logic from display gives three advantages:

1. You can create automated tests

2. You can reuse the modules in other applications

3. You can more easily interface with web technologies

Implement the Change Request

• With UI and Biz logic separate– And perhaps readability improvements

• Now ready to implement the change request

Code Change Process

• If change request requires modification of a subroutine– Convert it to a sub-procedure

• Remove global variables in the subprocedure– May force you to add parameters to the subprocedure

• But enables unit testing

• Note that file parameters are available for RPG with V6R1– File I/O was the last vestige of requirements for global

variables in modern RPG

Code Change Process continued…

• With global variables removed– The subprocedure can be placed in its own source

member

– And becomes a module

• Create unit tests for the subprocedure• How did we used to test subroutines?

– We didn’t, because we couldn’t.

• Subroutines weren’t modular– so automated atomic testing was not an option

• With subprocedures– Unit testing should be a requirement

Save the Program, Save the Platform

Code Inventory and Empirical Study

• Do an inventory of your applications– What are the assets

• Liabilities

• Do an empirical study – What programs give you the most trouble?

• Or require the most frequent changes

– Modernize those first

Getting Started

• It costs less to modernize than to – Put your coding heads in the sand

• And ignore code bloat

• Deal with the fear factor • Mitigate risk with unit testing and source control• Get a mentor• Promote the use of WDSc and improved tooling• Start code reviews

RPG is One of IBM i's Greatest Strengths

• Every RPG you rejuvenate strengthens the platform

systeminetwork.com Articles

• "Grokking Software Development" (August 2007, article ID 20979 at SystemiNetwork.com),

• iUnit from SystemiNetwork.com/code (select April 2007).

• Instructions for iUnit • "Developing Reusable RPG APIs" (February

2006, article ID 20395) • "Testing RPG-based Stored Procedures with RPG"

(April 2007, article ID 20866).• "RPG Source Control for Free" (August 2005,

article ID 20205),