mvQuery User Course
Using XP practices on 1960s green screen technologyThe system that time forgot
Synopsis: Travis Perkins has introduced some very modern technologies in the last few years, but half of the 120-strong software engineeringteam is actively developing in BASIC on its 1960s green screen technology platform. Brian and Nik show how it is possible to introduce XP practices to technology that predates the moon landings, and will discuss lessons learned. They will cover unit testing, source control, and continuous integration, and will touch briefly on future objectives. Slide 1ContextHow we wentfrom thisto this
Wider ContextSenior management buy-inMore team-workingIts not legacy, its a heritage system.
Pairing, independent teams, breaking down of knowledge silos, closer involvement of stakeholders.Knowledge silos and functional teams.
Travis Perkins has embarked on a transformation of its IT practices with support and leadership from senior management.Where before small teams chipped away at the code face in single-function isolated silos, now they have grown their teams and embraced Agile principles, pairing and close working with stakeholders. Slide 4The technologyWhat is It?The OriginalNo SQLDatabaseWhat is UniVerse? Slide 6The UniVerse Environment
UniVerse is a business platform with a rich history, categorized as a MultiValue Database (MVDBMS).
It can be accurately described as the original NoSQL database, with many of the features found at the heart of the NoSQL model complex data stored in dynamic hashed files, embedded languages and a separation between the metadata and the data. Optimized for transactional operations, it models real world business entities without excessive normalization whilst still retaining relationships between entities.
A History LessonNELSONPICKORACLESQLSQL SERVERUNIVERSENOSQL1960s1970s1980s1990sUNIX
REALITYSYBASEMySQLGIM-1 Slide 7The UniVerse Timeline
1965 Don Nelson writes the first complex database paper for the US military (Global Information Retrieval Language System, later GIM-1)1967 Don Nelson and Richard Pick co-develop the PICK System1970 Codd writes his paper on relational theory for visualizing data.1973 Microdata REALITY formed a flavour or PICK and the original host of the TP application.1974 IBM creates SQL with help from the PICK engineers.1977 ORACLE founded1984 SYBASE created to challenge the dominance of Oracle1987 Microsoft releases SQL Server 1.0 (Sybase 3)1987 UNIVERSE released1995 MySQL released1998 NoSQL buzzword coined
UniVerse BASICThe UniVerse Business LanguageRuntime LoadedData centricConcurrentProceduralBusiness focusedEmbedded
No encapsulationNo type-safetyNo standard tooling Slide 9The UniVerse Language
Its BASIC Jim, but not as you know it.
Nik speaks to the cultural challenges and the changes to the engineering teams.But there are specific challenges to applying XP techniques to what is genuinely legacy code.And so very briefly, here is an example of the UniVerse business language (one that fits on the screen!).
UniVerse code is designed for back end operations. For modern applications this combines the business language, data access and database procedure tiers allowing organizations to simplify the design and delivery and accessed through a range of APIs or web services. For applications of the age of the one we are talking about, it enables ancient green screen technologies.
The language runs inside the database, working directly with data and with issues ranging from concurrency control and to print spooling.
BUT like scripting languages it uses run-time typing, no type safety, no levels of encapsulation and has poor tooling support.Do you have global variables ?Hell, yesDoes it have Global Variables?
All variables have global scope within a program or subroutine.
It also has very global variables that can be shared between programs and subroutines.And very, very global variables that exist for the duration of a login session (permanent).
Slide 10Challenges from the UniVerse EnvironmentThe database is NOT a dependencyBad Codeis platform independent Slide 11Specific Challenges
Read any book on unit testing and you will learn from the start about dependencies.Every one of these deals with database as an external dependency.Its hard, its difficult, connections are slow, its someone elses concern.
Wrong! No wonder there are so many bad database systems out there.If you are writing to a database that is part of your concern. If the connection is slow that is a concern. Fix it.With UniVerse the code is running inside the database. We cant ignore it.
Actually you can write fast, well formed and fully test covered UniVerse code. The fact that it is procedural changes the rules and process but not the outcome. Good UniVerse code is good code.
This is old code. This is bad code. Thats a different story.TDDWhat engineers were seeingTDD + FitNesse training for Java/.NETSometimes learning Eclipse, and Java, tooThere was a basic test templateWith if/then instead of assertCould not see how unit testing applied to them:Procedural codeNot designed to be testedNot part of the cultureWhat Engineers were Seeing
The engineers were learning java or sitting next to teams using java.So they were seeing those engineers working with TDD, frameworks and standard build processes.And they thought does this apply to us? Slide 13The Challenge In Numbers40,000+ concurrent connections 4x HP Superdomes 1 million+ reads/machine/second 12,000+ programs 4,644,481 lines of application code
31% have cyc. complexity > 50 12 TB changes every day 120 software engineers ... And growingOver 700 lines of code added every working day for 25 years Slide 14The Challenge In Numbers
These speak for themselves.
Stu had to change four lines of code in a small routine.Here is the context .. Slide 15
.. And a bit more .. Slide 16
.. And a bit more ..
.. And a bit more ..
(This is one of the shorter routines. And Remember: every variable is global) Slide 18Lest we ForgetThis application has powered a business for over 25 yearsto an annual turnover of 5 billion pounds
Slide 19Lest We Forget
Titter Ye Not!
Before we launch into describing the work that has been done to modernize the IT culture, it is important to bear one critical fact in mind.
This application continues today to do what it has done for the last 25 years to deliver a business currently operating at 5 billion pound revenue and targeting a growth to 10 billion in the next few years. Warts and all it has allowed the company to grow through acquisition and investment to become the leading name in its sector and at time of writing ranked 80th in the FTSE 100.
Thats a lot of concrete.
What we didWrote our own frameworkCreated own refactoring catalogueStarted UniVerse-specific TDD trainingRewrote the frameworkSimpler APICollaborative effortExtended the frameworkFile configurationSubroutine fakes
What We Did
We wrote our own unit test framework for UniVerse and taught engineers about unit testing and pair programming.
Then we rewrote the framework to simplify and extend it as the engineers began to take ownership and to request features.The we extended it with configuration and faking. Slide 20Some Lessonsand shared..Via refactoring catalogueVia TDD championsCritically, they drive each otherStarting to inherit testsCommunity site for discussion of techniquesLearned ..Units are different in UniVerse.The database is under testThere are limits to breaking down procedural code.Some Lessons
Units in UniVerse are different. The concept of a unit covers a wider block you cant build in Single Responsibility Principle in the same way. So we developed patterns for splitting and refactoring code to make the sections addressable for testing.
The database is part of the code: we cant separate it out as a dependency (though faking to isolate tests is possible).
UniVerse engineers like TDD! And have taken it on board enthusiastically.. Eventually. Slide 21Areas of resistanceIts a waste of timeIt takes too longWere going to throw it away anywayWe cant spend time on it if our product owner doesnt prioritise itEffective pairing is more difficult with procedural code.We had to name the frameworkA framework has to have a name. All our unit test program names started UNIT. But UniVerse has a limit of 62 characters for a label, and the team was worried about wasting them. So they elected to drop one character, and thus the framework was named Slide 23We had to name the frameworkUNT
A token of Regard
From the team to Nik... Slide 25GitExperiences Introducing git for UniVerseFebruary 2013:TFS introduced for source controlVery slowFile-driven/fragmented in a central repository
April 2013:CloudShop moved to git (practical and strategic), trigger for similar move for UniVerse
Ambition:Switch over in June firebreak, but lots of anguishThe codebase did get source control in early 2013, but TFS had its problems. The Java teams move to git was the catalyst to move to the UniVerse teams, too. Slide 27Git AnguishChange of workflowWhole codebase snapshotsFile lockingInclination to want all problems are solved