22
C. Thomas Stover www.thomasstover.com New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18 th , 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST

New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

  • Upload
    others

  • View
    17

  • Download
    0

Embed Size (px)

Citation preview

Page 1: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

C. Thomas Stoverwww.thomasstover.com

New DevOps Tools for Native Code

LINUXCON North AmericaWednesday, September 18th, 2013Hyatt HotelNew Orleans, Louisiana11:50 CST

Page 2: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

Revision Control Source Configuration

Collaboration

Packaging &Deployment

Build Automation

Continuous Integration

Test Analysis

Project Management

Some DevOps Interest AreasSome DevOps Interest Areas

(Scope of BCA and Build Matrix)

Page 3: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

Build Configuration Adjust (BCA)

•Web Site: bca.stoverenterprises.com•Git Hub: ctstover/Build-Configuration-Adjust•Google Groups Mailing List: bca-discus•License: GPLv3

NONE.NONE.PROJECT_NAME = "Hello World "BINARY.MAIN.NAME = helloBINARY.MAIN.FILES = ./src/hello.c

Page 4: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

$ ./configureconfigure: running ./bca --configureconfigure: creating directories for build host NATIVE...configure: directory . already exists.configure: directory ./obj already exists.configure: generating Makefile.bcaconfigure: Next use GNU Make to process Makefile.bca, ie "make -f Makefile.bca" or "gmake -f Makefile.bca".$ make -f Makefile.bcagcc -c hello.c -o ./obj/MAIN-hello.ogcc ./obj/MAIN-hello.o -o ./native/hello$ ./native/hellohello world

Page 5: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

Build Tool Roles

•Dependency based, as needed, compilation lines (Make)•Project definition to Makefile generation (Automake)•Portable library abstraction (LibTool)•External / componentized code base integration (pkg-config)•Dynamic source / macro configuration (AutoConf)•Build environment / platform details detection (AutoConf)•Compiler / tool detection selection (AutoConf)•Optional Feature detection selection (AutoConf)•Custom Build Logic (Hard Coded Make, Scripting, utilities in native code)

Page 6: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

Build Tool Roles

•Dependency based, as needed, compilation lines (Make)•Project definition to Makefile generation (Automake)•Portable library abstraction (LibTool)•External / componentized code base integration (pkg-config)•Dynamic source / macro configuration (AutoConf)•Build environment / platform details detection (AutoConf)•Compiler / tool detection selection (AutoConf)•Optional Feature detection selection (AutoConf)•Custom Build Logic (Hard Coded Make, Scripting, utilities in native code)

Page 7: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

BCA Design Ideas

● “normal” build logic: ./configure; make; make install● “normal” command line usage: --disable-*, --enable-*, --without-*, --host=, --prefix=, etc.●installing additional programs not required●human readable Makefiles●human readable project files●human readable build configuration files●Full command line / scriptable, and text file interfaces●Always a simple manual override procedure●minimal to no dependencies: C99-ish●Configure time, and Build time performance really do matter●just abandon non-*nix style build hosts●ground up incorporation of pkg-config●build visualization (human readable graphviz dot output) helps.

Page 8: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

Unix Wars -> Portability in 2013

•Many of the detection tests in Autoconf are: valuable, esoteric, complicated, not worth replacing, not worth discarding, etc.

•Really, package configure (pkg-confg), is a good idea, really.

“Solution” - platform details checking now means either 1) pkg-config, 2) slightly modified autoconf script, 3) any other external script you want to make.

Page 9: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

Big deal:

pkg-config files (.pc) are used for all component interrelationships, both external and internal.

(btw Hacking a .pc to force something to work is infinitely superior to forcing LibTool to do anything.)

Page 10: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

External dependency

Internal dependency

Page 11: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

Packaging vs. Source Installations

•A practical, non-hateful, binary package support abstraction for all meaningful platforms is simply not possible. Really. No seriously.

•You know what? That doesn't matter.

•Source installations are the original portable distribution format.

•Source builds/installations that ACTUALLY WORK, perpetuate maintained, quality in-distribution packages. (and happy users)

Page 12: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

Optional GUI may increase source installation useability.

Page 13: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

Fighting Recursive Makefiles

•Miller, P.A. (1998), Recursive Make Considered Harmful•Console output utterly disgusting and confusing•Parallel build configurations can be accommodated by a single monolithic Makefile.•Use make -j X to full potential

$ bca --generate-graphviz$ dot -Tpng -o bcaproject.png bcaproject.dot

Page 14: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

New Work flow Model

Run a greater and greater number ofbuild configurations, before handingoff to continuous integration.

Page 15: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

New Work flow Model

Corollary: Let the build system - notcontinuous integration - driveparametrized builds.

Page 16: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

Build Matrix•Web Site: buildmatrix.stoverenterprises.com•Git Hub: ctstover/Build-Matrix•Google Groups Mailing List: build-matrix-disc

Page 17: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

Redundant Facilities in Continuous Integration (CI)

●SSH●Cron●Bash / Scripting Languages●User Access Paradigms●GNU Screen (slightly)●mail●Linux / Unix in general●build system logic (slightly)●version control logic (slightly)

Page 18: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

What are the missing pieces?

•A Scoreboard•A Test History•A Build History

Page 19: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

Build Matrix Design Ideas

● ssh (or fork()/execv() ) is the only relevant transport for this●use system users ●SQLite + directory tree for big files as backing store●distributed: peer to peer push + pull + list operations●web interfaces, GUI tools, test analytics can each get there own full copy to play with / mine●again just C99 ●very low overhead●no persistent daemon●just a “dumping ground” for build output and test results●minimum scope possible

Page 20: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

Leaving Plantation J/X-Unit

suite: stringstest: escapenull: passedtest: escapeempty: passedtest: escapenotneeded: passedtest: escapeqoutes: passedtest: escapespaces: passedtest: escapemixed: passedtest: containsnullsource: passedtest: containsnullsearch: passedtest: containsnot: passedtest: containstring: passedtest: stringarrayaddnull: passedtest: stringarrayaddstr: passedtest: freestringarray: passedtest: stringarraynoduplicates: passedtest: pathextractnullpath: passedtest: pathextractnulloutput: passed

Page 21: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision
Page 22: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision

BCA - very near first release candidateBuild Matrix - nearing feature complete alpha prototype

cts at thomasstover.com