CPAN6CPAN6Mark OvermeerMark Overmeer

YAPC::EU 2007YAPC::EU 2007LinuxConf EU 2007 (subset of slide)LinuxConf EU 2007 (subset of slide)

index

● historyhistory● collectingcollecting● collecting collectionscollecting collections● CPAN6 and Pause6CPAN6 and Pause6● releasesreleases● use-casesuse-cases● initial resultsinitial results● statusstatus

history

● Perl5 has CPANPerl5 has CPAN● CPAN is very successful:CPAN is very successful:

>12k modules by >3500 authors>12k modules by >3500 authors● designed in 1995: very simple web-interface, based on “old designed in 1995: very simple web-interface, based on “old

technologies”technologies”● strongly entangled with Perl5strongly entangled with Perl5

history

● Perl5 has CPANPerl5 has CPAN● CPAN is very successful:CPAN is very successful:

>12k modules by >3500 authors>12k modules by >3500 authors● designed in 1995: very simple web-interface, based on “old designed in 1995: very simple web-interface, based on “old

technologies”technologies”● strongly entangled with Perl5strongly entangled with Perl5

● Perl6?Perl6?● modules, PIR, PASM, ...modules, PIR, PASM, ...● ParTcl, Pieton, Lua, ... parrot friendsParTcl, Pieton, Lua, ... parrot friends

where wil we collect it? where wil we collect it? CPAN6 CPAN6

history

● CPAN6 started as non-(perl-)community investigation CPAN6 started as non-(perl-)community investigation into needs and possiblitiesinto needs and possiblities

history

● CPAN6 started as non-community investigation into CPAN6 started as non-community investigation into needs and possiblitiesneeds and possiblities

● CPAN6 can be used to collect and distribute everything CPAN6 can be used to collect and distribute everything (also Perl modules)(also Perl modules)

history

● CPAN6 started as non-community investigation into CPAN6 started as non-community investigation into needs and possiblitiesneeds and possiblities

● CPAN6 can be used to collect and distribute everything CPAN6 can be used to collect and distribute everything (also Perl modules)(also Perl modules)

● CPAN6 may never be used for Perl (and may need a CPAN6 may never be used for Perl (and may need a different name)different name)

CPAN, the goods and bads

● What makes CPAN successfulWhat makes CPAN successful● simple to use; anyone can contributesimple to use; anyone can contribute● open and accessibleopen and accessible● at lot of useful stuffat lot of useful stuff● very stable interfacevery stable interface

CPAN, the goods and bads

● What makes CPAN successfulWhat makes CPAN successful● simple to use; anyone can contributesimple to use; anyone can contribute● open and accessibleopen and accessible● at lot of useful stuffat lot of useful stuff● very stable interfacevery stable interface

● What are the problemsWhat are the problems● simple to use: no protectionssimple to use: no protections● no described rules; needs GBDsno described rules; needs GBDs● too much useless stuff; name-space polutiontoo much useless stuff; name-space polution● no major changes since longno major changes since long

abstraction

● Goal: distribute Perl software releases from the Goal: distribute Perl software releases from the publisher to the user.publisher to the user.

abstraction

● Goal: distribute Perl software releases from the Goal: distribute Perl software releases from the publisher to the user.publisher to the user.

● Goal: distribute Perl software releases from the Goal: distribute Perl software releases from the publisher to the user.publisher to the user.

abstraction

● Goal: distribute Perl software releases from the Goal: distribute Perl software releases from the publisher to the user.publisher to the user.

● Goal: distribute Perl software releases from the Goal: distribute Perl software releases from the publisher to the user.publisher to the user.

● Goal: distribute Perl software releases from the Goal: distribute Perl software releases from the publisher to the user.publisher to the user.

release = a set of filesrelease = a set of files

Nederlands Openluchtmuseum

● Nederlands OpenluchtmuseumNederlands Openluchtmuseum(Dutch Open-Air Museum)(Dutch Open-Air Museum)

● ““European Museum of the Year” in 2005European Museum of the Year” in 2005

● Old history (windmills!) andOld history (windmills!) andModern history (1950-)Modern history (1950-)

“Spaarstation Dingenliefde”

● Exposition about people collecting thingsExposition about people collecting things

garden gnomesgarden gnomes

shopping listsshopping lists money boxesmoney boxes crucifixescrucifixes

sick bagssick bags pipespipes tattoostattoos

Collecting

● People are collectors, in naturePeople are collectors, in nature

● We collect more than we can handleWe collect more than we can handle● archivists, librarians, and bookkeepers come to the rescue.archivists, librarians, and bookkeepers come to the rescue.

● Not everyone is well organizedNot everyone is well organized● have a look at your attic!have a look at your attic!

(or agenda, home-dir, brain, desk-top, kitchen, ...) (or agenda, home-dir, brain, desk-top, kitchen, ...)

where do we keep out things?

● Let us focus on digital Let us focus on digital collections:collections:● softwaresoftware● photos, filmsphotos, films● musicmusic● documentationdocumentation● e-maile-mail● ......

digital collections

● In a directory structureIn a directory structure● on (removable) disk, private useon (removable) disk, private use

accessible on the local systemaccessible on the local system

● on a network disk, group useon a network disk, group useaccessible on the local networkaccessible on the local network

● on an ftp-server, global useon an ftp-server, global usefor world-wide accessfor world-wide access

digital collections

● In a directory structureIn a directory structure● on (removable) disk, private useon (removable) disk, private use

accessible on the local systemaccessible on the local system

● on a network disk, group useon a network disk, group useaccessible on the local networkaccessible on the local network

● on an ftp-server, global useon an ftp-server, global usefor world-wide accessfor world-wide access

● Applications do not care about the needs for collecting. Applications do not care about the needs for collecting. Who does care?Who does care?

three levels of CPAN6

● the CPAN6 the CPAN6 infrastructureinfrastructure● distribution of “released” digital information over the network. distribution of “released” digital information over the network.

Platform and implementation independent. Platform and implementation independent.

three levels of CPAN6

● the CPAN6 the CPAN6 infrastructureinfrastructure● distribution of “released” digital information over the network. distribution of “released” digital information over the network.

Platform and implementation independent. Platform and implementation independent.

● the Pause6 the Pause6 archiverarchiver implementation implementation● upload/download rights, trust, name-space administration, upload/download rights, trust, name-space administration,

rules.rules.

three levels of CPAN6

● the CPAN6 the CPAN6 infrastructureinfrastructure● distribution of “released” digital information over the network. distribution of “released” digital information over the network.

Platform and implementation independent. Platform and implementation independent.

● the Pause6 the Pause6 archiverarchiver implementation implementation● upload/download rights, trust, name-space administration, upload/download rights, trust, name-space administration,

rules.rules.

● Application Application integrationintegration● like “open” and “save” menu entrieslike “open” and “save” menu entries

overview

● Once installed, CPAN6 let you start a new Once installed, CPAN6 let you start a new archivearchive as as simple as creating a directory.simple as creating a directory.

● The archives can be The archives can be local or remotelocal or remote; no difference.; no difference.

● Each archive has Each archive has rulesrules, which control the behavior., which control the behavior.

● Rule set from very strict to very forgiving.Rule set from very strict to very forgiving.

identities

● CreatorCreatorss: people (using a VCS?) create files which belong together: : people (using a VCS?) create files which belong together: distributed as end-product, a distributed as end-product, a releaserelease

● PublisherPublisher: someone uploads the release to the archive: someone uploads the release to the archive

● A release as a A release as a project project name (name (name-space location) and name-space location) and version version labellabel, together unique, both unicode, together unique, both unicode

● AuthorAuthorss own the project (name), and have to give their consent to own the project (name), and have to give their consent to the releasesthe releases

● Finally, the Finally, the UserUserss download the data download the data

archiver

● the rules of an archive are set by the the rules of an archive are set by the BoardBoard;;a group of peoplea group of people..

● releases are uploaded to the releases are uploaded to the CommissionerCommissioner (daemon) which (daemon) which checks the ruleschecks the rules

● releases are mirrored by releases are mirrored by DeployerDeployers, which can be searched by s, which can be searched by users, and then used for downloadusers, and then used for download

● uploading, downloading, and (smart) mirroring is performed by uploading, downloading, and (smart) mirroring is performed by ScribeScribe processes processes

scribes

● Scribes organize the best way of transport:Scribes organize the best way of transport:● compression: gzip, bzip2, ..., nonecompression: gzip, bzip2, ..., none● packaging: zip, tar, rpm, ..., nonepackaging: zip, tar, rpm, ..., none● protocol: HTTP, FTP, e-mail, CD/DVD, ...protocol: HTTP, FTP, e-mail, CD/DVD, ...

no new protocols no new protocols no fixed discissionsno fixed discissions

scribes

● Scribes organize the best way of transport:Scribes organize the best way of transport:● compression: gzip, bzip2, nonecompression: gzip, bzip2, none● packaging: zip, tar, rpm, nonepackaging: zip, tar, rpm, none● protocol: HTTP, FTP, e-mail, CD/DVDprotocol: HTTP, FTP, e-mail, CD/DVD

● Scribes can do filteringScribes can do filtering● only the last releases from a source archiveonly the last releases from a source archive● only certain mime-types or file-sizesonly certain mime-types or file-sizes● exclude the alphas and beta releasesexclude the alphas and beta releases

building complex hierarchies building complex hierarchies

release states

● Any release has a state:Any release has a state:● uploadinguploading● publishedpublished● embargoembargo● releasedreleased● deprecateddeprecated● expiredexpired● rejectedrejected

● Release state change require signaturesRelease state change require signatures

release types

● What can be put in files, can be released into an What can be put in files, can be released into an archive:archive:● publications (of course)publications (of course)● identities; PGP public keys, ...identities; PGP public keys, ...● references to related archivesreferences to related archives

they all follow the same rules they all follow the same rules

release types

● What can be put in files, can be released into an What can be put in files, can be released into an archive:archive:● publications (of course)publications (of course)● identities; PGP public keys, ...identities; PGP public keys, ...● references to related archivesreferences to related archives

● archive configuration; the archive configuration; the ConstitutionConstitution, as written by the , as written by the boardboard

● storage configuration, for storage configuration, for StoresStores

they all follow the same rules they all follow the same rules

project / release relations

target

● Improving my data organizationImproving my data organization

target

● Improving Improving my datamy data organization organization

target

● Improving Improving my wifesmy wifes data organization data organization

● if it works for her, it works for everyoneif it works for her, it works for everyone

My Photos

My Photos

Our Photos

use-cases

● Software distribution; like CPANSoftware distribution; like CPAN

● FTP-server replacementFTP-server replacementor added security, search, and maintenanceor added security, search, and maintenance

● Paper trails through the companyPaper trails through the company

● Developing and distributing Linux distributionsDeveloping and distributing Linux distributions

status

● > 120 pages detailed (XML Schema) design> 120 pages detailed (XML Schema) design

● Start with implementation in Perl5Start with implementation in Perl5● error handling/translation frameworkerror handling/translation framework● compliant XML schema processorcompliant XML schema processor● CPAN6 pluggable componentsCPAN6 pluggable components● jQuery graphical interfacejQuery graphical interface

● PromotionPromotion● I need help!I need help!● http://cpan6.orghttp://cpan6.org

short term needs

● Schema design proofreadersSchema design proofreaders

● Trust / security algorithmsTrust / security algorithms

● License text archiveLicense text archive

● Developer support in time or moneyDeveloper support in time or money

● Your ideas...Your ideas...

progress, design

● Requirements and Requirements and terminology with Sam terminology with Sam VilainVilain

● Open architecture Open architecture and functionalityand functionality

● Data design quite Data design quite detaileddetailed

● Websites, mailinglistWebsites, mailinglist

too many changes?

● Install a perl distribution using CPAN6:Install a perl distribution using CPAN6: $ perl -M$ perl -MCPAN6CPAN6 -e 'install DBIx::Simple' -e 'install DBIx::Simple' CPAN: Storable loaded ok CPAN: Storable loaded ok Going to read /root/.cpan/Metadata Going to read /root/.cpan/Metadata Database was generated on Sun, 19 Aug 2007 19:38:06 Database was generated on Sun, 19 Aug 2007 19:38:06 Running install for module DBIx::Simple Running install for module DBIx::Simple Running make for J/JU/JUERD/DBIx-Simple-1.31.tar.gz Running make for J/JU/JUERD/DBIx-Simple-1.31.tar.gz CPAN: LWP::UserAgent loaded ok CPAN: LWP::UserAgent loaded ok Fetching with LWP: Fetching with LWP: failed failed Fetching with CPAN6: Fetching with CPAN6: pause6 get perl5.perl.org DBIx-Simple 1.31 pause6 get perl5.perl.org DBIx-Simple 1.31 tar xzvf DBIx-Simple-1.31.tar.gz tar xzvf DBIx-Simple-1.31.tar.gz DBIx-Simple-1.31/ DBIx-Simple-1.31/ DBIx-Simple-1.31/lib/ DBIx-Simple-1.31/lib/ DBIx-Simple-1.31/lib/DBIx/ DBIx-Simple-1.31/lib/DBIx/ ... ...

CPAN6 standard services

● Simple data selection and distributionSimple data selection and distribution● no download of package list neededno download of package list needed● no packaging into tar.gz neededno packaging into tar.gz needed

● Smart dependenciesSmart dependencies● dependency `prediction'dependency `prediction'● parallel installation of modules (for Perl6)parallel installation of modules (for Perl6)● dependency classes: run, test, install, optionaldependency classes: run, test, install, optional

● Clear and strict rulesClear and strict rules● GDBs become board, with transparent rulesGDBs become board, with transparent rules

graphical interface

● ““Web2.0”: Web2.0”: JavaScript & JavaScript & AJAXAJAX

● resembles resembles Open/Save with Open/Save with optional meta-optional meta-datadata

● talks to local talks to local daemondaemon

challenge

● All Pause6 features are implemented the same way, All Pause6 features are implemented the same way, but the user will not understand this: translate it into but the user will not understand this: translate it into user space.user space.

Example, equivalent areExample, equivalent are● use a new public keyuse a new public key● opload a set of holiday photosopload a set of holiday photos● start a new archivestart a new archive● change the rules of an archivechange the rules of an archive● initiate a mirrorinitiate a mirror● submit a change request formsubmit a change request form

progress, general purpose

● Building general purpose modules first:Building general purpose modules first:

● OODocOODoc● generating heterogenous sets of manual pages

● Log::ReportLog::Report● exception handling / error reporting● translations

● XML::CompileXML::Compile● interpreting data definitions

● Implementing “Stores”Implementing “Stores”● archive, project, and version names are all UTF-8 of archive, project, and version names are all UTF-8 of

unlimited length (restrictions possible)unlimited length (restrictions possible)● plain (mutulated) projectionplain (mutulated) projection● encoded (%2E) projectionencoded (%2E) projection

● mapped projection (name -> something)mapped projection (name -> something)● Implementing “Identities”Implementing “Identities”

● (UNIX) username(UNIX) username● PauseIDPauseID● public-keypublic-key

progress in core

soon

● Log::Report::ViewsLog::Report::Views● combine translated (error) messages with tooltips and combine translated (error) messages with tooltips and

detailed documentation.detailed documentation.

● Frameworks to mergeFrameworks to merge● interface componentsinterface components● jQuery needsjQuery needs● ......

● LicensesLicenses● filling a license databasefilling a license database

whishes

● Perl(5) installation tools are (mostly) outside the scope Perl(5) installation tools are (mostly) outside the scope of CPAN6, but could we please:of CPAN6, but could we please:

● install multiple module versions in parallelinstall multiple module versions in parallel

● first download all dependencies, and test them together, first download all dependencies, and test them together, before installing thembefore installing them

● be much less verbose in the process; we frighten people off!be much less verbose in the process; we frighten people off!