Large drupal site builds a workshop for sxsw interactive - march 17, 2015

  • Published on

  • View

  • Download

Embed Size (px)


<ol><li> 1. Large Drupal Site Builds SXSW Interactive, 2015 </li><li> 2. Who the Heck are We? Rob Ristroph Technical Architect at Acquia, a digital experience company based in Boston, MA and with remote team members worldwide. Based in Austin, TX! ATX Hacker Space Member Drupal/Open Source contributor </li><li> 3. Who the Heck are We? Jenn Sramek Sr. Engagement Manager at Acquia, based in the San Francisco Bay Area Manage large-scale Drupal projects for organizations and companies that are often new to Drupal Formerly Dir. of Operations at CivicActions Board President for the Haight Ashbury Food Program </li><li> 4. Who the Heck are We? Alex Urevick- Ackelsberg Co-Founder/CEO of Zivtech Extensive large Drupal project experience Wears many hats Nick Lewis Site Architect at Zivtech Extensive Drupal site architecture experience Sock Puppet Master </li><li> 5. Who the Heck are You? Drupal experience? Large-scale web? Open Source? Have an active/current large-scale project? Planning one? Waterfall/Agile? Local/from other states/countries? Anything else interesting? </li><li> 6. Logistics Connectivity Power/Wireless Biology Restrooms Food/Drink Overall Schedule 9:30 am - 1:30 pm Shared Notes and Resources https://pad.lullabot. com/p/Large_Drupal_Sites_Workshop_SXSW_2015 </li><li> 7. Topic Overview Project Management - whats different about big projects ? Technical: Development workflow Custom APIs and the ecosystem of API consumers Scaling and performance at the enterprise level - configuring cache, finding bottlenecks Large-scale data migration Monitoring Social: Open Source and corporate culture Training Larger development team dynamics - culture of quality Multiple teams and vendors Tell us what to focus on </li><li> 8. Defining Large Projects SXSW Interactive, 2015 </li><li> 9. What is Large? Customer has a lot of money ? Site has a lot of traffic or data ? Project build runs for a long time ? Whatever is pushing past your comfort / experience zone in some way. </li><li> 10. What is Large More than one website Platform and distribution builds Another team uses your platform or distribution Multiple stakeholders Company vs. Divisions or Franchisees Technical vs. Marketing departments APIs and System interactions mail delivery SSO authentication, including roles and other metadata Web metrics Workflows, editorial processes Significant migration of data </li><li> 11. What is Large Content Staging Workflow on steroids / between servers Inflexible legal requirements Solutions Content staging server and clone Site Preview System (SPS) Deploy suite of modules Challenging, Interesting and Fun Inter-disciplinary development Complex feature development Cutting-edge interaction design </li><li> 12. What is Large The most common single feature of a large project is the interaction with more team members and more stakeholders. </li><li> 13. Working with Large Teams Working with large teams is the most likely source of project failure... If you are new to big projects, interacting with and managing big teams is probably the most unfamiliar area and the greatest area of risk Even large organizations can fail at large project implementation repeatedly All the same technical issues that show up in large projects can show up in smaller projects, but it is harder to maintain clarity, quality code, expectations, and scope when dealing with large teams </li><li> 14. Working with Large Teams You start with a client / customer Add a design / strategy vendor Add another developer team on the customer side Add other outside developer team(s), with sub-contractor relationship Add multiple other vendors, with no sub-contractor relationships Separate projects with out-of-sync timelines All still working on the same project with ultimately the same goal for success... </li><li> 15. Working with Large Teams Timeline differences Scope differences or gaps Resource gaps Process differences Deliverables differences or gaps that may be significant Coordinating vacations/time off Internal team mentorship, developer ladders, or other process for fully on-boarding new team members (including client) </li><li> 16. Working with Large Teams Multiple parties make DevOps essential Practices that just provide efficiency at the scale of one team become absolutely necessary: Pull request workflows Peer code review Standardized development environments Automated testing Tech demo ? </li><li> 17. Working with Large Teams The most rewarding parts of big projects Personal growth and learning Exposure to other teams is like having several jobs without switching bosses Part of a larger community </li><li> 18. Getting Large Projects Pick the projects, dont let them pick you Initiate contact with particular clients and projects that interest you Talk with your whole team when going after something large Long term projects should be something people find fulfilling Winning by the lowest bid is a red flag Spend much more time vetting/ do formal Discovery Be picky; build padding into quotes </li><li> 19. Picking a Vendor Is the vendor interested enough to be committed to a long project ? Unrealistically low bids are a red flag Budget for discovery, and pay for it If the vendor has not done projects of your size before: Allocate more to planning Smallest MVP possible A partnered bid may mitigate some risk If you fall behind, cut scope instead of adding resources </li><li> 20. Overall Process and Project Management SXSW Interactive, 2015 </li><li> 21. Project (Program) Management Overall... The basics become essential JIRA or another tool Discipline in using that tool Communications and Report Generation is key, and may involve X-facing veation Reporting is part of the effort of a Project or Program Manager More project managers on different teams, coordination Often multiple JIRA instances and trackers in play On big projects, project managers can really shine and make an obvious difference, and technical people who hate it are more likely to be able to avoid it (not necessarily a good idea ?) </li><li> 22. Discovery for Large Scale Drupal Curiosity is a Skill... </li><li> 23. Discovery for Large Scale Drupal Discovery is really a very long list of questions to answer before you start development: - Create shared understanding of the work to be done and its priority, and start technical requirements docs - Align everyone on the process that will be used for development, quality assurance, signoff/acceptance - Surface any risks or potential issues that will require mitigation planning - Document any schedule or resource considerations - Right-size the effort of development </li><li> 24. Discovery for Large Scale Drupal Technical Requirements: - Content types and fields - User roles/permissions - Content and site management workflows - Integrated service requirements (CDN, analytics, single-sign-on, Translation Management Services, payment processing, feeds,) - Migration details (size and complexity of database, content type/field mapping, consistency of data) Non-Technical Requirements: - Resources including third-parties, dev team(s), schedules, dependencies, internal technical resources, training needs - Internal timelines and schedule constraints (vacations, major internal events, conferences, product releases) - Security/Governance workflow and approval process (incl. clearance for those with data access and secure systems for data transmission and storage) - Access to associated systems and tools </li><li> 25. Planning for Large Scale Drupal - Timeline - Dependencies - Risks and Mitigation Planning - Resources (considering attrition of internal and external teams) - Technical/Architectural planning (including POCs/rapid prototypes) - Code management - Security management - Plan for addressing technical debt - Governance (assignment and acceptance) - Maintenance plan, development cycles, post-launch - Pre-launch/Post-launch planning </li><li> 26. Planning for Large Scale Drupal Involves LOTS of technical-specific planning (by the architect). A site architect may do as much technical planning as the PM does project planning (code management, maintenance, release planning, etc). Documenting risks protects everyone, and the project Interim maintenance and remediation Documentation needs may differ for large projects </li><li> 27. Now that all the pieces are in place... Waterfall? OR Agile? </li><li> 28. Working with the MVP </li><li> 29. Tools for Large Scale Development SXSW Interactive, 2015 </li><li> 30. Tools for Large Scale Development Plan for tools to scale - know what happens to them when the project is over Ticketing system Document repository Code Management tools UX/Design Management tools Communications Channel(s) Training tools </li><li> 31. Tools for Large Scale Development JIRA/Rally/others - plan for tools at scale and know what happens to them when the project is over. Code management tools - Github, BitBucket (more integrated with JIRA for one PR per JIRA ticket) Process more than product Onboarding dev documentation. UX/Design management tools - differ by firm, many use BaseCamp Living document for coding/best practices/standard practices </li><li> 32. Development Managing multi-team interactions Deployment tools and procedures will take up a lot of time Delays between code and UAT Deployment and DevOps can have huge (%30 ?) impacts on efficiency Re-work is the biggest cost </li><li> 33. Scalable Code Management SXSW Interactive, 2015 </li><li> 34. Iterative Development Developers cannot JUST develop, have to document how they developed. There is a much larger team depending on clarity. Can effect technology choices Iterative can differ at the enterprise level There can be more than one effective Agile team Include maintenance rounds in iteration Plan for longer lags between dev and signoff Get sprint signoff outside of JIRA or other ticket system </li><li> 35. Performance Planning Plan for addressing performance issues early Collect realistic requirements with context Benchmark legacy site for reference Test early How do you review contributed modules/code for performance? Not intuition - always check What tools to use to benchmark? Make sure they match your expected traffic How do you populate the site with enough content to test? If benchmarking can be automated, just repeat as project proceeds </li><li> 36. Scaling and Performance Scaling often matters more than Performance Editorial interactions, cache updating are more important Collect performance, scaling, time-to-refresh requirements just like any other, and test for them Optimizing Drupal for performance is a specific skill that requires Drupal experience But if you can automatically test it can just like any other debuggin Hard to designate the performance issues to one person or team; again interactions between teams are essential </li><li> 37. Scaling and Performance Scaling may mean more small sites not one big site See multisite debates Enforcing Code quality and standards Peer Reviews Views Cache Bully and similar Automated testing Automated code sniffer Automated testing is another thing you may not get to work on except on large projects. </li><li> 38. Deployment For each deployment of code, it will be important to have a consistent and documented process: Preparing for Deployment: What do you need to do to get ready? Includes reviewing all merged pull requests, updating release notes, and preparing and validating any needed deployment steps. QA Steps: What do you need to do to prepare for QA? Includes tagging the release, cloning UAT database to QA environment, smoke and unit tests. Acceptance: What do you need to do to gain client acceptance? Includes preparing a demo, regression testing, and preparing the client for UAT testing. </li><li> 39. Large-scale data migration and content management SXSW Interactive, 2015 </li><li> 40. Data Migration Review and get agreement on the process Consider data security Clean the data Do not scope it until you have mapped it Make sure you know what is all is! Dump and Map it! (content type by content type, field by field) </li><li> 41. Data Migration Re-Scoping migration after mapping Incremental migration Differential migration Special considerations for beta sites (where there may be two versions of a live site) Plan for deprecated systems Plan for wrapping up with manual migration (copy/paste or rewrite/edit) </li><li> 42. Migrate Module A complex migration can be a large portion of development effort Plan for continuous migration Run a scheduled migration until canonical DB switchover Iterate like any other development Make sure client can devote resources to checking content throughout the process </li><li> 43. Content Staging SXSW Interactive, 2015 </li><li> 44. Content Staging Often discussed as part of Discovery, because many enterprise companies have structures that utilize a content staging environment. With Drupal, this is most often not needed or can be fairly complex. Content staging, if needed, may be one of the interesting problems to solve that inspires your team to take on the project. </li><li> 45. Content Staging Draft / published Workflows Workflows with SPS - complexities multiply Staging server, simple DB copy - limiting Staging server, with workflow, and deploy module If editorial staff get involved late, this is often a late-in-project scope explosion. </li><li> 46. Training SXSW Interactive, 2015 </li><li> 47. Site Owner / Builder Training Talk about it during Discovery!!! Training needs to be built into the schedule Project Managers can become de-facto trainers Include teaching clients to do UAT (as requisite for end- of-project training) Leverage available resources (,,, BuildAModule and others) for training Plan to train the trainer </li><li> 48. Training Model Start with defined needs Decide on the format of training (in-person, training materials on separate site or tool, formal training) Quantify trainees and role-specific tasks required of them in order to plan and estimate Plan to train at least some roles incrementally (often some training will be needed for UAT ) Leverage QA and UAT testing steps for training materials </li><li> 49. Handoff and Transitions SXSW Interactive, 2015 </li><li> 50. End Goals and Transition Big projects will involve transitioning in some way: To other vendors To internal teams From active development to maintenance mode Re-acquiring the project later (in another phase) Documentation and consistent development practices are essential </li><li> 51. End Goals and Transition What happens next? Discuss at the beginning of the project what the end of it looks like. Maintainability - avoid risk to support obligations Technical debt - document it Who will manage code after the final launch? Process and tools changes? </li><li> 52. SXSW Interactive, 2015 Giving Back to the Open Source Community </li><li> 53. Takin it to the Community Contribution makes Drupal go! Discuss the value of and plans for contribution during Discovery (even if y...</li></ol>