Xavier Galvez Advanced Micro Devices, Inc.
© 2011
Abstract Granting access and creating accounts on the Perforce depots at AMD is a 24/7 operation with engineers working around the world. Given the highly-‐sensitive nature of the source code that resides on these depots, access to areas is tightly regulated and may s whim. The administrators that create accounts and grant access to these areas should be enabled to perform these tasks wherever they are, and they may not necessarily have access to a desktop or have the means to log in to the company network remotely. By creating an interface accessible by web browser with a predefined list of actions, our administrators are able to perform these tasks securely by mobile device.
Background In order to fully appreciate the solution in place, some context must be given on the problem. The discussion touches on the structure of the permissions in the protection table, the centralized authorization process, and the breadth of the organization. The evolution of the solution is then discussed, and its simplicity may be a boon to other organizations with limited resources. A basic level of Perforce administration may be required to understand the terms used, but most of the workflow will be appreciated by an ardent reader.
Access Permissions Source code is version-‐controlled on a Perforce server, and access to the source is dictated by a set of
(list), view their contents (read) or modify the files (write). Permissions on this particular Perforce server are set up using one level of groups and one level of subgroups fo groupsabstraction levels also make it easier to visualize how users are given access to areas of the depot. Permissions are not given directly to the users, i.e., lines in the protection table do not explicitly specify a user account. This is done to align with best practices.1 The protection table specifies mostly access groups; to minimize subgroups, occasional exceptions are granted to some teams. This is further illustrated in the following figures on the next page.
1 2007. <http://www.perforce.com/perforce/conferences/us/2007/presentations/jcreasy_p4admin_2007_paper.pdf>
2
read group ACC_rel * //depot/rel/... write group bm_rel * //depot/rel/... write group ACC_rel * //depot/rel/8.10/... (added by automation as release is branched) read group ACC_rel * -//depot/rel/*/access.txt write group REG_rel * //depot/rel/*/access.txt read group ACC_main * //depot/main/... read group ACC_mercury * //depot/main/mercury/... write group bm_mercury * //depot/main/mercury/... (main/mercury = promoted stg/mercury/src) write group ACC_mercury * //depot/stg/mercury/src/... write group ACC_mercury * //depot/stg/mercury/doc/... write group nda_mercury * //depot/stg/mercury/nda/... write group reg_mercury * //depot/stg/mercury/access.txt read group ACC_venus * //depot/main/venus/... write group bm_venus * //depot/main/venus/... (main/venus = promoted stg/venus/src) write group ACC_venus * //depot/stg/venus/src/... write group ACC_venus * //depot/stg/venus/doc/... write group nda_venus * //depot/stg/venus/nda/... write group reg_venus * //depot/stg/venus/access.txt super group grp_admins * //...
Figure 1. Protection table.
The protection table in Figure 1 above refers to , each of which is composed of subgroups respectively as seen in Figure 2: Access Group Description Subgroups ACC_rel Subgroups that have access to release branches. grp_northamerica, grp_israel, grp_china,
grp_releasemanagement REG_rel Subgroups that regulate write access into a release branch via Work ID. grp_releasemanagement ACC_main Subgroups that have read access to mainline. grp_northamerica, grp_israel, grp_china ACC_mercury Subgroups allowed to work on the Mercury component. grp_northamerica ACC_venus Subgroups allowed to work on the Venus component. grp_northamerica, grp_israel, grp_china
Figure 2. Access groups.
The protection table in Figure 1 -‐ in Figure 3: High-‐Level Team Description bm_rel Users that promote from mainline to release; normally just automation. bm_mercury Users that promote from stg/mercury to main; normally team leads or build managers of Mercury. bm_venus Users that promote from stg/venus to main; normally team leads or build managers of Venus. reg_mercury Users that regulate write access into stg/mercury via work ID. reg_venus Users that regulate write access into stg/venus via Work ID. nda_mercury Privileged users with nda/mercury access. nda_venus Privileged users with nda/venus access.
Figure 3. High-‐level teams.
The access groups in Figure 2 refer to the -‐ in Figure 4 below: Low-‐Level Team Description grp_releasemanagement The release management team. grp_northamerica Developers in North America. grp_israel Developers in Israel. grp_china Developers in China. grp_admins Perforce administrators.
Figure 4. Low-‐level teams.
3
Additional notes regarding the permission structure:
A user will always be a member of a low-‐level team (i.e., one of the grp_ groups). Users in high-‐level teams (bm_, nda_, reg_) also belong to a grp_ group. (Exception: bm_rel) The access.txt file is an additional mechanism that works with the submit trigger; this controls
check-‐ins based on a Work ID found in the changelist description. Access groups and teams can have designated owners that may grant existing users access to the associated area, without having administrative (i.e., superuser) rights. When a Perforce account is created, the user is assigned to one of the low-‐level teams by default. This enables the user to have access to designated areas off the bat. Additional access can then be granted by owners of the high-‐level teams on request, or superusers can grant these privileges. And here lies the crux of the problem we would like to solve.
Challenges AMD does not have one team handling administrative duties for all Perforce servers, but rather the company has that exist throughout the organization. Multiple departments have their own cluster of servers and depots that are maintained by their own respective administrative team. This decentralization enables each admin team to be experts of the process, depot structure and file contents of that specific department. With that, each department has specific rules on how content residing on their specific repository is accessed. Some departments are smaller than others; notably, these smaller departments have teams that reside in the same time zone. Given this scenario, the superusers or group owners in these departments can grant access rights to a requestor during regular working hours. Also, a properly resourced admin team, or a team using a properly structured superuser + group owner model in granting privileges, is able to fulfill such requests within a reasonable time. However, other departments may not have this luxury. One department at AMD in particular has teams covering almost all time zones making creating accounts and granting access a 24/7 operation. Another restriction in this department is its strict enforcement on code exposure; given the sensitivity of the content, only the administrative team is allowed to grant access. Alas, the administrative team with the depth of knowledge of the file hierarchical structure and the access to export control laws for properly discerning permissions exists in only one geographical area, and as such, authorization approval is centralized. Lastly, the admin team is not fully resourced as a support center, but is also tasked with other ongoing projects and meetings that make responding to requests somewhat difficult.
Creating User Accounts Creating a Perforce user account in this department started with calling the traditional Perforce command p4 user -i (and filling in the appropriate fields) from the command-‐line. To make user creation a less cumbersome task, a script was written that encapsulated this command; the script simply required a list of parameters (such as an email address and groups the user belonged to), generated a
4
strong, random password, and then sent an email to the user with all the information needed to access the Perforce depot. The sequence of events was where:
1) The user requested access to the Perforce depot by email. 2) The Perforce admin received the email, validated the request, and called the script
newuser.exe to create the account. 3) The script sent an email to the user with the information required to access the depot.
Eventually, the script was extended so that prior to the creation of the account, contractors and users with special access rights were sent an email with the terms and conditions for accessing the files in the depot. Once the terms were accepted (by clicking on a link in the email), the account was activated and the user received the access information. A list of the parameters required by this script newuser.exe can be seen in Figure 5. Calling the script from the command-‐line without any parameters described the usage syntax and inputs. > newuser.exe newuser 1.8.5 Usage: newuser -user Username -fname FullName -email Email -site Site -ports Server:Port1[,Server:Port2] -groups Group1[,Group2] [-outrule] Username - The user's login. (required) FullName - The full name of the user. (required) Email - The user's AMD email address. (required) Site - The site the user is at (e.g., TO). Server:Port - The server and port for access. (e.g., perforce:1666) Group - The groups the user needs to be added to. Outrule - Send email with terms and conditions of the account.
Figure 5. Parameters for script newuser.exe.
The script made the task of creating users much easier, but it had a couple shortcomings that are inherent in any published binary:
Newer versions of the script to address bugs or add new features had to be recompiled and packaged each time.
Diligence in deployment was necessary to ensure that all users in the admin team had the latest and greatest version.
This was not a serious problem, but was simply an annoyance that was easily rectified by a reminder email, a scheduled script (i.e., cron job) that the latest version of the binarydiligence to do this manually. However, this scenario provided an opportunity to devise a web-‐based solution instead. The team had its own web server on the intranet that was used to publish documents that were checked into the Perforce depot. This mechanism allowed users to examine files that were deemed safe for general consumption and did not require a Perforce account to access. Leveraging this web server, a secure area was created specifically for the Perforce admins.
5
Within this secure area, the newuser.exe binary was converted to a CGI script accessible only to the admins. The web-‐based model addressed the limitations of using the published binary model, and even
easily implemented before:
Newer versions of the script were available the moment these were uploaded to the server. Deployment was instantaneous, as admins had access to the latest version immediately.
Experimental versions of the script co-‐existed with production versions and could easily be tested by residing in relatively the same location.
Validation of the inputs was now possible, as the web interface was able to check for spelling mistakes against a live database (that was updated with employee info) or against the Perforce server itself (for existing users or groups).
The script was now OS-‐independent, as a web browser was simply needed to access the script. A simple web interface was sufficient to provide the functionality that the previous binary offered. Its simplicity also meant quick-‐loading times (which turned out to be a boon for a serendipitous discovery that followed shortly, to be discussed). Not much time was invested to get this running, as the objective was to have a functional replacement in place. A screen cap of the web interface can be seen in Figure 6.
Figure 6. Screen cap of the web interface.
6
The parameters required by the original binary newuser.exe can now be input more intuitively through the new web browser interface. The CGI script was written in Perl to leverage the source of the original binary (which was also written in Perl, although the internal mechanism has since been re-‐written). The Perl script generated the HTML files on the fly, and with the CSS files, comprised the web page. The full name, login, email address, groups and ports were still required parameters, but checkboxes and validation checks made the interface less error-‐prone. After the administrator filled in the required parameters, the script then processed the inputs, generated a strong password, and then sent this off to the user via email. The script called an instance of the command-‐line client p4.exe that is installed on the server to perform the task; other implementations of this model may call APIs such as P4Perl or P4Python as well. The administrator, however, still needed to be on the AMD domain in order to access the web server. The admin would need to be in front of a desktop computer or laptop while in the office. If the admin was not within the vicinity, accessing the domain remotely via VPN (i.e., virtual private network) was required. But was there a way to untether the admin?
Wireless Solutions AMD runs a BlackBerry Enterprise Server behind a firewall; one of the modules included in a BlackBerry Enterprise Server is the BlackBerry MDS Connection Service. This mechanism allows wireless applications on BlackBerry devices to connect to tservers or web servers, and to do so securely.2 Perforce admins at AMD are equipped with Blackberry devices. These were originally issued for admins to monitor the health of the Perforce servers, where administrators were quickly alerted of any anomalies that would occur via automated notifications. As this resource was already available, no other infrastructure was required for the Perforce admin to access the web server hosting the user creation script. By keeping the HTML simple, the web page can be quickly loaded into the BlackBerry devices. The web page works wonderfully even on the antiquated browser found in the BlackBerry 8700r, as seen in Figure 7. With this discovery, the Perforce admins are able to conduct administrative tasks on their mobile devices.
2 BlackBerry MDS Connection Service <http://docs.blackberry.com/en/admin/deliverables/7335/BB_MDS_Connection_Service_267709_11.jsp>
7
Figure 7. A BlackBerry 8700r accessing the web page. Another wireless option available to Perforce admins would be accessing the web page on an iOS device such as an iPhone or iPad. Users equipped with one of these are able to do so by setting up a VPN on their iOS device. This allows the administrator to communicate private information securely over a public network, but only if the VPN server hosted by the organization supports one of the protocols and authentication methods that works with iOS.3 The Perforce admin is able to use a variety of iOS devices to access this web page. Figure 8 is a screen cap of how this page looks when accessed from an iPad.
3 iOS: Supported protocols for VPN <http://support.apple.com/kb/HT1288>
8
Figure 8. Screen cap of the web page on an iPad.
Security and Risks Security over wireless devices is achieved via secure VPN for users accessing the web page by iOS device, and through the different mechanisms put in place by the BlackBerry Enterprise Solution (e.g., strong data and encryption key protection).4 HTTP (Hypertext Transfer Protocol) is not secure and can be exploited for man-‐in-‐the-‐middle and eavesdropping attacks. The web server runs in HTTPS, which is a combination of ordinary HTTP over SSL/TTL (encrypted Secure Sockets Layer or Transport Layer Security connection). This mechanism encrypts the HTTP message prior to transmission and decrypts the message upon arrival, which adds a layer of security when the web page is accessed by browser. The web page itself is not published to the internal development community within the company, and is only known to Perforce admins. The script that generates the HTML itself also does a check to ensure that only the authorized Perforce admins are allowed to access the URL to the web page.
4 BlackBerry Technical Overview <http://docs.blackberry.com/en/admin/deliverables/7225/FTO_X_Gold.pdf>
9
If a malicious non-‐administrator is able to access the web page, then the risks can still be minimized by having a limited list of actions that can be performed. This prevents any extended damage that can be done. Currently, the production version of the web page provides the following functionality:
Create a new user, given inputs such as full name, email address and groups. Displa s permissions (using p4 protects and p4 groups). Add an existing user to currently existing groups to extend access. Remove the user from a group to limit or prevent access. Change the password of an existing user.
The implications of being able to provide an abstraction to the Perforce commands but made accessible via web browser, and then through mobile devices, are endless. Extended features can be added for additional administrative tasks, such as checking the status of the Perforce servers and viewing submitted changelists and jobs. However, these should be deliberated carefully, as not all commands may be suited for a web interface. Also, limited actions lower the risk of damage by a non-‐admin.
Other Solutions The P4Web5 application is more suited for the developer that uses Perforce, and not for the admin that wishes to perform administrative tasks through a web browser. The interface of P4Web may also be a little bit too content-‐intensive for a mobile device. The P4Scout6 application runs natively on Android and iOS devices, and allows the administrator to check the status of the servers. P4Scout addresses a different need altogether.
Conclusion The solution presented is actually a very simple mechanism. It took advantage of existing resources that the Perforce team had at its disposal, and the latest web programming technologies were not necessary to get it up and running. This resulted in quick development and fast deployment. Having the mechanism run in a web page + mobile device model allows a much richer feature set, as the only limitation is the discretion of the Perforce admin team on the administrative tasks that they choose to support. New capabilities can be easily added and deployed. B
Compiling for different platforms is not needed, and the administrator will feel at home accessing the web page on a desktop as accessing this from a mobile device since the interface will remain relatively the same.
being tethered to a desktop or laptop, freeing up the most valuable resource anyone could ask for: time.
5 P4Web Release 2010.1 User Guide <http://www.perforce.com/perforce/r10.1/manuals/p4web/help/index.html> 6 http://blog.perforce.com/blog/?p=1047>