Broadband Geoprocessing Guide · State of Colorado Office of Information Technology Colorado Broadband Data and Development Program Page 6 of 99 10/10/2017 needs but include programs

Colorado Broadband Data and Development Program

Broadband Geoprocessing Guide

TRAINING GUIDE

Document Owners

Prepared by GIS Analyst: Kass Rezagholi

GIS Analyst: Tony Filipiak

Department Governor’s Office of Information Technology

Document Status

Project Number 001

Date Created January 27, 2012

Filename P:\SBDDGP-ARRA Project\BroadbandProductionArea\Documents\Team

Documents\Data_Processing_Guide.docx

Status Working Document

Last change author Kass Rezagholi on 10/10/2017 10:59:15 AM

Last change date October 10, 2017

Last change

description

Update for ArcGIS 10.3; current data processing, and training

State of Colorado

Office of Information Technology

Colorado Broadband Data and Development Program Page 2 of 99 10/10/2017

BROADBAND GEOPROCESSING GUIDE ................................................................. 1

1. OVERVIEW ....................................................................................................... 5

1.1. INTRODUCTION ........................................................................................................... 5 1.1.1. National Telecommunications Information Administration ......................................................... 5

1.1.1.1. The National Broadband Map ........................................................................................... 5 1.1.2. Colorado Broadband Data and Development Program .............................................................. 6

1.2. OBJECTIVE ................................................................................................................. 7 1.2.1. Helpful Links ....................................................................................................................... 7

1.3. DATA PROCESSING FLOW CHART .................................................................................... 8

2. GENERAL INFORMATION .................................................................................. 9

2.1. RESOURCE DOCUMENTS ................................................................................................ 9 2.1.1. Microsoft Access Database .................................................................................................... 9 2.1.2. Providers and Stats for monthYEAR ....................................................................................... 9 2.1.3. SBDD_TRANSFER_monthYEAR .............................................................................................. 9

2.2. WORKSPACE ............................................................................................................ 10 2.2.1. Broadband Production Area .................................................................................................. 10 2.2.2. Base Data .......................................................................................................................... 10

2.2.2.1. Publice Land Survey System – Quarter Quarters ............................................................... 10 2.2.2.2. CENSUS BLOCKS (Discontinued methods…) ..................................................................... 10 2.2.2.3. TIGER ROADS (Discontinued methods…) ......................................................................... 10 2.2.2.4. ADDRESS LOCATORS .................................................................................................... 11 2.2.2.5. Additional Base Data ..................................................................................................... 11

2.2.1. Personal Workspace ............................................................................................................ 12 2.2.1.1. Workspace Setup .......................................................................................................... 12

2.2.2. Staging Area ...................................................................................................................... 14 2.2.3. OIT Final Format ................................................................................................................ 14

2.3. RAW DATA ............................................................................................................... 15 2.3.1. Purpose and Procedure ........................................................................................................ 15 2.3.2. Raw Data File Types ........................................................................................................... 15

2.3.2.1. Microsoft Word Document (.doc/.docx) ............................................................................ 15 2.3.2.2. Text (.txt).................................................................................................................... 15 2.3.2.3. CAD (Computer Aided Drafting) ...................................................................................... 15 2.3.2.4. KML/KMZ ..................................................................................................................... 15 2.3.2.5. Outlook Item ................................................................................................................ 15 2.3.2.6. Excel Spreadsheet (.xls/.xlsx) ........................................................................................ 15 2.3.2.7. Shapefile (.shp) ............................................................................................................ 16 2.3.2.8. PDF ............................................................................................................................. 16 2.3.2.9. Image File (.jpg, .tif, .gif…etc) ........................................................................................ 16

2.4. BROADBAND PROCESSING TOOLBOX .............................................................................. 16

3. PRE-PROCESSING RAW DATA ......................................................................... 19

3.1. DATA SPECIFICATIONS ............................................................................................... 19 3.1.1. Identifying Data Type.......................................................................................................... 19

3.1.1.1. PLSS Quarter-Quarter Sections ...................................................................................... 19 3.1.1.2. Middle Mile Output ........................................................................................................ 19 3.1.1.3. Wireless Output ............................................................................................................ 19

3.1.2. Transmission Technology (TRANSTECH) ................................................................................ 20 3.1.3. Maximum Advertised Download and Upload Speeds ................................................................ 20

3.2. PRE-PROCESSING STANDARDIZATION ............................................................................ 21 3.2.1. Clean Tabular Data ............................................................................................................. 21

3.2.1.1. Latitude/Longitude ........................................................................................................ 22 3.2.1.2. Subscriber Addresses .................................................................................................... 22 3.2.1.3. Address Ranges ............................................................................................................ 24 3.2.1.4. Census Block ID ........................................................................................................... 24

State of Colorado



3.2.2. Image Raw Data (jpeg, pdf, etc) .......................................................................................... 25 3.2.3. Shapefile ........................................................................................................................... 25

4. PUBLIC LAND SURVEY SYSTEM (PLSS) QUARTER QUARTER SECTIONS .......... 27

4.1. INTRODUCTION ......................................................................................................... 27 4.2. DATA PROCESSING .................................................................................................... 28

4.2.1. dBase Table ....................................................................................................................... 29 4.2.1.1. Subscriber Addresses .................................................................................................... 29 4.2.1.2. Census Block ID’s ......................................................................................................... 29

4.2.2. Georeferenced Image.......................................................................................................... 32 4.2.3. Shapefile ........................................................................................................................... 32 4.2.4. CensusBlock/Roads Tool ...................................................................................................... 34

4.3. TOOL DESCRIPTION FOUND: FUNCTIONING TOOLS 12.1.4: CENSUSROADS.PY PLSS ANALYSIS 34

5. WIRELESS ...................................................................................................... 38


5.2.1. Digital Coverage ................................................................................................................. 39 5.2.1.1. Polygon Coverage ......................................................................................................... 39 5.2.1.2. Subscriber Point Data .................................................................................................... 40 5.2.1.3. Tower Point Data .......................................................................................................... 40

5.2.2. Radio Mobile ...................................................................................................................... 40 5.2.2.1. Introduction to Radio Mobile .......................................................................................... 40 5.2.2.2. Installing Radio Mobile .................................................................................................. 40 5.2.2.3. Processing Provider Data ............................................................................................... 42

5.3. WIRELESS ANALYSIS ................................................................................................. 53

6. MIDDLE MILE ................................................................................................. 54


6.2.1. dBASE Table ...................................................................................................................... 54 6.2.1.1. Address Data ............................................................................................................... 54 6.2.1.2. Latitude and Longitude Data .......................................................................................... 55

6.2.2. Point Shapefile ................................................................................................................... 55 6.2.2.1. Middle Mile Points ......................................................................................................... 55 6.2.2.2. Wireless Tower Points ................................................................................................... 56

6.2.3. Image ............................................................................................................................... 57 6.2.3.1. Georeference ............................................................................................................... 57 6.2.3.2. Digitize ........................................................................................................................ 57 6.2.3.3. Populating Required Fields ............................................................................................. 58 6.2.3.4. Deriving Data from Known Fields .................................................................................... 58

6.2.4. Middle Mile Python Tool ....................................................................................................... 59

7. STAGING AREA ............................................................................................... 59

7.1. STAGING TEMPLATE ................................................................................................... 59 7.2. LOADING DATA INTO TEMPLATE .................................................................................... 60

7.2.1. Calculate Fields .................................................................................................................. 62 7.2.1.1. Census Blocks .............................................................................................................. 62 7.2.1.2. Wireless ...................................................................................................................... 63 7.2.1.3. Middle Mile .................................................................................................................. 63

7.3. STAGING TOOL ......................................................................................................... 63 7.3.1. Lookup Table ..................................................................................................................... 63 7.3.2. Staging Template ............................................................................................................... 64 7.3.3. Staging Tool ...................................................................................................................... 64

8. OIT FINAL FORMAT ........................................................................................ 66

State of Colorado



8.1. DATA SUBMISSION PACKAGE (*.ZIP) ............................................................................ 66 8.2. OIT TEMPLATE (OITFINALTEMPLATE.GDB) .................................................................... 67

8.2.1.1. Load by Hand ............................................................................................................... 67 8.2.1.2. OIT Final Format Tool .................................................................................................... 68

8.3. ERROR CHECK ........................................................................................................... 70 8.3.1. SQL SDE Error check........................................................................................................... 70

9. POST-PROCESSING ........................................................................................ 74

9.1. MAPBOOKS AND VALIDATION ....................................................................................... 74 9.1.1. Mapbooks: Data Driven Pages .............................................................................................. 74

9.2. OIT DATA PACKAGE .................................................................................................. 78 9.2.1. Methodology Document ....................................................................................................... 78 9.2.2. Process Guide Document ..................................................................................................... 78 9.2.3. Additional Documents ......................................................................................................... 78 9.2.4. Warnings and Error Readme.txt ........................................................................................... 78 9.2.5. Compile Final Package ......................................................................................................... 79

9.3. CBDDP .................................................................................................................. 79 9.3.1. Broadband Mapservices ....................................................................................................... 79

10. FUTURE ENHANCEMENT TOOLS .................................................................... 81

10.1. FUNCTIONAL TOOLS ................................................................................................. 81 10.1.1. CalcMaxMinAddr.py(discontinued) ....................................................................................... 81 10.1.2. CalcStreetName.py(discontinued) ....................................................................................... 82 10.1.3. XYCode.py (discontinued) .................................................................................................. 82 10.1.4. CensusRoads.py(discontinued) ........................................................................................... 82 10.1.5. OITFinalFormat.py ............................................................................................................ 83 10.1.6. StagingTool.py ................................................................................................................. 84 10.1.7. SuperRegionPoly.py .......................................................................................................... 84

11. APPENDIX .................................................................................................... 85

11.1. GENERAL GIS SKILLS .............................................................................................. 85 11.1.1. Geocoding (10.0 Instructions) ............................................................................................ 85 11.1.2. Georeferencing ................................................................................................................. 87 11.1.3. Digitizing ......................................................................................................................... 90 11.1.4. Latitude/Longitude Coordinates .......................................................................................... 90 11.1.5. KML/KMZ Conversion ........................................................................................................ 92 11.1.6. CAD ................................................................................................................................ 93

11.1.6.1. CAD Coordinate System ............................................................................................... 93 11.1.6.2. CAD Georeference ....................................................................................................... 94 11.1.6.3. CAD Spatial Adjustment ............................................................................................... 97 11.1.6.4. CAD Data Selection ..................................................................................................... 98

State of Colorado



1. Overview

1.1. Introduction

1.1.1. National Telecommunications Information Administration

The National Telecommunications and Information Administration (NTIA), located within the

Department of Commerce, is the Executive Branch agency that is principally responsible by law for

advising the President on telecommunications and information policy issues. NTIA’s programs and

policymaking focus largely on expanding broadband Internet access and adoption in America,

expanding the use of spectrum by all users, and ensuring that the Internet remains an engine for

continued innovation and economic growth. These goals are critical to America’s competitiveness in

the 21st century global economy and to addressing many of the nation’s most pressing needs, such

as improving education, health care, and public safety.

Specific NTIA activities include:

Managing the Federal use of spectrum and identifying additional spectrum for commercial

use;

Administering grant programs that further the deployment and use of broadband and other

technologies in America;

Developing policy on issues related to the Internet economy, including online privacy,

copyright protection, cyber security, and the global free flow of information online;

Promoting the stability and security of the Internet’s domain name system through its

participation on behalf of the U.S. government in Internet Corporation for Assigned Names

and Numbers (ICANN) activities; and

Performing cutting-edge telecommunications research and engineering with both Federal

government and private sector partners.

In addition to working with other Executive Branch agencies to develop Administration positions,

NTIA represents the Executive Branch in both domestic and international telecommunications and

information policy activities. NTIA is also a leading source of research and data on the status of

broadband availability and adoption in America.

1.1.1.1. The National Broadband Map

The National Broadband Map (NBM) is a searchable and interactive website that allows

users to view broadband availability across every neighborhood in the United States. The

NBM was created by the National Telecommunications and Information Administration

(NTIA), in collaboration with the Federal Communications Commission (FCC), and in

partnership with 50 states, five territories and the District of Columbia. The NBM is part of

NTIA's State Broadband Initiative. The NBM is updated approximately every six months and

was first published on February 17, 2011.

The State Broadband Initiative was created by the NTIA, to encourage economic growth by

facilitating the integration of broadband and information technology into state and local

economies. In addition to the NBM, the State Broadband Initiative accomplishes this goal by

providing grants. NTIA has awarded a total of $293 million to 56 grantees, one each from

the 50 states, 5 territories, and the District of Columbia, or their designees. Grantees are

using this funding to support the efficient and creative use of broadband technology to

better compete in the digital economy. These state-created efforts vary depending on local

http://www.ntia.doc.gov/sbdd

State of Colorado



needs but include programs to assist small businesses and community institutions in using

technology more effectively, research to investigate barriers to broadband adoption,

innovative applications that increase access to government services and information, and

state and local task forces to expand broadband access and adoption.

States are also using more than 50 percent of these grant funds to gather data twice a year

on the availability, speed, and location of broadband services, as well as the broadband

services that community institutions, such as schools, libraries and hospitals, use. This is

the broadband data used to populate the National Broadband Map. Detailed descriptions of

how this data was collected and processed can be found in the technical overview.

Through the State Broadband Initiative, NTIA has assembled a comprehensive broadband

dataset as well as a data review and validation process to ensure data integrity. The SBI

data is an ongoing, collaborative data collection, review and revision process that involves

the combined efforts of local, state and federal governments, broadband providers, private

contractors, community anchor and academic institutions, and many community members

across the country. Broadband providers voluntarily provided a majority of the data, which

was subjected to evaluation at both the state and federal levels to maximize data accuracy,

reliability and consistency. Notwithstanding the validation process, NTIA cannot guarantee

the accuracy of all data. Furthermore, broadband deployment in the United States is

continually changing and developing. Therefore, the biannual SBDD data release represents

a best-efforts snapshot of the state of broadband deployment at a particular time. Following

each release, NTIA welcomes and encourages input regarding the accuracy of the data and

will use this feedback, together with the state grantees, to inform its iterative data

improvement processes. Please explore the Engage section of the website to see all of the

ways to participate in the National Broadband Map

1.1.2. Colorado Broadband Data and Development Program

The Colorado Broadband Data and Development Program (CBDDP) was created via a grant from

the National Telecommunications and Information Administration (NTIA) for the purpose of

broadband mapping and planning. The Governor's Office of Information Technology (OIT) received

a $2.1 million federal grant in November 2009 from NTIA. The grant provides approximately $1.6

million over two years to continue the State of Colorado's assessment of broadband deployment

across the state and its development of a comprehensive and verified geographically-based

inventory and database of broadband availability. The grant also provides approximately $500,000

over five years to engage in planning and outreach activities for coordination with local groups to

promote broadband adoption and enhance broadband market information.

OIT was subsequently awarded $3.3 million in additional funding, which extends the mapping

project an additional three years, enables the development of a statewide address data

maintenance program, creates a statewide task force to provide interactive learning opportunities

for K-12 public schools, and increases government transparency through the development of a

portal for the discovery of and access to state data.

The CBDDP gathers data wire line, wireless, mobile, and middle mile broadband data directly from

broadband services providers across the state. The data collected contains transmission technology

types, and maximum advertised and typical speeds. The CBDDP also pulls together Community

Anchor Institutions broadband data such as schools, libraries, medical facilities, public safety

entities and other community support agencies. The data is processed and transformed to match

areas used by the US Census Bureau for its population tabulation or to follow streets, where these

http://www.broadbandmap.gov/about/technical-overview

State of Colorado



areas are large, as required by NTIA. The CBDDP also delivers the resulting database to NTIA for

the National Broadband Map. The CBDDP performs several data validation and verification

functions including comparing it third party vendor and Federal Communications Commission

databases, drive testing of mobile service, and surveys of businesses and residences. Consumer speed tests also help validate the broadband service data

The final data delivery for the NTIA was in October 2014, thus ending the State Broadband

Initiative. The State of Colorado general assembly currently funds the continuation of mapping

efforts for the State of Colorado. While moving away from NTIA guidelines and regulations, the

State of Colorado and OIT have the ability to focus in on the needs and unique situations that Colorado presents.

1.2. Objective

NTIA requires that broadband data be updated and delivered to them every six months. The

CBDDP will continue to refine the broadband service data through this update and validation

process. The CBDDP attempts to contact all known broadband Providers in the State of Colorado in

an effort to provide the most accurate and detailed representation of broadband coverage within

the State. Post processing involves attempts to automate the process and refine the accuracy of

the data. The CBDDP also updates the Colorado Broadband Map biannually, in an attempt to keep

the map as a useful tool for Providers and Consumers.

1.2.1. Helpful Links

Colorado Broadband Data and Development Program: http://broadband.co.gov/

National Broadband Map: www.broadbandmap.gov

http://www.broadbandmap.gov/

http://broadband.co.gov/

http://www.broadbandmap.gov/

State of Colorado



1.3. Data Processing Flow Chart

State of Colorado



2. General Information

2.1. Resource Documents

2.1.1. Microsoft Access Database

The ProviderStatusCURRENT.accdb is used to keep track of all Providers in the State of Colorado.

The contents of the database are not limited to only the Providers participating in the collection of

data, but also Providers who are no longer in business, not interested in participating, do not meet

Broadband requirements, etc.

In addition to keeping a record of all Providers in Colorado, the database consists of contact

information, raw data types, and data submitted in previous deliveries. The database has links to

file folders and README.txts. The contents of the file folders are outlined below, along with a

description of the README.txt. The database is located:

P:\SBDDGP-ARRA Project\BroadbandProductionArea\Team_Documents\Team Documents\ProviderStatusCURRENT.accdb

2.1.2. Providers and Stats for monthYEAR

Every six months, a new Provider and Stats document is derived from the data submissions. The

documents consist of a list of Service Providers who are participating and the types of Broadband

services available from them. The Stats section of the document outlines the type of information

collected from the broadband Providers and the percentage of data meeting those requirements

which is derived from the data submissions. Since the document is created every six months, a

new folder is created for every delivery.

Consists of four separate documents.

(1) Colorado Service Providers disclosing the list of current broadband Providers in Colorado,

FRN, Company Name, DBA, Census, Roads, Wireless, and Midmile.

(2) Colorado Middle Mile outlines Data Types, Code, Data Element, Count, %, for Ownership,

Facility Capacity, Facility Type, Lat/Long, and Elevation available in Colorado.

(3) Colorado PLSS outlines Data Types, Code, Data Element, Count, and % for Technology,

Speeds, and Spectrum

(4) Colorado Wireless outlines Data Types, Code, Data Element, Count, and % for

Technology, Speeds, and Spectrum.

The data folder for each delivery is located:

P:\SBDDGP-ARRA Project\BroadbandProductionArea\Documents\NTIA_ReportsMaterials

***The information is available inside the Methodology Document

2.1.3. SBDD_TRANSFER_monthYEAR

The purpose is to provide a current file meeting the submission requirements for the next

delivery. Object model diagram outlining the Geodatabase Design for Colorado Broadband

Coverage using the OIT Standard. Document consists of Feature Class Descriptions, Subtype

State of Colorado



Descriptions, Domains, CODES, Default Values, and Field Names for PLSS, Middle Mile, and

Wireless.

2.2. Workspace

2.2.1. Broadband Production Area

The BroadbandProductionArea is reserved for storing, processing, training, outreach, workspaces,

and documentation for all broadband related data.

Broadband Production Areas is located at:

P:\SBDDGP-ARRA Project\BroadbandProductionArea

2.2.2. Base Data

The Base data folder consists of *.gdb’s, feature classes, templates, tools, and address locators

associated with the processing of raw data from Providers. A description is provided below for each

aspect of the BaseData folder.

Base Data is located in:

…\BroadbandProductionArea\Workspaces\BaseData\

2.2.2.1. Publice Land Survey System – Quarter Quarters

base_data.gdb\OIT_PLSS_QQ

OIT’s PLSS QQ shapefile was created by OIT to replace Census Blocks and Roads geography

due to their size and ability to exaggerate broadband coverage in the state. OIT was able to

fill in gaps of Spanish Land Grant areas. The OIT PLSS QQ is not available for public use for

any decision making.

2.2.2.2. CENSUS BLOCKS (Discontinued methods…)

base_data.gdb\Blocks_2010_WGS84

Census Block Shapefile is an extract of selected geographic and cartographic information

from the U.S. Census Bureau’s MAF/TIGER and Referencing MTDB Database. The Shapefile

is used to distinguish broadband coverage between Providers, in addition to Road networks.

All Census Blocks in Colorado exist within the Shapefile and .dbf. Unnecessary Fields have

been eliminated for ease in processing and to comply with NTIA requirements.

base_data.gdb\Blocks_2010_WGS84_LT2SqMi

This is a subset of the Blocks_2010_WGS84.shp that consists of Census Blocks with areas

less than 2 square miles.

2.2.2.3. TIGER ROADS (Discontinued methods…)

base_data.gdb\Roads_2010_WGS84

State of Colorado



The TIGER/Line Files are shapefiles and related database files that are an extract of selected

geographic and cartographic information from the U.S. Census Bureau’s MAF/TIGER and

Referencing MTDB Database. The TIGER shapefile consists of the Colorado road network and

is used in conjunction with Census Blocks to build a broadband coverage for the state of

Colorado. Unnecessary Fields have been eliminated and attributes have been modified to

comply with NTIA formatting requirements. The following fields were added and calculated

using the following tools:

ADDMIN & ADDMAX were added and calculated using CalcMaxMinAddr

STREETNAME was added and calculated using CalcStreetName

base_data.gdb\Roads_2010_WGS84_Outside

This is a subset of the Roads_2010_WGS84 feature class that consists of the Colorado road

network outside of Census Blocks less than 2 sq. mi. The file is used in conjunction with

Census Blocks to build a broadband coverage for the state of Colorado.

2.2.2.4. ADDRESS LOCATORS Colorado State Address Dataset Locator (CSAD) / QGIS Address Locator

In December 31, 2013, ESRI has discontinued its free geocoding for batch geocoding. While

single address geocoding is still free, it would require too much time to process.

CSAD Locator:

1. Add CSAD locator when prompted

(http://maps.co.gov/copubgis/rest/services/Addresses/PALocation/GeocodeServer)

2. Geocode

QGIS Method(use as a backup):

1. Download QGIS - http://www.qgis.org/en/site/forusers/download.html

2. Add MMQGIS extension inside QGIS

a. Open QGIS > Plugins > Manage and Install Plugins > MMQGIS and Geocding

extensions

2.2.2.5. Additional Base Data

Counties.shp Feature Class

Feature class created to display the 64 counties in Colorado, includes numerical attribution,

county name, GEOID10 (FULLFIPSID), Area, etc.

COZipCites

dBASE table consists of cities and zip codes for state of Colorado.

Lookup.gdb

Within the lookup.gdb, is a lookup table and lookup template. The lookup table is used in

the StagingTool along with the staging template in order to

populate fields: Provider Name, DBName, FRN, EndUserCat,

and Provider Type. The lookup.dbf should be updated every

cycle to ensure an accurate list of participating Providers.

http://www.qgis.org/en/site/forusers/download.html

State of Colorado



StagingTemplate.gdb

The Staging Template consists of 3 empty feature

classes: midmile, plss, and wireless. The feature

classes are used to load processed ‘Standard GIS

Format’ feature classes from *[ProvAlias].gdb’s within

your workspace. Additionally, the template is used with

the Staging Area tool in order to produce a final Staging Area gdb within the specified

Staging Area. The template meets all OIT requirements. When using the Staging Template

for loading processed feature classes, copy and paste only the necessary feature classes

into your workspace.

***NOTE: NEVER load data into the primary template within the Base Data folder.

2.2.1. Personal Workspace

The raw data, base data, and supporting documents can be found at the locations listed above,

but for processing purposes the data must be copied to your personal workspace, which is

located at:

…\BroadbandProductionArea\Workspaces\[Your Name]

(This will be referred to as “…Your_Workspace”)

***NOTE: Although you setup a workspace in the BroadbandProductionArea, it is recommended to do ALL data processing on your local machine.

2.2.1.1. Workspace Setup

Provider Folders

Correspondence

Consists of all correspondence with Provider via email. Can be saved as PDF from Gmail

account. Microsoft Outlook correspondence may be present in some folders from

previous deliveries since the State of Colorado used Outlook until May 2012

FromProvider Folder

Folder consists of all raw data sent from Provider for data processing. Data can be in the

form of all previously listed types of raw data

Working Folder

Consists of modified raw data, *.gdb’s, *.mxd’s, etc; created while processing raw data

to standard GIS format

Processed Folder

Consists of the final *.gdb from Provider ready for the Staging Area.

README.txt

The README.txt is used to describe methods of raw data processing for each Provider,

for each delivery. It should also include any correspondence insight. Each new entry

should be placed at the top of the text file, dated, and have Analyst initials. With each

new Delivery, specific information should be listed as follows:

1. Current Delivery (e.g. April 2013 Delivery)

a. PLSS Values

b. Wireless Values

c. Midmile Values

2. Previous Delivery (e.g. October 2012 Delivery)

a. PLSS Values

State of Colorado



b. Wireless Values

c. Midmile Values

3. Processing Methods, including complications with

processing (if any)

4. Detailed explanation for the difference between delivery

values (if any)

ASSUMPTIONS.DOCX

The assumptions document is used to describe and clarify

any judgement calls used in processing. Post NTIA

delivery, OIT has used a confidence scale to rate the

quality of data received from a provider. This document is

to demonstrate why/how the data was processed with

screen captures.

Naming Convention

A naming convention has been established for Provider folders to ensure consistency.

Naming convention as follows:

[ProviderAlias] Folder

Correspondence

FromProvider

Working

Processed

[ProvAlias].gdb

plss

midmile

wireless

README.txt

To set up “…Your_Workspace”:

(1) Create a folder for each provider you

process in your workspace with the four

sub-folders listed above. Copy raw data

from Provider into the “FromProvider”

folder. Put correspondence in

Correspondence folder and copy the

README.txt.

(2) Create an empty File Geodatabase in

the “Working” folder that you will use to

process all of your data

Open ArcCatalog and navigate to

Workspace

Right Click>New>File Geodatabase.

Name the file gdb>rename:

“provalias.gdb”

(3) Create a copy of the StagingTemplate.gdb in the “Processed” folder>rename

[ProvAlias].gdb. You will load processed feature classes into the *.gdb after

your data is processed to the GIS Standard Format.

State of Colorado



StagingTemplate location:

…\BroadbandProductionArea\Workspaces\BaseData\StagingTemplate.gdb

2.2.2. Staging Area

The Staging Area is ONLY intended for Provider data post-processing. Each [Provider].gdb must be

loaded into the Staging Area separately and with proper naming convention. Each geodatabase will

stand alone. Do not place folders inside staging area.

The current Staging area for April 2013 Data Delivery is located at:

…\BroadbandProductionArea\Workspaces\2013April_StagingArea

2.2.3. OIT Final Format

The OIT Final Product area is located at:

…\BroadbandProductionArea\Workspaces\2013April _FinalProduct

Example of 2012October_StagingArea with

stand-alone Provider geodatabases with proper naming convention and format

State of Colorado



2.3. Raw Data

2.3.1. Purpose and Procedure Raw Data refers to the files or correspondence relating to a Provider’s Broadband coverage. The

Raw Data is sent routinely by Providers bi-annually, whether or not broadband coverage has

changed.

Providers can provide raw data in numerous file forms. In order to extract pertinent data, the raw

file type needs to be evaluated and, in many cases, converted to a usable file type for

geoprocessing. Other Raw Data file types need to be georeferenced for data extraction.

2.3.2. Raw Data File Types

2.3.2.1. Microsoft Word Document (.doc/.docx)

Microsoft Word Documents contain data like addresses, XY coordinates, typical speeds, and

advertised Speeds. The information presented needs to be converted into a table in Excel

spreadsheet for geoprocessing.

2.3.2.2. Text (.txt)

Text refers to tab delimited text documents created in programs like Wordpad and Notepad.

An example can be found in Delta/Strasburg raw data. The document contained information

like addresses, latitude/longitude, speeds, techtype, etc. The data can easily be copy/paste

into an Excel spreadsheet for geoprocessing.

2.3.2.3. CAD (Computer Aided Drafting)

CAD files consist of several files: Annotation, MultiPatch, CAD Drawing, Point Feature Class,

Line Feature Class, and Polygon Feature Class. Metadata associated with the CAD files may

be provided and imported into the GIS File Geodatabase. Although the images can be

viewed in ArcGIS, they cannot be edited. All files must be converted to GIS Feature Classes.

2.3.2.4. KML/KMZ

KML or KMZ (Keyhole Markup Language) is a file format used to display geographic data in

an earth browser, such as Google Earth. This can contain Point, Line, and Polygon data.

KML files are not compatible with ESRI software, but the layers can be converted into

shapefiles.

2.3.2.5. Outlook Item

Correspondence through Outlook email can contain important provider data. The

information is used to fill in attribute data and can contain location information as well.

Please read all correspondence with the raw data and extract any necessary information.

2.3.2.6. Excel Spreadsheet (.xls/.xlsx)

Excel files are the easiest file type to translate into ArcMap and ArcCatalog. All raw data file

types need to be converted to this format in order to create dBASE tables for processing. In

order to have a working excel table, the data within the table needs to be clean. A clean

State of Colorado



table refers to not having any special characteristics about the data: bold font, merged

columns or rows, special characters, etc.

2.3.2.7. Shapefile (.shp)

Although data is rarely given by Providers in a Shapefile format, little processing needs to

occur because the data is already in GIS Format. If the shapefile consists of point data, then

most processing steps still need to occur, with the exception of midmile. If the provider

sends census block or road segment data in the form of a shapefile then little processing

needs to occur prior to Standard GIS Format. The shapefile data needs to be evaluated with

scrutiny to ensure all the necessary fields exist and the data is accurate.

2.3.2.8. PDF

Portable Document Format File used by Adobe or Acrobat plug-in. The file displays as an

image. If the PDF has an image demarcating the boundary of the Provider’s broadband

coverage, then the image needs to be extracted and treated as an image. If the PDF

contains tabular data, the data can be extracted and imported into an Excel spreadsheet.

2.3.2.9. Image File (.jpg, .tif, .gif…etc)

Joint Photographic Expert Group format is indicative of an IMAGE file. The JPEG typically

delineates the boundary of a Provider’s broadband coverage; however, the image could also

contain speeds and technology transmission types. The image may need to be

georeferenced for future processing.

2.4. Broadband Processing Toolbox

The custom toolbox (BroadbandProductionTools.tbx) is

located at:

…\BroadbandProductionArea\Workspaces\Tools

This toolbox contains the ArcGIS tools that are used during

data processing, as well as custom tools specifically built by

the Colorado Broadband Production team. To add the

toolbox:

Open ArcMap or ArcCatalog and click the ArcToolbox

button

Right click on the ArcToolbox folder and choose Add

Toolbox…

Navigate to the custom toolbox location. Add the

State of Colorado



BroadbandProductionTools.tbx – it should appear as

shown to the right

The tools contained in the toolbox will be referenced throughout this document as needed. The

following table contains descriptions of each tool. See Tool Development section at the end of this

document for more detail about the Custom Tools

***Note: Add toolbox to local machine (C://), otherwise, you will have to add the toolbox every

time you use ArcMap or ArcCatalog

Name Function

Clip Cuts out a piece of one feature class using another feature class. (Used in DSLAM process)

Spatial Join Transfers the attributes from one feature class to another feature class based on the spatial relationship

Buffer Creates buffer polygons around input features to a specified distance

Frequency Creates a new table containing unique field values and the number of occurrences of each unique field value

Summary Statistics Calculates summary statistics for field(s) in a table.

KML To Layer Converts a KML or KMZ file into a feature class along with a layer file derived from the source renderer information inside the KML/KMZ.

Copy Features Copies features from the input feature class or layer to a new feature class

Append Appends multiple input datasets into an existing target dataset

Merge Combines multiple input datasets of the same data type into a single, new output dataset

Dissolve Aggregates features based on specified attributes.

Add Join Joins a table view to a layer (or a table view to a table view) based on a common field

Make XY Event Layer Create Feature class from XY data

Define Projection This tool overwrites the coordinate system information (map projection and datum) stored with a dataset. The only use for this tool is for datasets that have an unknown or incorrect coordinate system defined

Project Creates a new dataset or feature class with the coordinate system specified

Create Address Locator

Creates an address locator

Geocode Addresses Matches the addresses against the address locator and saves the result for each input record in a new point feature class

Rematch Addresses Rematches addresses in a geocoded feature class

CUSTOM TOOLS

CalcMaxMinAddr (discontinued)

Calculate AddMin & AddMax for Roads_2010_WGS84 - Developed by Larry

CalcStreetName (discontinued)

Calculate STREETNAME for Roads_2010_WGS84 - Developed by Larry

CensusRoads (discontinued)

Selects Census Blocks and Roads from coverage feature class (point, line, and polygon). Gives output ready for staging tool.

State of Colorado



Metrics Calculation Script

Each census block will show the Max Advertised Download speeds generated from the census, roads, and fixed wireless coverage in the feature classes for the NTIA delivery.

MidmileTool (discontinued)

Specific for the midmile feature class needed in the NTIA Data Submission; This tool will load a dBASE table properly formated to the staging template; Input empty point feature class; and will batch convert KML/KMZ's located in a folder. If the census block FIPS are not present in the raw data, the tool will use census block feature class to assign a FIPS ID to each midmile

OITFinalFormat

This script reads through the Staging Area and appends data from each feature class into a single geodatabase. The resulting geodatabase contains all provider data for delivery to the NTIA. Use this final gdb to load data into the NTIA data submission template.

Staging Tool Check fields & attribute data of provider supplied Shapefile and put in Standard GIS Format (add & calculate fields). Populate provider information from lookup table

VALIDATION TOOLS

CAI Validation (discontinued)

Validates CAI speed tests against the Terrestrial_MAXADDOWN and the Terrestrial_MINADDOWN of current delivery cycle. Outputs a file geodatabase called, CAI Validation, 2 frequency tables, and 2 validation feature classes.

Drive Test Validation (discontinued)

The Drive Test Validation script provides frequency tables and feature classes associated with: ATT, Sprint, and Verizon. The script uses buffered points and spatially joins to appropriate coverage layer to provide the necessary tables used in the Drive Test section of the Methodology Document

FCC Validation (discontinued)

Creates the required frequency tables to populate the FCC Speed Test Validation section in the Methodology Document due for every delivery cycle: Terrestrial, Composite, and Mobile. The tables are created using, the mobileMAXADDOWN validation layer, the NTIA Final Wireless Feature Class, the Terrestrial_MAXADDOWN , and the Terrestrial_MINADDOWN coverage layers. The layers are spatially joined to the appropriate speed test feature classes: CO_MobileApp_points and CO_CBTDATA_points. The final output consists of frequency tables.

Mobile Pulse Validation (discontinued)

Mobile Pulse Validation tool is used to create frequency tables to use in the Methodology Document for every delivery cycle

NTIA Validation Layers (discontinued)

Creates a file geodatabase containing validation layers: terrestrial_MAXADDOWN, terrestrial_MINADDOWN, and mobile_MAXADDOWN using the NTIA Final feature classes: census, roads, and wireless

MAPSERVICE TOOLS

numProv COBB Mapservice

Creates numProv_ layers for Fixed, Mobile, Wireline, Satellite.

provNameSpeed COBB Mapservice

Creates the five feature classes required for the provNameSpeed broadband coverage for the Colorado Broadband Mapping Application Mapservices: provNameSpeed, provNameSpeed_Wireline, provNameSpeed_Fixed, provNameSpeed_Mobile, provNameSpeed_Satellite.

serviceValue COBB Mapservice

serviceValue COBB Mapservice creates four feature classes

serviceValue Wireless

Used to create feature classes for serviceValue_Wireless ONLY. Creates only three of the four feature classes required to give the complete Colorado broadband coverage.

serviceValue Wireline

Used to create serviceValue_Wireline coverage ONLY. Will create one of four feature classes needed to show complete Colorado Broadband coverage

speedTier COBB Mapservice

numProvTool Finds overlapping polygons and gives a count. Used to generate the number of

State of Colorado



provides for a specific area.

3. Pre-Processing Raw Data

3.1. Data Specifications

3.1.1. Identifying Data Type

3.1.1.1. PLSS Quarter-Quarter Sections

A PLSS feature class is indicative of landline or ‘wireline’ broadband service. The raw data

for a PLSS feature class output can be presented in multiple data types. The most

recognizable raw data type for the collection of PLSS are subscriber addresses. Subscriber

addresses can be presented as tabular data or within a shapefile. A list of addresses is the

first hint, but not necessarily restricting addresses to ‘wireline’. Census Block ID’s

(FULLFIPSID) are also indicative of PLSS output. FULLFIPSID or GEOID10 is a 15 digit

sequence representing state, county, tract, and block ID according to the Census Bureau.

Whether tabular (xlsx., csv, txt) or Shapefile, the data is used to collect all census blocks

less than 2 square miles for the final output.

If coverage is ‘wireline’ and provided as an image file (jpeg, pdf, KML), the coverage is used

to extract census blocks from the Base Data Blocks shapefile. Transmission Technology

codes for census blocks found in the raw data are as follows: 10, 20, 30, 40, 41, and 50. If

transmission codes are not provided, then words like: cable modem, DSL, Copper, and fiber

are used to describe the ‘wireline’ service. Raw Data should also contain ‘Speed’ values.

Processing PLSS is described in Section 4.

3.1.1.2. Middle Mile Output

A middle mile output is probably the simplest and easiest to identify. If the raw data will

provide or currently provides point data via an address or latitude/longitude, and also

contains information for Ownership, Facility Capacity and Facility Type; hands-down, the

output will be a midmile feature class. The midmile feature class is ONLY represented as

points. Midmile processing is described in Section 7. TranTech codes and Speeds should not

be present in the raw data, since neither are indicative of midmiles.

Special circumstances surrounding midmiles includes wireless tower points. If a provider has

wireless tower points, those points are also collected as midmiles---in addition to following

the wireless tower processing procedure described in Section 6 to provide a wireless

coverage polygon.

3.1.1.3. Wireless Output

Wireless data can be a bit more difficult to discern. The easiest hint is to look for

information on Spectrum, tower location, azimuth, beam radius, etc within the raw data.

Wireless data is presented as either a polygon or as point data. If the raw data is displayed

as a polygon on an image or shapefile, it is extracted as such. If, however, the data is

presented as tower locations, the data is processed in Radio Mobile to provide a polygon

State of Colorado



coverage output. As explained above, the tower points are also processed as midmiles.

TransTech codes for wireless are as follows: 70 and 71 for fixed wireless; 80 for mobile

services, or 60 for satellite services. So keep in mind, fixed wireless, mobile, and satellite

services are all considered a wireless output. Wireless raw data should also have speed

information present for processing. Wireless data processing is described in Section 6.

3.1.2. Transmission Technology (TRANSTECH)

Transmission technology refers to the type of technology used to transmit the broadband service.

TRANSTECH codes are collected for Census Blocks, Road Segments, Wireless, and Address Points.

Please review the table below:

Technology Codes

10 Asymmetric xDSL

20 Symmetric xDSL

30 Other Copper Wireline

40 Cable Modem-DOCSIS 3.0

41 Cable Modem-Other

50 Optical Carrier/Fiber to End User

60 Satellite

70 Terrestrial Fixed Wireless-Unlicensed

71 Terrestrial Fixed Wireless-Licensed

80 Terrestrial Mobile Wireless

90 Electric Power Lines

0 All Other

TRANSTECH Codes should be easy to identify in raw data. Words like: transtech, techcodes,

transmission, technology, etc are indicative of a transtech field. However, in some cases, the type

of technology is described using words like: DSL, Asymmetrical, Symmetrical, cable, copper,

satellite, mobile, modem, fiber, T1, etc. The TRANSTECH field can never be ‘NULL’ or empty. The

information needs to be collected from the Provider if no semblance of transmission technology is

provided in the raw data.

3.1.3. Maximum Advertised Download and Upload Speeds

OIT no longer collects data in Speed Tier format, this was a guideline from the NTIA. Post October

2014, OIT decided the Speed Tiers were less accurate and were becoming a bit data. OIT decided

to collect data by actual speed instead of tiers. There are some aspects of the program that

eventually uses speedtiers, such as mapservices.

MAXADDOWN and MAXADUP refer to maximum advertised speeds by the Provider. Download

speeds less than 768 mbps are considered too slow to be broadband. Upload speeds less than 200

kbps are considered too slow to be broadband. Broadband speeds are broken down into Speed

Tiers. It is the speed tier code, not the broadband speed, used to populate the MAXADDOWN and

MAXADUP fields. When reviewing raw data, be sure to recognize the difference between a speed

tier code and broadband speed. Some conversion or ‘clean-up’ of the raw data may be necessary.

See the examples below:

Speed Tier

Codes Speed Ranges

1 <= 200 kps Speed Tiers used in Data Processing

State of Colorado



2 >200 kps, < 768 kps

3 >= 768 kps, < 1.5 mbps

4 >= 1.5 mbps, < 3 mbps

5 >= 3 mbps, < 6 mbps

6 >= 6 mbps, < 10 mbps

7 >= 10 mbps, < 25 mbps

8 >= 25 mbps, < 50 mbps

9 >= 50 mbps, < 100 mbps

10 >= 100 mbps, < 1 gbps

11 >= 1 gbps

Recognize download speeds should ALWAYS be faster than upload speeds. If the raw data shows a

faster Upload speed, the data is inaccurate. The Provider will need to be contacted for corrections.

3.2. Pre-Processing Standardization

3.2.1. Clean Tabular Data

First, and foremost, do not make changes to original raw data. ALWAYS make a copy on your local

machine or within your workspace BEFORE cleaning tabular data. With the start of pre-processing,

make sure to have your workspace folders setup as described in 2.2.1 Personal Workspace.

Tabular data begins in either an excel spreadsheet (*.xlsx), a text file (*.txt) or a comma

separated values file (*.csv). If the raw data is not in any of the aforementioned file types, it is not

tabular data. If the file is in csv or txt, it would be advantageous to copy and paste into an excel

spreadsheet for ease of use. Excel spreadsheets are the easiest to clean and edit. Formulas exist

for conversions, concatenations, and parsing.

Once in Excel, make sure the data has no special characters, void of fill colors within cells, no

active filters or frozen frames, and no special fonts or colored font. Make sure column headings

have underscores (_) instead of spaces, do not start with numerical values, and are reasonable in

length. In addition, consider renaming the column headings the same name as the fields within the

expected data output. Attempt to recognize the fields required for processing.

Tabular data can have transtech codes for wireline and fixed wireless mixed in together. Be sure to

separate Wireline transtechs (10, 20, 30, 40, 41, 50) from fixed wireless transtechs (60, 70, 71,

80) beforehand, so the data isn’t compromised.

Upstream and Downstream refer to

the MAXADUP and MAXADDOWN

data fields. Here is an example of

data given in Mbps standard instead

of Speed Tier Code. Data Needs to be converted

State of Colorado



3.2.1.1. Latitude/Longitude

Latitude and Longitude raw data should be in Decimal Degrees NOT Degrees-Minutes-

Seconds!

To Convert Degrees-Minutes-Seconds to Decimal Degrees, use the following formula:

***Note: Do not forget the conversion formula does not account for the Longitude value being (-). After the Conversion is complete, multiply ONLY the Longitude by (-1). The adjustment needs to be documented in the Excel table. Also, if the raw data indicates the Latitudes and Longitudes are already in DMS, verify the minutes-Seconds are correctly broken into fractional parts

The following is an example of how Lat/Long should look in raw data pre-processing:

Latitude and Longitude raw data can be presented for midmile, wireless towers, and address

points. Once the spreadsheet is clean, create a dBASE table in ArcCatalog.

a) Navigate to ‘working’ Folder

b) Right Click on ‘[provalias.gdb]’>Select Import>Table(Single)

c) In ‘Input Rows’ field, Navigate to the *.xlsx> expand to see appropriate ‘sheet’

d) Ensure Output Location path is into new geodatabase and rename Output Table>OK

The creation of the table may take a minute. If the table creation fails, evaluate spreadsheet

to ensure all special formatting has been removed.

With a functioning dBASE table in ArcCatalog, proceed to the geoprocessing section for the

desired output.

3.2.1.2. Subscriber Addresses

Cleaning subscriber addresses can be a daunting task. The tabular data will eventually be

geocoded. In order to avoid having to geocode multiple times, the best method is to clean

raw tabular data before using a geocoder. Subscriber Addresses can come as a full address

or a parsed address. See examples below:

Table 1: Example of Concatenated Address

State of Colorado



Table 2: Example of parsed address

Removing Apartment Numbers or Suite Numbers

Once all special formatting has been removed, the next step is to evaluate the data.

Addresses with apartment numbers or suite numbers have a greater likelihood of not

geocoding successfully. To remove: ‘insert’ a new column> give column heading>select the

first empty cell and input the following code in the fx:

=IF(ISNUMBER(SEARCH(“APT”, A2)) = TRUE, (TRIM(LEFT(A2,((FIND(“APT”,A2)-1))))), A2)

The formula is searching for values APT within column A and row 2. Thus, the formula can

be changed if column A does not contain addresses. Change the formula to match the

tabular data. If ‘STE’ for suite, needs to be removed instead of APT, modify the formula.

Once the formula is likened to the tabular data, double click the bottom right corner of the

cell to apply to following cells.

Concatenating Addresses

A concatenated address is needed for most geocoders. Keep in mind, the full address

consists of house number, street prefix, street name, and street suffix. City, State, and Zip

should remain separate columns. Concatenating a parsed address can be relatively simple.

‘Insert’ new column>give column heading>select following empty cell>

Concatenate by typing formula in new Column: =CONCATENATE(A2&B2&C2&D2)

‘=CONCATENATE()’ is the command for concatenating, the ‘&’ ensures a space between the

data from each column. The example above is supposing the address data is parsed into 4

columns. If you look at Table 2 above, the address is only parsed into three columns. Once

again, be sure the formula is likened to the tabular data. Double click the bottom right

corner of the cell to apply formula to remaining addresses.

County Names or Military Facilities

Sometimes the raw data uses alternate names in the ‘City’ column: County name, Indian

Reservation, Military base, etc. The best option is to use Google Maps or Bing Maps to plot

the address point in order to learn the actual city name. The geocoder will most likely not

recognize a military base name (e.g. Fort Carson), Indian Reservation name (e.g. Ute

Indian Tribe Reservation), or county name (e.g. Jefferson County).

dBASE Table for Geoprocessing

Address data can be submitted for PLSS output, midmile locations, and wireless towers. Be

aware of the data output type before converting to a dBASE table. To create a dBASE table,

follow steps below:


State of Colorado






***NOTE: Exporting .xlsx to dBASE Tables fails in ArcGIS***



3.2.1.3. Address Ranges

If address data is provided as address ranges instead of individual address numbers that

does not match the TIGER Streets data, then geocoding will not be successful.

Solution: grab the highest high and lowest low of each address range. Apply toward tabular

data, then follow process outlined in Subscriber Addresses.

3.2.1.4. Census Block ID

Evaluate data layout in Excel table. If Census Block ID’s consist of 15 numbers (no letters),

then the Census Blocks table just needs to be converted to a dBASE table. If, however, the

Census Blocks are divided by State, County, Tract, and Block; then the fields will need to be

concatenated in a new field named “FULLFIPSID”. ‘Insert’ new column> Column Header

“FULLFIPSID”.

In order to concatenate the fields, the numerical values need proper preceding “0”s and

must be in a ‘text’ format.

***NOTE: If cell values lose preceding ‘0’s with concatenation, try this method instead:

o Insert 6 new columns with names: State2, County2, Track2, Block2, Concatenate, and FULLFIPSID o In State2 Column, first cell, type:=TEXT(value, format_text); whereas value, in this case is equal to

A2, and format _text is equal to “00”. Paste formula through column by selecting value cell and dragging down the right bottom corner. Using the table above as an example, here is the formula for each cell to be typed in fx:

State2: =TEXT(A2, “00”) County2: =TEXT(B2, “000”) Tract2: =TEXT(C2, “000000”) Block2: =TEXT(D2, “0000”) Concatenate: =CONCATENATE(A2,B2,C2,D2) FULLFIPSID: Copy value>Paste Special>Select Paste Values

o Eliminate unnecessary fields now that you have a concatenated FULLFIPSID

Receiving raw data with census block ID usually means a PLSS output will be required, and

possibly a subsequent road segment output. Ensure TransTech and Speed fields are also in

the raw data before converting to a dBASE table.

To create a dBASE table, follow steps below:


Concatenate by typing formula in new Column: =CONCATENATE(A2,B2,C2,D2) Once calculation is made, drag the

‘paste’ box down to automate formula for all Census Blocks

State of Colorado








3.2.2. Image Raw Data (jpeg, pdf, etc)

If raw data is presented as an image with point, polygon, or line coverage; the image will need to

be georeferenced and mostly likely digitized. Georeference and Digitizing procedures are

considered general GIS skills and can be found in the Appendix: Georeference and Digitizing.

Image data can also present Address information or Latitude/Longitude. The provider may have

opted to send the data as a PDF instead of spreadsheet or word document. In such cases, a

manual extraction of the data may be necessary. Put in Excel spreadsheet and review sections on

cleaning tabular data listed above.

3.2.3. Shapefile

Shapefiles from provider can be imported directly in the working.gdb. No pre-processing

required. To verify data output type:

Identify the coverage type of the shapefile (wireline, fixed wireless, or middle mile)

Verify the data has all necessary fields for processing and final NTIA Fields

When the data output is verified and imported into working.gdb, proceed to processing

State of Colorado



Coverage Type

Shapefile Pre - Processing Instructions Final Format

ALL ALL Verify attribute information and import to working.gdb

Middle Mile Point No pre-processing necessary - Refer to Section: 7 Point

Wireless Point No pre-processing necessary - Refer to Section 6 Polygon

Wireless Polygon No pre-processing necessary - Refer to Section 6 Polygon

Wireline Point No pre-processing necessary - Refer to Section 4 and 5 Roads/Census Blocks

Wireline Polyline - Roads

Verify that roads format matches NTIA data structure exactly. For example, road layers from providers may have the address information listed in 5 fields instead of 4. Often the simplest solution is a spatial join or an attribute join (based on TLID) to the base data roads. You must also verify that the provider roads are outside Census blocks < 2 sq mi. Any road data located within census blocks < 2 sq mi, must be extracted and used to select Census Block coverage

Roads/Census Blocks

Wireline Polyline - Coverage

No pre-processing necessary - Refer to Section 4 and 5 Roads/Census Blocks

Wireline Polygon - Census Blocks

Verify that census blocks format matches NTIA data structure exactly. For example, census block layers from providers may contain census block > 2 sq mi. These must be extracted and used to create the roads layer. If data is inconsistent, a spatial join or an attribute join (based on Full FIPS ID) to the census block base data can be performed.

Roads/Census Blocks

Wireline Polygon - Coverage

No pre-processing necessary - Refer to Section 4 and 5 Roads/Census Blocks

State of Colorado



4. Public Land Survey System (PLSS)

Quarter Quarter Sections

4.1. Introduction

PLSS quarter quarter sections are the most common output for providers. This format is used to

represent landline broadband coverage across Colorado. The PLSS dataset features uniquely

identified features and is used in conjunction with address point locations and imagery in order to

determine and process provider landline coverage.

The Table below outlines the data fields required by OIT for the final delivery:

PLSS Quarter Quarter Sections

Field Name Description Source Notes

PROVALIAS Provider Alias Lookup.gdb Add to link data to all other data tables

PROVNAME Provider Name Access database Unique to each provider. A complete list is stored in the Lookup.gdb along with other base data

DBANAME “Doing Business As” Name Lookup.gdb Unique to each provider. A complete list is stored in

the Lookup.gdb along with other base data

FRN FCC Registration Number Lookup.gdb Unique to each provider. A complete list is stored in


TRANSTECH Transmission Technology Raw data From provider or derived from provider’s website

MAXADDOWN Max Advertised Download Speed Raw data From provider or derived from provider’s website

MAXADUP Max Advertised Up Speed Raw data From provider or derived from provider’s website

TYPICDOWN Typical Download Speed Raw data From provider ONLY - - or “Null”

TYPICUP Typical Upload Speed Raw data From provider ONLY - - or “Null”

MAXSUBDOWN Subscriber Download Speed Raw data From provider ONLY - - or “Null”

MAXSUBUP Subscriber Upload Speed Raw data From provider ONLY - - or “Null”

PRICE Price of Maximum Advertised

Speed Package at Location

Raw data From provider or derived from provider’s website

CONFIDENCE Provider Data Confidence Rank Staging Assigned by analyst

PROVIDER_TYPE Provider Type Raw data From provider or derived from provider’s website

ENDUSERCAT End User Category Raw data From provider or derived from provider’s website

PLSSID PLSS ID Number plssTemplateUpdate.gdb In PLSS feature data

REFGRIDNO Reference Grid Number plssTemplateUpdate.gdb In PLSS feature data

OIT_ID Unique PLSS Identifier plssTemplateUpdate.gdb Unique ID created by OIT for PLSS quarter quarter

sections

Table 3: Orange highlighted fields are required for data processing; Remaining fields provided or calculated by Staging Tool

State of Colorado



PLSS Data Processing Flow Chart

4.2. Data Processing

Step 1: Verify PLSS Quarter Quarter sections is the desired data output

Step 2: Make sure a dBASE table, shapefile, or georeferenced image is available. If neither is

available then refer back to Section 3: Pre-Processing Raw Data

State of Colorado



4.2.1. dBase Table

4.2.1.1. Subscriber Addresses

Geocode Subscriber Address points. Being a general GIS skill, the Geocoding process can be found

in the Appendix: Geocoding. Once the points have been geocoded, separate points by TRANSTECH

code. You can end up with up to six different layers of points depending on the number of wireline

transtech codes present. Any subscriber addresses which are not successfully matched via

geocoding must be created manually. This can be accomplished by searching for these addresses

using an online search engine with map applications such as Google or Bing. Once the location of

the address is found, use imagery within ArcMap in order to find the location seen on Google or

Bing maps and create a point feature for that address. Despite this effort, some address simply

cannot be located; document these specifically in the provider readme.

What you need: Point.shp (Divided Point Feature Classes)

and

OIT_PLSS_QQ

(P:\SBDDGP-ARRA\Project\BroadbandProductionArea\Workspaces\BaseData\base_data.gdb\OIT_PLSS_QQ)

FOR EACH TRANSTECH point feature class

Add Point.shp and OIT_PLSS_QQ.shp into ArcMap.

Do a ‘Spatial Join’ with the OIT_PLSS_QQ. The Target Feature is the OIT_PLSS_QQ and the

Join is the Point.shp. UNCHECK “Keep All Target Features”. In the Field Map, keep all the

fields from OIT_PLSS_QQ. Join Operation > JOIN_ONE_TO_ONE. Select ‘INTERSECT’.

Ensure Maximum Merge rule is selected for all speed fields (i.e. MaxAdDown). Run Process.

A polygon shape file with PLSS quarter quarter sections will be the export; these should be

named based on their TransTech, i.e. “plssTT10”, “plssTT50”, etc.

You will need to repeat this Spatial Join process for each TT

Merge all resulting spatial join shapefiles into single feature class titled “plss”.

Save output file in Working.gdb.

Load into “plss” template in [ProvAlias].gdb

Run Staging and QC tools

Verify Appropriate Fields Exist as seen below with proper value:

PROVALIAS FRN OIT_ID TRANSTECH MAXADDOWN MAXADUP TYPICDOWN TYPICUP MAXSUBDOWN MAXSUBUP

(String) 50 (Sring)

10 Long Short Double Double Double Double Double Double

PROVNAME DBANAME PRICE CONFIDENCE PROVIDER_TYPE ENDUSERCAT PLSSID REFGRIDNO WHO WHEN

(String)

200

(Sring)

200 Double Short Short (String) 1

(String)

16 (String) 20

(String)

10 Date

4.2.1.2. Census Block ID’s

Create separate tables for each Transmission Technology (some Census Blocks may have multiple

types or Transmission Technology available)

State of Colorado



For each TransTech table:

(1) Join Subscriber DBF table to 2010_Blocks_WGS84.shp based on the 15 digit Census Block

ID (Note: if the Census Block ID field of the DBF table is the wrong format, the Join will not

work)

(2) The Census Blocks layer should now only include matching records. If so, export Census

Blocks as a new feature class in the .gdb: “TT##_Census”

(3) Do Spatial Join(s) with the OIT_PLSS_QQ. The Target Feature is the OIT_PLSS_QQ and the

Join is the “TT##_Census”. UNCHECK “Keep All Target Features”. In the Field Map, keep all

the fields from OIT_PLSS_QQ. Join Operation > JOIN_ONE_TO_ONE. Select ‘INTERSECT’.

Ensure Maximum Merge rule is selected for all speed fields (i.e. MaxAdDown). Run Process.

A polygon shape file with PLSS quarter quarter sections will be the export; these should be

named based on their TransTech, i.e. “plssTT10”, “plssTT50”, etc.

(4) Merge all resulting “plssTT##” feature classes into single “plss” feature class. Save output

file to Working.gdb.

(5) Use OIT address point data AND imagery to identify quarter quarter sections in the resulting

plss feature class which don’t have homes and/or are undeveloped. Save edits. Load result

into “plss” template in respective [ProvAlias].gdb.

(6) Run Staging and QC tools

FULLFIPSID Incomplete

State of Colorado



In some instances, a spreadsheet/dBase is submitted with an incomplete FULLFIPSID. The

FULLFIPSID is a 15 digit numerical code created by the combination of the Colorado State Code

(08), County Code (3 Digits), Tract Code (6 Digits), and Block Code (4 Digits). Typically, the data

may only have up to the TRACT Code, thus only the Block codes are missing. The Provider is

essentially claiming the coverage consists of ALL the blocks within the Tracts.

Data Processing

(1) Add the processed plss feature class from the previous delivery cycle, the provider

submitted tract spreadsheet/dBASE, and a copy of Tracts_2010_WGS84.shp.

(2) Make sure the fields in both provider dBase and tracts feature class use the same

amount of digits. Sometimes providers will omit the ‘0’ in the state code ‘08’.

(3) Do an attribute join of the dBase to the tract data based on the full tract ID. This

may need to be concatenated in the dBase table as providers may or may not submit

it as three different fields. Keep only matching and export the resulting tract join into

a separate feature class “ProviderTracts” in your Working.gdb.

(4) Add “ProviderTracts”. Select by location all provider submitted tracts which intersect

with previous plss coverage. Export these tracts into separate feature class in

Working.gdb titled “PreviousTracts”. Reverse selection. Export into separate feature

class in Working.gdb titled “NewTracts”.

(5) Create a copy of previous processed plss feature class in Working.gdb and delete all

fields except OIT_ID, PLSS_ID, and REFGRIDNO and call it “PreviousPLSSEdit”.

(6) Separate both “PreviousTracts” and “NewTracts” by TransTech and label them

“PreviousTractsTT##” and “NewTractsTT##” respectively for each TransTech and

save them in the Working.gdb.

(7) For each “PreviousTractsTT##” you will run a spatial join with “PreviousPLSSEdit”

configured as follows –

Target – PreviousPLSSEdit

Join – PreviousTractsTT##

One to One

Keep All Target Features = No

Maximum Merge Rule on All Speed Fields

INTERSECT

Output – “previous_plss_TT##”

(8) Merge all resulting “previous_plss_TT##” into single feature class “PriorPLSS”

(9) For each “NewTractsTT##” you will run a spatial join with “OIT_PLSS_QQ”

configured as follows -

Target – OIT_PLSS_QQ

Join – NewTractsTT##

One to One



INTERSECT

State of Colorado



Output – “new_plss_TT##”

(10) Merge all resulting “new_plss_TT##” into single feature class “NewPLSS”

(11) Use OIT address point data AND imagery to identify and delete quarter quarter

sections in “NewPLSS” which don’t have homes and/or are undeveloped.

(12) Merge edited “NewPLSS” and “PriorPLSS” into feature class “plss” in Working.gdb

and make sure all required fields are populated, including confidence value of 1 for

tract level providers. Load into plss template in [ProvAlias].gdb


This process allows us to limit the over representation that results from use of tract level data to

some degree. It is still very limited in accuracy due to the nature of the data from the provider,

thus the assignment of confidence 1 (the lowest).

4.2.2. Georeferenced Image

Georeferencing

Please see Appendix 13.1.2 Georeferencing

**Considered a General GIS Skill

Digitizing

Please see Appendix 13.1.3 Digitizing

**Considered a General GIS Skill

Extract PLSS QQ Sections

After the Image has been georeferenced and the coverage area digitized, take the

coverage shapefile and do a ‘Select by Location’ with the:

OIT_PLSS_QQ

Create a layer with the selected quarter quarters> rename > export to Working.gdb

Use OIT address point data AND imagery to identify and delete quarter quarter sections

in the resulting plss feature class which don’t have homes and/or are undeveloped. Save

edits. Load result into “plss” template in respective [ProvAlias].gdb.

[ProvAlias].gdb

Run Staging and QC tools

4.2.3. Shapefile

FULLFIPSID Available

If the Provider submits a census.shp with FULLFIPSID and pertinent speed/transtech information,

then you will need to use part of the Census ID process outlined above. Separate into individual

feature classes based on TransTech, then being with Step 3 in the Census ID process.

FULLFIPSID Incomplete

State of Colorado



In some instances, a census.shp is submitted with an incomplete FULLFIPSID. The FULLFIPSID is a

15 digit numerical code created by the combination of the Colorado State Code (08), County Code

(3 Digits), Tract Code (6 Digits), and Block Code (4 Digits). Typically, the data may only have up to

the TRACT Code, thus only the Block codes are missing. The Provider is essentially claiming the

coverage consists of ALL the blocks within the Tracts.

Data Processing

(1) Add the processed plss feature class from the previous delivery cycle and the provider

submitted tract shapefile.

(2) Select by location all provider submitted tracts which intersect with previous plss coverage.

Export these tracts into separate feature class in Working.gdb titled “PreviousTracts”.

Reverse selection. Export into separate feature class in Working.gdb titled “NewTracts”.

(3) Create a copy of previous processed plss feature class in Working.gdb and delete all fields

except OIT_ID, PLSS_ID, and REFGRIDNO and call it “PreviousPLSSEdit”.

(4) Separate both “PreviousTracts” and “NewTracts” by TransTech and label them

“PreviousTractsTT##” and “NewTractsTT##” respectively for each TransTech and save

them in the Working.gdb.

(5) For each “PreviousTractsTT##” you will run a spatial join with “PreviousPLSSEdit”

configured as follows –

Target – PreviousPLSSEdit

Join – PreviousTractsTT##

One to One



INTERSECT

Output – “previous_plss_TT##”

(6) Merge all resulting “previous_plss_TT##” into single feature class “PriorPLSS”

(7) For each “NewTractsTT##” you will run a spatial join with “OIT_PLSS_QQ” configured as

follows -

Target – OIT_PLSS_QQ

Join – NewTractsTT##

One to One



INTERSECT

Output – “new_plss_TT##”

(8) Merge all resulting “new_plss_TT##” into single feature class “NewPLSS”

(9) Use OIT address point data AND imagery to identify and delete quarter quarter sections in

“NewPLSS” which don’t have homes and/or are undeveloped.

(10) Merge edited “NewPLSS” and “PriorPLSS” into feature class “plss” in Working.gdb and

make sure all required fields are populated, including confidence value of 1 for tract level

providers. Load into plss template in [ProvAlias].gdb

State of Colorado




This process allows us to limit the over representation that results from use of tract level data to

some degree. It is still very limited in accuracy due to the nature of the data from the provider,

thus the assignment of confidence 1 (the lowest).

4.2.4. CensusBlock/Roads Tool

A tool has been created to take any type of coverage feature class (point, line, polygon) and

extract the necessary Census Blocks and Roads. The outputs should be put in your working.gdb. Outputs include:

finalCensus

finalRoads

Intermediary layers (i.e. "final_CB_TT10", "final_CB_TT20", etc.) will also be saved in the

workspace. Other temporary files are deleted.

Take the census block and road outputs and run them through the staging tool. The final outputs

should be exported to your [ProvAlias].gdb.

Tool Location:

…\BroadbandProductionArea\Workspaces\Tools\ BroadbandProductionTools.tbx\CustomTools\CensusRoads

4.3. Tool Description Found: Functioning Tools

12.1.4: CensusRoads.py

PLSS Analysis

Once provider submitted raw data has been processed into PLSS quarter quarter (QQ) section

format, it is important to analyze for overrepresentation of coverage. Many quarter quarter

sections may be included as a result of processing from raw data, but not all of them may cover

areas of Colorado which feature development, residential homes, or commercial buildings. For

example, a QQ may feature 3 residential homes within it, while another may be entirely comprised

of cornfields or crops. By OIT standard, a QQ such as the one with crops is not considered served.

The steps below will explain the process of identifying such QQs and eliminating them from

provider landline coverage.

(1) Open ArcMap and add the provider PLSS features, the Imagery basemap using the “Add

Basemap” function, and OIT’s address point feature class which is located at -

P:\SBDDGP-ARRA Project\BroadbandProductionArea\Workspaces\BaseData\base_data.gdb\ColoradoAddresses

(2) The OIT address dataset, while large, is still incomplete, therefore it cannot be the only

form of verification. However, it should be used first, THEN analyze via imagery.

(3) Select by location all provider PLSS QQs which intersect with the address points. Export this

selection as “WithAddresses”, then reverse the selection and export as “WithoutAddresses”.

(4) Add “WithoutAddresses” feature class and edit symbology so that the polygon fill is ‘No

Color’ and the outline is a color of your choosing. This will allow you to see the imagery

inside each QQ.

State of Colorado



(5) Start an edit session for “WithoutAddresses”. Ideally the QQs should be analyzed zoomed

into imagery at a 1:10,000 scale, meaning anything you cannot clearly distinguish as

housing or development at that scale can be ignored. Notable development and housing

appears clearly at 1:10,000 so the assumption is that the QQ is undeveloped if nothing can

be seen at said scale. The image below is at 1:10,000 scale and shows examples of both

undeveloped and developed QQs. The green circles in the image indicate instances of

development/residence. Notice that QQs 1, 2, 3, 7, 11, and 12 show no evidence of

structural development, while in QQs 4, 5, 6, 8, 9, and 10, presence of buildings is obvious

right away.

State of Colorado



(6) Removal of undeveloped QQs - create an integer field in “WithoutAddresses”, then mark

each QQ as a 0 or 1 depending on whether or not they have development/homes. Using 0

or 1 also allows you to color code them based on values in that field, allowing you to see

which ones have and have not been verified. This can be done by setting a different color in

Symbology for “Null”, “0”, and “1”. DO NOT delete any undeveloped QQs yet. Once

finished with imagery analysis, save edits and exit editing session.

State of Colorado



(7) Select all the QQs with no development in “WithoutAddresses” and export into a new

feature class “[ProvAlias]PLSSRemoved” to be saved in the Working.gdb for that provider.

Once the undeveloped QQs have been exported, you can start an edit session on

“WithoutAddresses”, select all undeveloped QQs, and delete them. Save edits and close edit

session.

(8) Merge edited “WithoutAddresses” and “WithAddresses” into single feature class. This will be

your final plss feature class for a provider, overrepresentation anaylsis inclusive.

State of Colorado



5. Wireless

5.1. Introduction

Wireless Data is the second most common type of data output. Wireless data is processed in three

ways: Wireless tower location in Radio Mobile, Wireless Polygon Coverage, and midmile points. If

data consists of wireless tower locations, as described below, Radio Mobile is used. If the data is

submitted as a polygon, whether a shapefile or IMG, the data is processed in ArcMap. For all

situations involving wireless point data, the points are processed as midmiles. When point data is

provided, two types of coverage outputs will exist: wireless polygon coverage and middle mile

point coverage. IMPORTANT: When using Radio Mobile, recognize the last steps of the process use

ArcMap, where polygon coverages are created. The data processing will need to follow the steps

outlined in Polygon Coverage: 6.2.1.1 in order to finish the wireless output.

The Table below shows the necessary fields required by OIT:

Wireless



PROVNAME Provider Name Access database Unique to each provider. A complete list is stored in the Lookup.gdb along with other base data



FRN FCC Registration Number Lookup.gdb Unique to each provider. A complete list is stored in


TRANSTECH Transmission Technology Raw data From provider or derived from provider’s website

SPECTRUM Licensed or Unlicensed

Frequencies used to transmit

Raw Data From provider ONLY

MAXADDOWN Max Advertised Download Speed Raw data From provider or derived from provider’s website

MAXADUP Max Advertised Up Speed Raw data From provider or derived from provider’s website

TYPICDOWN Typical Download Speed Raw data From provider ONLY - - or “Null”

TYPICUP Typical Upload Speed Raw data From provider ONLY - - or “Null”

MAXSUBDOWN Subscriber Download Speed Raw data From provider ONLY - - or “Null”

MAXSUBUP Subscriber Upload Speed Raw data From provider ONLY - - or “Null”

PRICE Price of Maximum Advertised

Speed Package at Location

Raw data From provider or derived from provider’s website

CONFIDENCE Provider Data Confidence Rank Staging Assigned by analyst

ENDUSERCAT End User Category Raw data From provider or derived from provider’s website CAT

1, 2, or 5

STATEABBR State Raw data “CO”

Wireless Data Processing Flow Chart

State of Colorado




5.2.1. Digital Coverage

5.2.1.1. Polygon Coverage

Digitized Image

Ensure all required attributes are present in raw data

Load coverage into template> populate required attributes > Dissolve by TRANSTECH,

SPECTRUM, SPEED.

Run StagingTool

State of Colorado



Shapefile

Ensure all required attributes are present in raw data

Load coverage into template> populate required attributes > Dissolve by TRANSTECH,

SPECTRUM, SPEED.

Run StagingTool

5.2.1.2. Subscriber Point Data

dBASE Table> Geocode (Appendix: Geocoding) or Make XY Event Layer

(Latitude/Longitude)

Buffer 150’>Dissolve by TRANSTECH, Spectrum, and Speeds

Load into wireless.shp template from StagingArea>

Run Staging Tool

5.2.1.3. Tower Point Data

dBASE Table> Geocode (Appendix: Geocoding) or Make XY Event Layer

(Latitude/Longitude)

Point.shp>Run Middle Mile data processing for first output

AND

Proceed to Radio Mobile data processing

5.2.2. Radio Mobile

5.2.2.1. Introduction to Radio Mobile

Radio Mobile is used from tabular data specifying wireless towers.

Radio Mobile is written and maintained by Roger Coudé, VE2DBE. The website is:

http://www.cplus.org/rmw/english1.html

Tutorials and reference websites are listed below:

http://www.g3tvu.co.uk/Radio_Mobile.htm

http://radiomobile.pe1mew.nl/

http://www.pizon.org/radio-mobile-tutorial/index.html

http://www.ve2dbe.com/getting_started.html

http://groups.yahoo.com/group/Radio_Mobile_Deluxe/

5.2.2.2. Installing Radio Mobile

Although the software can be installed directly from the Radio Mobile website, the process is

complicated and lacks straightforward instructions.

Recommend: Use the Radio_Mobile_Setup.zip and corresponding instructions found at:

http://www.g3tvu.co.uk/Quick_Start.htm

Download Radio_Mobile_Setup.zip. Unzip the folder. Double click

“Radio_Mobile_Setup.exe”

Accept all of the defaults and click “next” through all of the installer panes. Be sure

to click “View Map_Link.txt” on the 6th screen.

Preparing Raw Data for Radio Mobile

(1) First divide tower data into Networks: A Network contains a unique frequency. For

each frequency value found in tower data, create one Network. For example, some



http://www.g3tvu.co.uk/Radio_Mobile.htm

http://radiomobile.pe1mew.nl/

http://www.pizon.org/radio-mobile-tutorial/index.html

http://www.ve2dbe.com/getting_started.html

http://groups.yahoo.com/group/Radio_Mobile_Deluxe/

http://www.g3tvu.co.uk/Quick_Start.htm

State of Colorado



towers may have 2.4 GHz and some with 5.8 GHz. Divide Networks by the Frequency

(GHz, MHz), and group by ranges (5.2 GHz, 900 MHz, etc).

Now, divide the Networks by Systems. A System contains the following unique

information:

Transmit Power

Receiver threshold

Line Loss

Antenna type

Antenna gain

Additional cable loss

(2) A single system will need to be created for each Antenna type.

Antenna Types:

- Omni Directional – the easiest and most common (has 360 Azimuth; aka default)

- Sectorized – the sectorized antenna is mapped based on the sector type.

Sectorized refers to any Azimuth not Omni (90, 120, 310, 115, etc). Some

grouping is allowed, within reason, based on ranges if too many variations exist.

An example: All the Omni systems are derived from Antenna Azimuth with 360 degree

or called OMNI in the raw data. So a network with 900MHz and System with Omni can

be identified. Similarly, a Network with 900 MHz and Azimuth of 90 can be identified

(3) Finally, name each Network/System group by the pertinent information . The naming

convention helps in Radio Mobile to know which group consists of the same Network and

within the same System. Example: TowerName_900MHZ_OMNI or

TowerName_900MHZ_90. The actual tower name submitted by the Provider is not

important because we are just gathering coverage as a whole; the networks and

systems groups may be dissolved together in the final output. The network and system

division is so Radio Mobile knows how to draw the coverage base on the

aforementioned.

Plot data in ArcMap for future steps and save in the same point.shp.

Setting up the Program in Radio Mobile

Open the program. Click Options on the toolbar and choose Internet. In the Internet

Options dialog box, click SRTM and choose “Download from Internet if a file is not found on

local path and keep a local copy”, as shown below:

State of Colorado



Do the same for “Land Cover”, “OpenStreetMap”, “Terraserver”, and “Toporama”

‘Landcover’> Internet ftp directory> Site 1

‘OpenStreetMap’ accept defaults

‘Terraserver’ accept defaults

‘Toporama’ accept defaults

Click ‘OK’

5.2.2.3. Processing Provider Data

Step 1 – Create Base Map

Select File > Map Properties > a Window will Open (shown below); values will need to

be calculated. Do not just use default values in window!

Find the center and size of basemap by referencing point.shp in ArcMap:

o Load point feature class into ArcMap

o To find ‘Center’> Open ‘Layer Properties’ of point.shp > ‘Source’ tab > Has ‘Extent’

of the data. Take the average of the Top/Bottom and Left/Right to determine the

center point.

State of Colorado



o Use the ‘Measure’ tool to determine the height and width of the map> measure from

the most northern point to most southern point, add a buffer (roughly 50 km) to

account for the beam radius of the towers and where the plots will be drawn.

o Enter the calculated Center of the map into the ‘Centre’ Latitude and Longitude

windows as shown below.

o Enter the calculated height (km) in the ‘Size(km)’> Height(km) window

o Make sure the Properties of basemap window is populated accordingly as described

in the image below> click ‘Extract’

Be careful when changing the ‘Size [pixel]’ – if the value is set too high, the program

may not be able to download the elevation data.

Populate ‘Draw mode’ window as shown below:

State of Colorado



Click ‘Draw’

Select to ‘Keep in actual picture’ > OK

Step 2 – Load Tower Locations

Tower locations can be entered manually, but a text file or KML can be imported to supply

the location data if loads of towers exist.

*.txt or *.xlsx or *.csv: Should be Tab Delimited and include the following. Fill out the

highlighted fields and leave the others as default:

Unit name

Enabled Latitude(°) Longitude(°) Elevation(m) Icon Forecolor Style Backcolor Text

Name 1 1 38.1234 -102.1111 1085.5 1 FFFFFF 0 0

Name 2 1 38.1234 -102.2222 1005.0 1 FFFFFF 0 0

State of Colorado



KML: Can load the KML directly into Radio Mobile. However, some KML files will not load.

If an issue arises, use the following steps to create a new KML, and load into Radio Mobile

again.

o Open the KML in Google Earth

o Save each tower location to “My Places” (you can create a new folder under the ‘My

Places’ directory)

o Right-click on the folder > Save Place As… > name the file and choose KML (not

KMZ)

Load Data

Select File > Unit Properties

An extra “Mobile” location is needed in order to create the coverage plot. A center-point

can be created to represent the ‘Mobile’ tower, unless specific information is available

regarding a mobile unit.

Mobile Coordinates using central point:

State of Colorado



To manually enter data, click “Enter LAT LONG or QRA”

To import data, click “Import”:

o Go to File > Load > navigate to tower file

o The tower locations should show up in the column on the right. Choose towers to

import and where to start the overwrite (existing units can be kept or overwritten).

o When finished, click OK

o If the elevation does not match the specific elevation files within the Radio Mobile

software or elevation data is not available> choose each unit location individually>

click “Enter LAT LONG or QRA” and then click OK. The Elevation field should

automatically update.

Step 3 – Define Networks, Systems and Memberships

Select File > Network Properties

Networks:

State of Colorado



As described above, contains a unique frequency. For each frequency value found in tower

data, create one Network. For example, some towers may have 2.4 GHz and some with 5.8

GHz.

For each Network, fill in:

Net Name

Minimum Frequency (MHz)

Maximum Frequency (MHz)

Climate – choose “Continental temperate”

Systems:

A System contains the following unique information:

Transmit Power

Receiver threshold

Line Loss

Antenna type

Antenna gain

Additional cable loss

You will need to create a single system for each unique combination of these attributes.

Note that Antenna Height is also listed in the System attributes, however this can be

changed for individual Units in the Membership section.

Click on the Systems Tab

To create a new system, use the “Base” system as a starting point. Fill in any

information that you have from the provider and leave the other fields as their default

values.

State of Colorado



Antenna Type:

- Omni Directional – this is the easiest and most common – just choose omni.ant from the

dropdown list.

- Sectorized – the sectorized antenna is mapped based on the sector type. Work with

your provider to get the correct antenna pattern. Antenna patterns can be downloaded

You will enter the Azimuth (i.e. the direction of the antenna) in the Membership section

Memberships

The Memberships section defines which Network and System each Unit belongs to:

Click on the Membership tab

Choose a Network in the column on the left, and click the checkboxes for all Units that

belong to that Network. A Unit can belong to multiple Networks, but the attributes for

that Unit will need to be changed for each separate Membership

For each Unit, enter the following attributes:

Role (default is Command)

System

Antenna Height (the System height is defined under the Systems tab, but you can

enter heights for individual Units here)

Antenna Direction – Enter the azimuth of the antenna; this is only valid for

directional antenna patterns (i.e. not Omni)

State of Colorado



Add the Mobile Unit to all Networks

Step 4 – Create Coverage Map

Select File > Picture Properties. Choose “White” as the Draw mode and click “Draw”. This

helps create a clean export picture.

Select Tools > Radio Coverage > Combined Cartesian

Select all Units in your data as “Fixed units” in the column on the left.

Select “Mobile” as the Mobile Unit

Select a Network – you will need to do a different coverage plot for each Network. The

coverage will only draw members of the chosen Network that are checked in the left-

hand column, it will ignore Units in other Networks.

Enter the Maximum Range (this is the Beam Radius found in the Provider data – make

sure to convert to km).

State of Colorado



Set the Draw size (pixels). You can experiment with this number based on your

coverage area. The default is 5, but smaller numbers will give a more detailed plot (with

longer processing time).

Set the Signal Range to “S-Unit” and the “From (>=)” value to 5

Check the “Rainbow” box

Click “Draw”

Again, you will be prompted:

Always save coverage groups separately and process separately in ArcMap!

Your output should look similar to the image below. Select File > Save Picture As…>

choose location and save as TIF.

Check the box for “White is Transparent” and click OK.

State of Colorado



NOTE: Remember to export each Network and System separately before processing in ArcMap

Step 5 – Georeference the Image

Georeference the image in ArcMap requires a shapefile with control points. Use the

Latitude and Longitude points in the basemap Properties window in Radio Mobile, shown

below, as reference points to create control points. Four corner points are needed. Each

corner point corresponds with a reference points from the basemap Properties

NOTE: The basemap properties gives four coordinates in Degrees-Minutes-Seconds; the points will need to be converted into Decimal Degrees

Convert coordinates to Decimal Degrees and map them in ArcMap.

o Put converted Latitude and Longitude in excel spreadsheet

o Convert to dBASE Table

o Make XY Event to create a point.shp of the Control Points

State of Colorado



o The controlPoints.shp can be used to georeference all network/system TIFFS

exported from Radio Mobile

Import Raster TIFFs into ArcMap

Use these as control points to match with the corners of the image. Refer to the Image

Georeferencing instructions in APPENDIX: Georeferencing.

When the TIFF is added to ArcMap, the colors may not look right, as shown below –

Render symbology in Colormap, instead of RGB.

Figure 1: RGB symbology is displayed on the left; whereas, Colormap symbology is rendered on the

right

Step 6 – Convert to Shapefile

Use the ‘Raster to Polygon’ tool to convert the image to a shapefile. The shapefile will need

to be cleaned up after conversion.

Open the Raster to Polygon tool and fill in the dialog box as follows:

o Input Raster: TIF image. This often works best if you use one band from the TIF

image. Preview each band in ArcCatalog to decide which one is best.

o Field: Value

o Output: choose a location for the feature class

The output polygon feature class needs to be edited to delete all of the extraneous data

(remember the image includes a legend).

o Add the shapefile to ArcMap and Start Editing

o Select all of the features that make up the Legend and the border, and delete

them from the shapefile.

State of Colorado



Step 7 – Final coverage plot

Once the extraneous features are gone, 5-10 GRIDCODE values should remain in the

attribute table.

From here, choose appropriate signal levels to keep in the final coverage – generally,

the first 3 or 4 tiers (i.e. about 15 S-Unit and above) will be kept.

Delete the unwanted data by ‘selecting by attribute’ ‘GRIDCODE value’ and deleting

features.

Merge remaining features: Select all desired features> Click Editor > Merge

If you have multiple networks, you will need to combine the resulting shapefiles

REFER TO FINAL STEPS IN WIRELESS POLYGON PROCESSING Section 6.2.1.1 up above.

5.3. Wireless Analysis

Analysis of provider wireless coverage is limited in our ability to edit it, however it is still useful in

deciding what confidence value to assign it, as well as identifying any issues worth discussing with

the provider. For this process you will need the OIT wireless format result of processing provider

data, processed provider middle mile result if available, OIT’s address point feature class, and the

Imagery basemap. Add these to ArcMap, then change the symbology for the wireless features to

have ‘No Color’ under the fill category.

Things to look for –

Do all the middle miles fall within the wireless coverage area?

How much of the coverage area is located over undeveloped land, inclement terrain?

Are the address point locations evenly dispersed across the coverage area or localized?

Due to incomplete address point data, are developed/residential areas dispersed across the

coverage area or local to a few spots?

Does the wireless coverage coincide with terrain patterns?

Below is an example of a wireless provider, in this case AlignTec.

State of Colorado



6. Middle Mile

6.1. Introduction

Middle Mile



PROVNAME Provider Name Access database Unique to each provider. A complete list is stored in




FRN FCC Registration Number Lookup.gdb Unique to each provider. A complete list is stored in the Lookup.gdb along with other base data

OWNERSHIP Transmission Technology Raw data From provider or derived from provider’s website

BHCAPACITY Licensed or Unlicensed

Frequencies used to transmit

Raw Data From provider ONLY

BHTYPE Max Advertised Download Speed Raw data From provider or derived from provider’s website

LATITUDE Max Advertised Up Speed Raw data From provider or derived from provider’s website

LONGITUDE Typical Download Speed Raw data From provider ONLY - - or “Null”

ELEVFEET Typical Upload Speed Raw data From provider ONLY - - or “Null”

STATEABBR Subscriber Download Speed Raw data From provider ONLY - - or “Null”

CONFIDENCE Subscriber Upload Speed Raw data From provider ONLY - - or “Null”


6.2.1. dBASE Table

6.2.1.1. Address Data

Geocode Points if address information is present. Procedure described in APPENDIX:

Geocoding. Middle Mile requires 100% Match Rate. Once geocoding is complete, move on to

the Point Shapefile data processing instructions down below.

State of Colorado



6.2.1.2. Latitude and Longitude Data

‘Make XY Event’ ArcGIS tool if Latitude and Longitude are present. Once a point shapefile

has been created, move on to the Point Shapefile data processing instructions down below.

6.2.2. Point Shapefile

6.2.2.1. Middle Mile Points

The raw data should consist of the fields required for data processing, highlighted in yellow

in the table above. If the information is not present, contact Provider immediately to gather

data.

Deriving Data from Known Fields

Latitude/Longitude (if not present in point data)> use ‘Add XY Coordinates’ tool in

ArcMap>input Point Feature Class.

ELEVFEET> Cannot be more than a few hundred feet above the ground. If the value is in

the thousands, chances are the measurements were from ‘sea level’, thus calculate proper

elevation from known elevation point (i.e.: city of Denver is 5,280 ft above sea level). If

blank, put ‘NULL’ value. If outside of a reasonable elevation, contact Provider.

IMPORTANT: Here is the opportunity to check the ‘Properties’ of the fields to ensure they

load into the Staging Template easily! Right click field Name> select ‘Properties’ >

Examples:

OWNERSHIP Short integer

BHCAPACITY Short integer

BHTYPE Short integer

AND

LATITUDE Double

LONGITUDE Double

Click> OK

State of Colorado



Load into StagingTemplate> run StagingTool

6.2.2.2. Wireless Tower Points

When wireless tower information is available, we run the tower information in Radio Mobile

for the wireless coverage. In addition to wireless coverage, the wireless towers are

processed like middle miles.

Deriving Data from Known Fields

OWNERSHIP Derive from Company

BHCAPACITY Determine from Speeds

BHTYPE Derive from Raw Data

Latitude/Longitude (if not present in point data)> use ‘Add XY Coordinates’ tool in


ELEVFEET> Cannot be more than a few hundred feet above the ground. If the value is in

the thousands, chances are the measurements were from ‘sea level’, thus calculate proper

State of Colorado



elevation from known elevation point (i.e.: city of Denver is 5,280 ft above sea level). If

blank, put ‘NULL’ value. If outside of a reasonable elevation, contact Provider.



Examples:




AND

LATITUDE Double

LONGITUDE Double

Click> OK


6.2.3. Image

6.2.3.1. Georeference

If an image is present with middle mile points, georeferencing the image is required.

Georeferencing is described in the APPENDIX: Georeferencing

6.2.3.2. Digitize

Once an image is georeferenced, digitize the midmile point locations by creating a point

shapefile. Digitizing is described in the APPENDIX: Digitizing.

State of Colorado



6.2.3.3. Populating Required Fields

Contact Provider if information is not available in correspondence or raw data:

OWNERSHIP

BHCAPACITY

BHTYPE

6.2.3.4. Deriving Data from Known Fields

Latitude/Longitude (if not present in raw data)> use ‘Add XY Coordinates’ tool in




Examples:




AND

LATITUDE Double

LONGITUDE Double

Click> OK

State of Colorado




6.2.4. Middle Mile Python Tool

Kass Rezagholi PYTHON TRAINING

7. Staging Area

7.1. Staging Template

The template consists of a File Geodatabase with plss, midmile, and wireless. Each feature

class has the proper field names, alias, data type, and value length required by OIT

standard.

State of Colorado



7.2. Loading Data into Template

(1) Copy StagingTemplate.gdb to your local folder for the provider. The

StagingTemplate.gdb should be copied into the “Processed” subfolder. Each provider

folder should be structured as seen here -

(2) Delete the feature class templates that are not applicable to the Provider data. (i.e. if

the provider has only wireless coverage, the plss, and midmile can be deleted).

(3) Rename the geodatabase to [ProvAlias] and rename the feature classes to “plss”,

“midmile”, and “wireless”. THE GEODATABASE MUST FOLLOW THIS NAMING

CONVENTION IN ORDER FOR THE FINAL LOAD TOOL TO WORK!

(4) Select a Feature Class template, ‘Right Click’ > Load > Load data.

(5) A window form will appear and prompt you to click ‘Next’. In the future, this screen can

be eliminated from the process by ‘checking’ the box.

State of Colorado



(6) Click the Folder Path button under “Input Data” and navigate to the appropriate feature

class within your working.gdb for that Provider. For example, if you are loading data into

the plss feature class, then navigate to the plss feature class that you created in the

working.gdb.

Click ‘Add’. The source data will now appear in the List of Source Data to Load

window. Click ‘Next’.

(7) In the next window, make sure the I do not want to load all features into a subtype

radio button is selected, click ‘Next’

(8) Match the corresponding Fields between the Target Field and the Matching Source

Field. Click ‘Next’

***Note: Not all the Target Fields will have a field to match to in the Matching Source Field

(9) Load all of the Source Data should be selected, click ‘Next’. A Summary will be

displayed, click ‘Finish’.

State of Colorado



(10) The data will now ‘Load’ into the template. Open the attribute table in ArcCatalog to

confirm data loaded correctly.

(11) Repeat steps 4 through 9 for each feature class.

7.2.1. Calculate Fields

7.2.1.1. Census Blocks

(1) Provider Type and ENDUSERCAT

Calculate Manually

PROVIDER_TYPE ENDUSERCAT

Provider Type End User Category

Short Integer String

2

See lookup table for values

(2) PROVNAME and DBNAME

Calculate from PROVALIAS

PROVNAME DBNAME

Provider Name DBA Name

String String

200 200

Lookup Table Lookup Table

State of Colorado



7.2.1.2. Wireless

(1) PROVNAME, DBNAME, and STATEABBR


PROVNAME DBNAME STATEABBR

Provider Name DBA Name State_Abbreviation

String String String

200 200 2

Lookup Table Lookup Table ='CO'

7.2.1.3. Middle Mile (1) PROVNAME, DBNAME, STATEABBR


PROVNAME DBNAME STATEABBR

Provider Name DBA Name State_Abbreviation

String String String

200 200 2

Lookup Table Lookup Table ='CO'

7.3. Staging Tool

The Staging Tool requires three separate inputs: the Lookup Table, the Staging Template, and the

Provider Feature classes. The lookup.dbf and StagingTemplate.gdb are written into the script. The

feature classes are entered when the tool is executed.

7.3.1. Lookup Table

The ‘lookup.dbf’ consists of PROVALIAS, PROVNAME, DBNAME, FRN, ENDUSERCAT, and

Provider_Type for all participating Providers. The list will need to be kept current with each delivery

cycle. The Staging Tool calls to the lookup.dbf to populate the aforementioned fields once

appended to the staging template feature class output.

Lookup Table Location:

BroadbandProductionArea\Workspaces\BaseData\lookup.gdb\lookup

Lookup Table Properties:

State of Colorado



7.3.2. Staging Template

The Staging Template is a file geodatabase with three empty feature classes as described in 9.1

Staging Template. The purpose of the template is to load Provider feature classes into standardized

feature classes with the same corresponding output. The template feature classes may need to be

updated for every delivery cycle, depending on NTIA changes in requirements. The staging

template is called in the script when an append is needed. The data is then loaded into the

specified output feature class.

7.3.3. Staging Tool

Purpose

The Staging Tool is used to automate the process of moving and standardizing Provider feature

classes post processing. The Staging Tool is located:

BroadbandProductionArea\Workspaces\Tools\BroadbandProductionTools.tbx\CustomTools\StagingTool

State of Colorado



Staging Tool Execution

Folder Location > Navigate to current delivery cycle Staging Area

myGDB > name corresponding [PROVALIAS] in lowercase (See examples below):

The following (optional) inputs> Navigate to all the necessary feature classes

analystName>choose username from dropdown list

Click ‘OK’

Verify attributes populated correctly in the staging area [PROVALIAS].gdb

If not, document issue and populate necessary attribution

State of Colorado



8. OIT Final Format

Every data delivery, OIT produces a new Data Submission zipfile with changes or corrections. Some

of the changes reflect updated speeds, technology types, etc. Corrections typically impact the

template.gdb or the error check python script. In either case, when processing, data from the

Staging Area needs to be combined and formatted to meet the OIT’s current data submission

standards. Thus, instead of individual file geodatabases for each Provider, all plss, midmile, and

wireless feature classes need to be combined by data output and placed in the OITtemplate.gdb

8.1. Data Submission Package (*.zip)

A data submission package consists of the following

2015_October_FinalProduct (Year and month will change with delivery cycle)

Submission_tools_YYYY folder

o CheckSubmission_ChangeNotes.doc

o CheckSubmission_ChangeNotes.txt

o SBDD_Check.tbx

o SBDD_CheckSubmission.py

o sbdd_checkSubmission_Readme.txt

o SBDD_Receipt_Overview_v1.xls

Data_Model_Changenotes_YYYYMMDD.txt

DataPackage_YYY_DD_MM.xlsx

ReadMe.txt

Main Folder Contents:

Submission_tools_YYYY Contents:

State of Colorado



8.2. OIT Template (OITFinalTemplate.gdb)

8.2.1.1. Load by Hand

(1) Load data from all Providers into the SQL Server Database:

OIT-GISDB-02.Broadband_Ops.sde on server 10.12.1.28

(2) Run SQL Server scripts

(3) Load data from all providers into OITFinalTemplate and CO_SBDD_MM_YYYY_template

State of Colorado



8.2.1.2. OIT Final Format Tool

Created to simplify and automate the process of loading individual file geodatabases into the final

OIT GDB.

Located:

BroadbandProductionArea\Workspaces\Tools\BroadbandProductionTools.tbx\CustomTools\NTIAFinalFormat

State of Colorado



Execution of Tool:

Navigate to Staging Area Folder>OK

Navigate to Final GDB Template>OK

Navigate to Log File Locations>OK

Finally, Click ‘OK’

Review Log File for errors with individual GDBs of Providers

Review OIT final gdb for obvious errors or missing data in fields.

State of Colorado



8.3. Error Check

8.3.1. SQL SDE Error check

Located on the Database Server: Server 19

8.4. Pricing Calculations and Tools

8.4.1. Overview

The first step regarding pricing is to verify and update our pricing lookup tables with speed and

pricing information for each delivery cycle. Speed packages and associated prices may change from

year to year so for each broadband processing cycle, the pricing lookup tables should be updated

with current speeds and prices for each provider. There are two lookup tables, one for landline

coverage and one for wireless coverage. Pricing is only recorded for coverage that offers residential

service, therefore landline and wireless coverage with ENDUSERCAT = 2 is excluded. Additionally,

we are only tracking fixed wireless pricing, so satellite and mobile coverage will also be excluded.

The lookup tables can be found here –

P:\SBDDGP-ARRA Project\BroadbandProductionArea\Workspaces\BaseData\lookup.gdb

For each tool below, it is ideal to use the same Destination Folder path as each tool relies on the

output results of the previous one, therefore it is helpful in regards to keeping everything organized

to have all results stored in one folder location. The tools can also be copied and run locally; all

four tools can be found at the following location on the P:Drive –

P:\SBDDGP-ARRA Project\BroadbandProductionArea\Workspaces\Tools\BroadbandProductionTools.tbx

8.4.2. Tools and Usage

8.4.2.1. Provider Pricing Tool

***PROVIDER PRICING TOOL EXPLANATIONS

8.4.2.2. QQ Price Average Tool

Once the Provider Pricing Tool completes, the resulting geodatabase will contain two feature

classes, “plss” and “wireless”. These will include all coverage features which meet the conditions

explained in the introduction above with available pricing data. The QQPriceAverageTool will

generate price per Megabit average values by quarter quarter section. In order to accomplish this

for the wireless features, the tool will use a spatial join to convert wireless coverage into PLSS

Quarter Quarter format. The tool uses frequency analyses in order to calculate and populate fields

appropriately with Quarter Quarter mean prices. The below figure provides an visual example of

the this process –

State of Colorado



You must specify the path for the output folder (1) where you would like your results saved and

what name you want the resulting GDB (2) to have. You will use the Landline (3) and Wireless (4)

feature class results from the Provider Pricing Tool as inputs. The figure below illustrates the tool

interface –

State of Colorado



Your resulting geodatabase upon the completion of running this tool should provide two feature

classes called LandlineFinal and WirelessFinal. Both of these will have price per megabit down

and up average values by quarter quarter section and wireless features will be in QQ format.

8.4.2.3. TT Price Average Tool

This tool will use the LandlineFinal result from the previous tool in order to generate individual

landline feature classes for each transmission technology. The tool will calculate average price per

Megabit values specific to each transmission technology and Quarter Quarter section. You will once

again need to specify the path for your output folder (1) as well as what name you wish the output

GDB to have (2). Additionally you must include the path for the location of the LandlineFinal

result (3) from the QQ Price Average Tool. The tool interface is shown below –

The final result of this tool will have multiple landline feature classes each named according to

transmission technology. The tool is only inclusive of landline coverage since the wireless features

considered in the pricing process are limited to fixed wireless coverage.

8.4.2.4. Final Product Pricing Tool

This tool acts as the final stage in pricing and generates feature classes inclusive of pricing

averages statewide. Specifically it will create feature classes of counties, urban regions, and rural

regions complete with pricing averages based on rural/urban regions, individual counties, and

transmission technologies for each county. Once again you must specify a Destination Folder (1)

and naming choice for the output GDB (2). You then need to specify the path of the GDB of

TransTech results from the TT Price Average Tool (3) followed by the Urban (4), Rural (5), and

Counties (6) feature classes; these can be found at the following locations –

State of Colorado



P:\SBDDGP-ARRA Project\BroadbandProductionArea\Workspaces\BaseData\base_data.gdb\Urban P:\SBDDGP-ARRA Project\BroadbandProductionArea\Workspaces\BaseData\base_data.gdb\Rural P:\SBDDGP-ARRA Project\BroadbandProductionArea\Workspaces\BaseData\base_data.gdb\CountiesPricing

These feature classes can also be copied locally like the tools and associated scripts. Finally you will

need to reference the paths for the LandlineFinal (7) and WirelessFinal (8) results from the QQ

Price Average Tool.

The results in the output GDB will have a CountiesFinalOutput feature class which has overall

download and upload price averages based on county for landline and wireless service, as well as

download and upload price averages by transmission technology and county. Additionally it will

have LandlineUrbanOutput and LandlineRuralOutput feature classes with price averages based on

transmission technology and region. Finally the WirelessUrbanOutput and WirelessRuralOutput

show price averages based on wireless coverage across urban and rural regions respectively.

Further documentation on calculations, logic, and workflow of pricing can be found here –

P:\SBDDGP-ARRA Project\BroadbandProductionArea\Team_Documents\Team

Documents\TeamResourceDocs\PricingMethodsUpdated.docx

State of Colorado



9. Post-Processing

9.1. Mapbooks and Validation

9.1.1. Mapbooks: Data Driven Pages

Map books will be created using the Data Driven Pages (“DDP”) toolset in ArcGIS 10. The map

books will be emailed to providers for validation and verification purposes.

Use the following steps to create both Landline and/or Wireless map books (depending on the

provider data)

***Note: You must be in Layout View in order to set up the export

(1) Make copies of the necessary mapbook templates to your workspace. The Landline

mapbook will show Census Blocks, Roads, and MidMile. The Wireless mapbook will show

Wireless and MidMile. Rename the mxd “[ProvAlias]_Landline.mxd” or

“[ProvAlias]_Wireless.mxd”

Template Location:

…\BroadbandProductionArea\Workspaces\Validation\MapBook_templates\

(2) Open the MXD, the Table of

Contents should have ! next to

the provider data layers. The

template is set up this way so that

it is easy to map to your provider

data.

(3) Right click on each layer and choose Data > Repair Data Source. Navigate to your provider

data.

State of Colorado



(4) After you map the data source for each layer, open the Properties for each layer and go to

the Symbology tab. Click on the “Count” heading in the Category table (this will show the

number of records for each category – the symbology might not register if you skip this

step)

(5) Replace “Provider Name” with your provider’s name in the title. Make sure that the Title

still fits in the allocated space, i.e. your provider name may take up 2 lines or you may

have to change the text size (easiest solution is to enter your text and then just click the

“Size and Position” tab and enter a width of 2.3 inches).

(6) If the DDP toolbar is not showing, go to Customize > Toolbars, and click on Data Driven

Pages.

(7) Click on the DDP Setup button and verify that the settings match what is shown below.

State of Colorado



(8) The DDP Toolbar lets you page through to view how each county will be exported.

However, we only need to export counties that contain provider data.

(9) Select the counties that contain provider data (this selection determines which

counties are exported to the mapbook document)

State of Colorado



- Make sure the MainMap data frame

is active and choose Selection >

Select by Location.

- Selection method: “add to the

currently selected features”

- Target Layer: counties (the top

counties layer)

- Source Layer: choose one of your

provider data layers

- Click ‘Apply’

- After hitting ‘Apply’, choose a

different provider data layer as a

Source Layer, and click ‘Apply’

again

- Repeat for all provider data layers

(10) Export the mapbook:

- Click File > Export Map…

- Navigate to your workspace and

name the file

[ProvAlias]_landline.pdf or

[ProvAlias]_wireless.pdf

- Change the output image quality

to “Normal” on the ‘General’ Tab

under Options (this will reduce

the PDF file size)

- Click the ‘Pages’ tab under

Options and click the ‘Selected’

radio button

- Export Pages As: Single PDF File

- Click ‘Save’

- Open the PDF to verify accuracy

(i.e.. all necessary counties are

included, all pages were

exported correctly)

Note: there have been some issues with the “Local Roads” layer interfering with the export. The temporary solution is to change the Transparency of that layer to 0% for the export.

(11) If the PDF export is too large to send in one email, you can make multiple copies of the

PDF, and use Adobe Acrobat Pro to delete pages from each copy accordingly

State of Colorado



9.2. OIT Data Package

The NTIA Data Package requires a SBDD_TRANSFER.gdb(Final gdb) formatted to meet the OIT

requirements, Methodology document, Process Guide Document.

9.2.1. Methodology Document

The methodology documents consists of FCC, CAI, and Drive test tables, the STAT_QA reports,

status of data collection, Analysis of change table, Summary of process, Data Summary and

feature class Stat Tables, and NTIA Assessment tables. The information is compiled and saved to a

PDF. Every delivery the tables and summaries are recorded and updated to match the current

SBDD_TRANSFER.gdb.

9.2.2. Process Guide Document

The Process Guide is a compilation of every step taken in order to process data from Providers, to

the final output. The guide consists or images and explanation, step-by-step instruction, and

outline of the process. When a process is changed, a tool is developed, or a software update

occurs, the document is updated to match the change. The Process Guide is a product of the GIS

team.

9.2.3. Additional Documents

9.2.4. Warnings and Error Readme.txt

The OIT Error Check toolbox consists of a script developed by OIT to check attribution of required

fields and limitations. The script is provided in python form and has an associated readme.txt

explaining the tool, the process, and the updates from previous submissions. Once run on the final

OIT gdb, a receipt output text file is produced listing the errors and warnings as shown below.

Errors must be fixed and the tool rerun until no errors exist. Warnings must be explained.

Run QC tool on final product.

Example of Submission Receipt

State of Colorado



9.2.5. Compile Final Package

9.3. CBDDP

9.3.1. Broadband Mapservices

Areas with broadband coverage overlap need to be identified with the number of providers for the

same type of service.

Mobile Fixed Wireless Wireline Satelitte

Save NATL_BroadbandMap locally, for the latest updates to coverage.

Buffer roads by 150 feet and merge the buffer with the census blocks

You may need to dissolve the roads and census blocks separately first and then merge

Remove individual provider overlap.

State of Colorado



There should not be any census blocks duplicates from a specific provider.

Run coverage with no individual provider overlap through SuperRegionPoly tool.

Put into proper format and load to web data.

SuperRegionPoly is a tool designed to help analyze vector-based polygon layers that contain

overlapping features. Detect and count occurrences of overlapping features. Build a planarized output layer that has no overlapping features with a spatially/tabular

unique polygon ID field. Build a one-to-many look up table that can tie the many overlapping records back to the

polygon ID field in the planarized layer. Populate a field in the planarized layer containing the max, min, sum, or mean field values

from a numeric field in the input layer.

State of Colorado



XY Tolerance:

To create an output that is exact to your input coverage then leave at .001. If you notice that

there are slivers created in the output, or you have small slivers that are irrelevant to your overall

coverage and would like to get rid of them for a better looking map then increase the tolerance.

This is not a solution for a bad input.

Decimal Tolerance:

Adjusting this to increase or decrease the accuracy of the centroid of polygons shattered during the

process. Having inaccurate centroids can cause false overlapping.

10. Future Enhancement Tools

10.1. Functional Tools

10.1.1. CalcMaxMinAddr.py(discontinued)

Purpose

Designed for TIGER Streets Feature Class, the script checks address ranges on each row,

puts them into a list, sorts the max/min range numbers, and then populates the

ADDMAX and ADDMIN address cells.

Tool Initiation

Initiated on TIGER Base Layer with each new update

State of Colorado



Tool Properties

Developed with: Python Script 2.5.1

Created By: Larry Norden

January 2012

10.1.2. CalcStreetName.py(discontinued)

Purpose

Designed for TIGER streets feature class, concatenates parsed street address fields

‘PRETYPABRV’, ‘NAME’. TIGER Standard parsed streets into 6 components; whereas,

NTIA parses street addresses into 4 components

Tool Initiation

Initiated on TIGER Base Layer with each new update

Tool Properties

Developed with: Python Script 2.5.1

Created By: Larry Norden

January 2012

10.1.3. XYCode.py (discontinued)

Purpose

Automates the process of taking Excel Data, import into .gdb as dBASE, convert DMS to DD

if necessary, and produce a point.shp as final output with all necessary fields.

Tool Initiation

Tool initiated on Raw Data and automates to GIS Format; output is spatial data.

Tool Properties

Developed with: Modelbuilder 9.3.1, Microsoft Silverlight C#: Windows Application, Python

Script 2.5.1

Created By: Windy Fischer

December 2011

10.1.4. CensusRoads.py(discontinued)

Purpose

Automates process where Census Blocks and Roads are selected from coverage shapefile.

Tool Initiation

Tool initiated on Coverage shapefile (point, line or polygon) and selects Census and Roads;

output is Census and/or Roads layer

Tool Properties

Developed with: Modelbuilder 9.3.1, Python Script 2.5.1

Created by Megan Chadwick

January 2012

State of Colorado



10.1.5. OITFinalFormat.py

Purpose

State of Colorado



Loads all gdbs in the identified staging area folder into the identified template IF the feature

classes are the same schema.

Tool Initiation

Tool initiated on Staging area folder and loads into template. Full dataset template will

need to be manually loaded into the final SBDD_TRANSFER

Tool Properties


Created by Kassrah Rezagholi

December 2014

10.1.6. StagingTool.py

Purpose

Adds and calculates all fields necessary for NTIA delivery. Calculates defaults where NULL

values are found. Populates provider information from lookup table. Moves

processed.gdbs to the Staging Area for appropriate delivery.

Adds Analyst information for WHO and WHEN

Adds “ZZ” to unpopulated TYPICs

Calculates STATE code

Parses FULLFIPSID and populates fields in census output

Populates ENDUSERCAT and Provider_Type

Uses Staging Template for standardization

Tool Initiation

Tool initiated on processed.gdb for individual Providers. Creates a provider gdb for Staging

Area

Tool Properties


Created by Windy Fischer, Megan Chadwick, and Tom McKean

August 2012

Last Revision: December 2014

10.1.7. SuperRegionPoly.py

Purpose

Analyze vector-based polygon layers that contain overlapping features.

Detect and count occurrences of overlapping features.

Build a planarized output layer that has no overlapping freatures with a

spatially/tabulary unique polygon ID field.

Build a one-to-many look up table that can tie the many overlapping records back to

the polygon ID field in the planarized layer.

Populate a field in the planarized layer containing the max, min, sum, or mean field

values from a numeric field in the input layer.

Tool Initiation

Tool initiated on a coverage shapefile (Polygon).

Tool Properties

Developed with: Python 2.5.1

Created by: Kassrah Rezagholi

Last Revision: March 2013

State of Colorado



11. APPENDIX

11.1. General GIS Skills

11.1.1. Geocoding (10.0 Instructions)

Once you have data with a standardized address in a single column of data, you can geocode the

address to create a spatial location for each data value.

(1) Load a dBase or other data table into ArcMap

(2) Right-click on the data table and select “Geocode Addresses”

(3) In the Choose an Address Locator to use… box, select the Address Locator you would like to

use to perform the Geocode.

(4) Match the Street or Intersection information with the loaded data if it is not already done

(5) Set the Output shapefile or feature class to your working.gdb and click OK

State of Colorado



(6) Once the geocoding process begins, you will see a status of how well your addresses are

adhering to the geocoder. Ideally you would want to get this number as high as possible.

Once you have finished the geocoding, you can open the attribute table and find the

address data that was unable to geocode.

An alternative Address locator from NavTeq is stored on the P: drive in:

P:\SBDDGP-ARRA Project\BroadbandProductionArea\Workspaces\BaseData\base_data.gdb

This can be used as a supplemental geocoding method for attempting to gather additional results.

(1) Copy the NavTeqCitySt address locator to your local machine

(2) When beginning the geocoding process, click on Geocode address, then click Add

State of Colorado



(3) Navigate to the Address Locator on your local machine and click Add

(4) You will see the NavTeqCitySt Address Locator listed in your predefined list of locators.

Select the locator and continue with the geocoding process as described above.

****NOTE: 10.1 requires ArcGIS Online Log In; use QGIS as alternative

11.1.2. Georeferencing

If you receive an Image File (*.jpg, *.tif, or image embedded in a *.pdf), the spatial information

must be digitized.

(1) If the file is a PDF, open in Acrobat and Save As… *.tif

(2) Open ArcMap and Add basemap data that will be used as a reference for georeferencing

If your image has County boundaries, then add the County boundary layer to use as

reference point. Or if your image has roads, then you can use the TIGER roads layer as

reference points.

Example below shows image from provider and a suitable reference layer to be used for

georeferencing.

The major road locations can be found on the image, and then matched to the same

intersection on the roads layer

State of Colorado



(3) Add your image

(4) Zoom to the approximate area of your image (this way when you ‘Fit to Display’ in the next

step, your image is in the general area that it should be).

(5) In the Georeferencing toolbar (if this isn’t showing, go to View>Toolbars>Georeferencing)

select the *.tif file from the dropdown list, and then click Fit to Display

(6) You can use the Rotate, Scale and Shift buttons to move

your image around before you start adding control points

(7) You can use the Rotate, Scale and Shift buttons to move your image around before you

start adding control points

(8) To add control points

Click the “Add Control Points” button

Click on the location on the image first (blue circles in image below)

Then click on the corresponding location on the base data (end of blue arrow in image

below)

Notes:

State of Colorado



The control points should be spread out over the extent of the image – if the points

are too close together, the outer edges of the image may warp.

You should use NO LESS THAN 3 control points, but as many as you need to achieve

accurate georeferencing. Be careful though, if you add too many points, the

image can warp!

If you would like to delete a control point, click the “View Link Table” button ,

select the control point that you would like to delete, and click the X in the top

right corner.

(9) Once you are satisfied with the placement of the image, click “Update Georeferencing”.

This will create a “world” file in the same directory as the image file (*.tfwx or *.jgw)

State of Colorado



11.1.3. Digitizing

Now that you have a georeferenced image, you can use it to map your coverage area. If your

image shows a polygon of the coverage area, follow the instructions below. If the image contains

other information

(1) In ArcCatalog, Right Click on working.gdb>New> Feature Class. Name the feature class

and choose the type according to the data you are digitizing

Example: if you are drawing a polygon of the wireless coverage area, name the file

“Wireless” and choose “Polygon Features”

(2) Add the newly created feature class to ArcMap and Start Editing. Click the Sketch tool,

select Create New Feature, and set your new feature class as the Target.

(3) Draw each point or polygon (separate polygons for every TRANSTECH and speeds combo)

as shown in your image file and add attributes manually.

11.1.4. Latitude/Longitude Coordinates

To Convert Degrees-Minutes-Seconds to Decimal Degrees, use the following formula:

Note: Do not forget the conversion formula does not account for the Longitude value being (-). After the Conversion is complete, multiply ONLY the Longitude by (-1). The adjustment needs to be documented in the Excel table. Also, if the raw data indicates the Latitudes and Longitudes are already in DMS, verify the minutes-Seconds are correctly broken into fractional parts

The following is an example of how a Midmile Lat/Long table should look:

Creating A Feature Class (Option 1):

(1) Open ArcMap and select a coordinate system

Right click on “Layers” in the Table of Contents and choose “Properties…”

Select GCS_WGS_1984 (Predefined>Geographic Coordinate Systems>World>WGS

1984)

(2) Add your dBASE table to ArcMap. Right click and choose Display XY Data.

State of Colorado



Select the correct X and Y

fields. ***NOTE: Do not confuse X

Field and Y Field as your data will not be projected properly.

Select GCS_WGS_1984

(Geographic Coordinate

Systems>World>WGS 1984)

Click OK

This creates an XY Event Layer

that will need to be exported

as a feature class

(3) kRight click on the Event Layer, choose Data > Export Data…

(4) Click the Browse Folder button and navigate to your working.gdb. Choose the Save

as type: File and Personal Geodatabase feature classes and name your file.

Creating A Feature Class (Option 2):

(1) Once the dBASE table is created, the next few steps can be completed in either ArcCatalog

or ArcMap by way of individual Analysis tools or Modelbuilder. The process is essentially

the same. In order to avoid multiple *.lyr’s and *.shp’s from being created, Modelbuilder

is preferred. In Tools>Make XY Event Layer> Select .gdb table> and fill in the necessary

fields:

State of Colorado



***NOTE: Do not confuse X Field and Y Field as your data will not be projected properly.

(2) This merely creates an event layer, not a true shapefile layer, which is needed. In order to

create a *.shp, use the Copy Features tool in the Tool Box of ArcCatalog or ArcMap.

Alternatively, ‘Export’ the data from the drop-down menu and export to the File

geodatabase. Verify the data is accurately projected and review anomalies.

11.1.5. KML/KMZ Conversion

In order to use this data with ESRI software, the KML must be converted to Shapefile using the

process described below:

Open the KML/KMZ in Google Earth to analyze the data. Familiarize yourself with the

location and attribute information so that you can verify that all data is converted into the

new format.

Open the KML To Layer tool, input the KML/KMZ file and choose output information.

The output will include a Layer file as well as a file geodatabase with a feature dataset and

feature classes. The feature class and the layer file should be exactly the same, therefore

either one can be used for processing.

State of Colorado



All of the attribute information from the KML will be contained in the “PopupInfo” field of the

converted file. You will need to extract attributes from this field as necessary.

o To do so, create a new text field (Open attribute table and choose “Add Field”)

o Calculate the field using sections of the PopupInfo field. The PopupInfo field will

contain a long string of text in html formatting within the [CDATA[ ]] tag.

i.e. Mid( [PopupInfo],35,20) (this will extract 20 characters of text, starting at

the 35th character of the field).

11.1.6. CAD CAD Files can vary with data and file types. Files typically provided: a .dwg, annotation, polygon,

polyline, and point. The Import to CAD tool does not always convert CAD data cleanly into a

geodatabase. The following Steps are provided when the conversion tool fails to keep important

attributes and field required to derive broadband coverage data. CAD drawings can be immediately displayed in ArcGIS. This step does not create GIS data, it only displays CAD data in the GIS data view.

11.1.6.1. CAD Coordinate System

Many CAD drawings, especially survey drawings, are drawn to a real-world coordinate

system that GIS will recognize. When a CAD drawing is drawn to a standard coordinate

system (State Plane, for instance), it can be reprojected in GIS for proper alignment with

other GIS data. If the drawing has no standard coordinate system, it must be spatially

adjusted in GIS to align with other data. Spatial adjustment is usually less accurate than

reprojecting. The CAD file’s coordinate system may be noted on the drawing itself, in the

title bock, in the drawing notes, or on a layer showing a GPS point of origin. If there is no

information on the coordinate system and datum from the drawing or the creators, compare

drawing coordinates in CAD with GIS coordinates of the same area using a georeferenced

State of Colorado



orthophoto or other dataset. You will need this information to correctly align the data within

GIS. ***NOTE: The methods described below are also addressed in different parts of the processing document (i.e.: georeferencing and creating a file geodatabase. In order to create working Digital Raw Data and convert to GIS Format, these steps are represented here because they differ slightly from the aforementioned.

11.1.6.2. CAD Georeference

(1) Add the *.dwg to ArcMap for georeference. Only an image can be georeferenced, so

recognize the different file types presented and select the proper file as noted below:

Notice the units associated with the image at the bottom right of the screen as pictured

below, these units are what will be collected in the two corners in order to create a WLD

(WORLD) doc for georeferencing:

(2) Add the Base Data Layer needed for georeferencing. Notice, they may not be in the

same window. In order to view both files in the same window: View>Zoom Data>Full

Extent.

(3) Add Georeference Toolbar and Spatial Adjustment Toolbar:

View>Toolbars>Georeference and Spatial Adjustment:

(4) Make sure the .dwg file is the highlighted layer in the georeferencing Layer window.

(5) Now, two procedures exist in order to georeference. The steps will highlight both but go

in depth with only one. If the .dwg CAD file has a coordinate system but still needs to be

georeferenced because reprojection did not work, then the ‘Add Control Points’ tool can

be used as described in the georeferencing steps described in Digital Raw Data.

However, if not, follow the steps below.

Identified by

the white icon

with the

compass. No

spatial data

should exist

within the file;

thus no

attribution is

associated with the image.

State of Colorado



(6) Zoom in to a corner or identifiable point location and collect unknown units. The units

need to be typed in a *.txt file.

(7) Zoom to the corresponding corners in the base data layer and collect measurement units

(Decimal Degrees most likely). Add unit information to *.txt. Save .txt with the exact

same name as the .dwg file and in the same file as pictured below:

Figure 2: Notice Unknown Units across from corresponding Decimal Degree Coordinates.

(8) Copy/Paste *.txt file and rename extension with *.wld. A Rename “Warning” will ask if

you are sure you want to change the file extension>Select Yes.

***NOTE: If you do not see file extensions at the end of your documents, follow these steps:

(Windows 7)

Start>Control Panel>Appearance and Personalization>Folder Options>View tab>

Advanced Settings>clear the ‘Hide extensions for known file types’ check box>OK.

(9) Start new ArcMap Session>Add Point, Polyline, or Polygon file. Georeference>Fit to

Display

State of Colorado



(10) On the Georeference Toolbar>Select ‘View Link Table’ Tool>Load>Navigate to *.wld

file>Load>OK. (check Auto Adjust).

(11) Close ArcMap. Open a new ArcMap Session. Add Georeferenced CAD file and then Base

Data Layer. The two georeferenced corner should now be adjusted.

State of Colorado



***NOTE: IF NOT: Original CAD file could have been corrupted. Delete file and copy/paste new file. Redo steps (8-11).

OR IF NOT: Make sure no special ArcMap extensions are in operation causing reprojection to be skewed (i.e. Business Analyst)

(12)Add remaining layers: Point, Line, or Polygon. All should now automatically adjust to the

georeference points in the *.wld file.

(13)Export files to a File Geodatabase with appropriate names in order to create shapefiles:

Right Click>Data>Export>Navigate to .gdb and Save>OK>Add Features to ArcMap.

11.1.6.3. CAD Spatial Adjustment

(1) Start Editor Session>Select .gdb with shapefiles/feature classes

(2) Spatial Adjustment>Set Adjust Data…>Select ‘All Features in these layers’>Leave only the

shapefiles needing spatial adjustment checked’>OK.

(3) Select Link Tool. First click on the CAD Shapefile and then click on the corresponding

point Base Data layer as shown below:

State of Colorado



(4) Collect at least 4 links. More may be required in order to properly Adjust.

(5) Spatial Adjustment>Links>Save Links File…>Navigate to proper folder>Name and

Save>Select ‘YES’ to save ID’s and Coordinates.

(6) Spatial Adjustment>Adjustment Methods>Transformation-Affine

(7) Spatial Adjustment>Adjust. All layers should adjust to coordinates. Check quality of

adjustments. More Links may be needed.

***NOTE: If Spatial Adjustment requires more points, do not do multiple Spatial

Adjustment sessions. Instead, ‘Undo’ Spatial Adjustment by selecting the ‘Undo’ arrow on

the main ArcMap tool bar. Add more links or remove bad links. Save Link file

again>Overwrite old Link file.

(8) When Spatial Adjustment is complete, ‘Save edits’ in Editor>Stop Edit Session.

(9) Verify changes by closing ArcMap and Open a new session. Add ‘adjusted’ shapefiles and

Base Data Layer to review.

11.1.6.4. CAD Data Selection

(1) Review attribute data in Shapefiles/Feature Classes. Notice AutoCAD will have an abundance

of information not necessary for the broadband coverage processing.

(2) Extract Points, Lines or Polygons pertinent to coverage with a ‘Select by Attribute’ Query. In

the case of Phillips County, the ‘Layers’ field consisted of 96,665 points but only 4500 were

necessary to collect coverage data. Those attributes were identified as: Businesses, Schools,

Churches, Residences, and Central Offices. The points were extracted selected from the

‘Layers’ file with the Query:

“Layers” IN(“Businesses”, “Schools”, “Churches”, “Residences”, and “Central_Offices”)

Base Data Layer

Georeferenced CAD Shapefile

State of Colorado



(3) Once Data Points are selected: Right click Layer>Selection>Make Layer from Selection.

Rename Layer and export to .gdb. Processing to Standard GIS Format can now occur.