GIS DATA MANAGEMENT
A PRESENTATION FOR THE UNIVERSITY OF ZIMBABWE GIS STUDENTS
TALKING POINTS
Why Data Management
Approach to GIS Data Management
Practical Use Case: A Data Management Example
Challenges to Data Management
Modern and Future Trends in Data Management
WHY DATA MANAGEMENT
DEFINITION: Data management is the practice of collecting, processing, storing, securing and disseminating an
organization’s data, where it is then consumed for meeting business requirements and outcomes.
Growing appetite for reliable, up-to-date data.
To drive data-driven decision making and actions within an organization.
High-quality analyses and reporting requires high-quality GIS data.
Today’s digital economy makes it an inevitable defacto practice.
Data is an intangible asset to create value.
Tim Berners-Lee, once said, “Data is a precious thing and will last longer than the systems themselves.”
APPROACH TO GIS DATA MANAGEMENT
1. Develop an effective data management strategy – data management begins with a strategy!
Answers how the data is managed and governed in the organization.
Involves assessing the current state of the company’s data, developing policies and procedures (data governance).
Eradicates wasting valuable time and resources collecting, storing, and analyzing the wrong types of data.
Eliminates ad hoc and reactive data collection and processing.
OBJECTIVE: Make the data easy to govern.
2. Data needs assessment
A needs assessment determines the gap between current and desired.
Inventory of all the required data and outcomes (e.g. analyses).
APPROACH TO GIS DATA MANAGEMENT
3. Data collection
Data sourcing, field collection, data creation.
Data type to be collected (spatial, non-spatial or a combination of both?)
Data collection techniques: How will the data be collected?
Manually as needed, extract scheduling, surveying, UAS, mobile, web editing?
4. Data Processing (Preparation)
Data cleaning and transformation (which tools/technology: ETL tools, ArcGIS, QGIS, Excel (Power Query), etc.?)
Datasets consolidation into single database objects (feature classes, tables, raster catalog, mosaic)
Data engineering (identify patterns, incomplete and disparate data).
Implement guidelines for naming data, populating lineage attributes, and adding metadata.
Data quality control and validation.
Setup automated geoprocessing, data reviewer and QC workflows for repetitive data preparation tasks.
THE MOST TIME CONSUMING ASPECT, especially in big data companies!
APPROACH TO GIS DATA MANAGEMENT
5. Data Storage
Where will you store your data? Which DB technology: RDBMS, SharePoint, MS Dataverse (warehouse) or data lake?
Consider the data formats
Professional database schema designs
Standardization (makes the data easy to govern)
Use international standardized schemas wherever possible: topographic (e.g. MGCP); operational (various ESRI-designed industry-specific schemas, etc.)
Implement database management best practices
Replicas, versioning, archiving, compressing, enable editor tracking, etc.
6. Data Security
How will you keep your data secure?
Implement best practices:
Create OS access groups, database admin users, and/or email ID (cloud active directory) users
Grant/revoke permissions for users PER database object (feature class, table, raster catalog, mosaic, etc.)
Setup client web app logins
APPROACH TO GIS DATA MANAGEMENT
7. Data Dissemination
For teams or departments requiring the ability to collaborate?
Data distribution via database replicas (online, offline, check-out/check-in)
Online collaborative editing and/or mobile applications
Make access to data and analysis easier for the end-user
Dedicated interactive GIS web map applications
Dashboard applications (ArcGIS Dashboards, Power BI Dashboards, etc.)
GIS map products
APPROACH TO DATA MANAGEMENT
PRACTICAL USE CASE: A DATA MANAGEMENT EXAMPLE
MONUSCO (UN Mission in DRC)
How each step in the above process was executed and the results
GIS
Operations
HQ
PRACTICAL USE CASE: A DATA MANAGEMENT EXAMPLE
MONUSCO (UN Mission in DRC)
How each step in the above process was executed and the results
1. Data strategy
From a centralized to decentralized data management approach
Formulated policies and procedures for data management governance.
2. Identify organization-wide needs
Mission’s primary business objectives are defined in the mandate: protection of civilians; disarmament, rehabilitation and re-integration; capacity building in security reform.
Discovered a great appetite for GIS analyses to support the Mission’s situation analytical capability but with no or disparate datasets:
Incidents analyses,
Hotspots identification,
Affected populations (including IDPs),
Area security gaps,
Armed groups monitoring,
Mission field activities analyses and statistics, in response to the security incidents or in support of the mandate.
Identified data gaps
Required data inventory list
PRACTICAL USE CASE: A DATA MANAGEMENT EXAMPLE
3. Establish the data storage
Identified the database technology (MS SQL Server, PostgreSQL)
Lean and mean databases - only what is necessary, and determined to work effectively in order to deliver.
GIS Data Previous No. of Databases Current/Planned No. of REMARKS
Databases
Topographic data 6 2-3 Due to different
international schemas and
scales
Other base data 9 2 one custom design, one
OSM
Operational data 9 2 COGI, MOGI
Imagery 10 1 Satellite and drone imagery
mosaic
TOTAL 34 8
PRACTICAL USE CASE: A DATA MANAGEMENT EXAMPLE
4. Establish the data storage (Contd..)
Identified repositories (share drives) and already standardized schemas (e.g., Common Operational Geospatial Information - COGI)
to load data
Designed schemas for data which could not fit in COGI (Mission-specific Operational Geospatial Information - MOGI)
Configured and implemented best database management practices
Versioning, archiving, compressing, enable editor tracking, data distribution techniques (scripts, replicas).
Established user security groups, roles and permissions
5. Data Collection
Data sourced from various teams and departments
Configured a mobile data collection system (open-sourced SMASH digital field mapping) to drive data collection by anyone and
everywhere.
Promoted drone data collection for enhanced situational awareness
PRACTICAL USE CASE: A DATA MANAGEMENT EXAMPLE
6. Data Preparation (Processing)
Consolidated all datasets from the previous multiple databases into the “lean and mean” databases (COGI, MOGI, MGCP, Image Mosaic DB, etc.)
A lot of data cleaning, transformation, engineering and loading.
Data quality control and validation (accuracy, completeness, integrity, etc. checks)
7. Information Dissemination
Interactive web GIS applications.
Published web feature editing services.
GIS map products (hard copies)
Website or GIS hub (for easy searching, cataloguing, and visibility)
8. Data Governance
Metadata (feature class, tables, mosaic dataset)
Policies and procedures
Developed an internal GIS Data Catalogue online application (for easily accessible metadata by GIS staff)
CHALLENGES TO DATA MANAGEMENT
A few major challenges to GIS data management are:
Shortage of data management skills: those who can implement, manage and monitor the complete aspects of data
management (not just an aspect), more so in organizations handling increasing amounts of data .
Increased data volumes
Constantly changing compliance requirements: make it a challenge to ensure people are using the right data.
Resistance to change
No or weak data governance
MODERN AND FUTURE TRENDS IN DATA MANAGEMENT
Cloud Migration: migrate systems to the cloud infrastructure
Seamless data management from data collection to dissemination
Scalability
Data automation
Data on auto-pilot
Web Editing
for data maintenance and enabling self-service data editing throughout an organization
GIS DATA MANAGEMENT
THANK YOU
