ACSPRI Conferences, ACSPRI Social Science Methodology Conference 2016

Font Size:  Small  Medium  Large

Open Source Software for Data Collection Support

Danny Jonathan Smith, Andrew Craig Ward

Building: Holme Building
Room: Cullen Room
Date: 2016-07-21 11:00 AM – 12:30 PM
Last modified: 2016-05-06

Abstract


Free and open source software provides many useful tools for data processing, cleaning and data collection support. Where proprietary software provides formal software support, open source software benefits from a greater breadth of functionality and much shorter development cycles due to community engagement. In many cases free and open source software is functionally superior to existing proprietary tools. The free nature of this software also allows smaller organisations to use advanced and well developed tools where a proprietary alternative would be cost-prohibitive. Although often the focus is on the use of open source software for statistical analysis, it has many benefits at all stages of the data collection process from sampling to output production.

We present a detailed case study of an end to end data collection process to demonstrate the advantages of using open source tools in conjunction with an established proprietary software stack. We will show how open source software can be used to easily integrate heterogeneous data systems, with a particular focus on R and the growing number of libraries allowing for simple data munging and efficient connectivity between systems. Issues with confidence in open source software (both reputation and functionality) are also addressed, specifically in relation to integration with existing proprietary systems.

Examples of the use of various open source tools and their integration with proprietary software are provided at various stages of the data collection process:
• Sampling and contact cleaning (R)
• Contact management over multiple modes of data collection (R integrated with SQL Server, proprietary data collection software, web APIs)
• Real time collection progress reporting (R Shiny, Linux, Apache, SQL Server, Django)
• Data cleaning and massaging (R)
• Automated file based and online reporting (R integration with MS Office for mass production of automated reports, production of high quality graphs with ggplot, simple interactive maps using leaflet)