Will data collection consume us?
Some argue that big data is a bad thing while others say it is the only way to analyze data repositories for better and more strategic decision-making.
Both the public and private sectors use big data extensively. Public officials use it to collect information from cell phone GPS systems as a way to predict better and more efficient traffic patterns.
Private firms collect metadata from e-mails, on-line orders, tweets and Web sites as a way to understand shopping and buying patterns. Political campaigns consume data from Twitter accounts to determine hot issues and to seek out support for candidates. Transportation officials use big data to determine dangerous stretches of roadways. Medical facilities collect data for research and teaching.
The government collects and analyzes many types of data about citizens. Much of it is made public, but not all because of privacy issues. A Web site, www.data.gov, allows state and local governments to upload data sets to a public portal. On another end of the spectrum is a closed data portal generated by organizations like the National Security Administration (NSA). The data stored here is not available to the public.
The National Security Administration (NSA) constructed a very large sophisticated data center in Utah last year. The project was known as the Intelligence Community Comprehensive National Cybersecurity Initiative and the cost was $1.5 billion. Few government information repositories will ever reach the scale of this facility, but it makes one question how many more data centers will be needed in the not-so-distant future.
The amount of data generated usually depends on the size of whatever entity is collecting information. A school district may only struggle with preserving and transmitting electronic student records, but a state health services agency has a completely different mandate. These agencies are tasked with the curation and management of extremely sensitive records. Collectively, however, data is being stored at historic rates. Without big data solutions, it would be almost worthless because of its sheer abundance.
In 2013, it was reported that 90 percent of the world’s stored data had been produced in the previous two years. How startling is that? If the trend continues – and it will – the world will soon have so much data, one must question whether it will be possible to maintain and secure it in the future.