Big data is bigger than you think, and it gets bigger every second, and so far shows no sign of slowing down. in in 2010 Google chief executive Eric Schmidt noted that the amount of data collected since the dawn of humanity until 2003 was the equivalent to the volume we now produce every two days. the last date I can find, for data volume created in 2011 ,we created 1.8 zettabytes (or 1.8 trillion GBs) of information, which is enough data to fill 57.5 billion 32GB Apple iPads, enough iPads to build a Great iPad Wall of China twice as tall as the original. I put the last comparison in, simply because I have no idea of numbers this big and the storage required to hold it.
According to IBM, every single day we create 2.5 quintillion bytes of data. IBM argues that the exponential growth of data means that 90 percent of the data that exists in the world today has been created in the last two years. “This data comes from everywhere: sensors used to gather climate information, posts to social media sites, digital pictures and videos, e-commerce transaction records, and cell phone GPS coordinates, to name a few.”
Birmingham, a city I grew up in and have lived for a good proportion of my life, is on the leading edge of the Big City Data, in a project called HiTemp, a team from Birmingham University have place 280 wifi enabled sensors across the city, which are sub divided into three main groups :
- Network 1 (coarse array): 30 AWS will be sited in primary electricity sub-stations (average spacing of 3km)
- Network 2 (wide array): 150 air temperature sensors located on schools (one in every medium super output area (MSOA), or areas containing a population of 7,200)
- Network 3 (fine array): 100 air temperature sensors located on lampposts in the CBD (50/km2)
A map of all the above can be seen on a live map showing the location of each sensor, see todays photo, clicking on each sensor, takes you to the sensor page, which has a link to the data feed, and here we get all the info we need, live, from air pressure to solar radiation, to wind direction and speed, and more, each of these can be drilled to show the last three months of data.
The point of all this research is that we are now in a position to design houses, offices and almost any building, that can use this data to manage its own energy or more, say the sensors in Birminghan showed cloud cover increasing to the west, a building watching this could prepare, making sure its own solar array was getting the best energy, it could also team up with other buildings in a local a network to pool resources till the cloud cover opens up, and by looking at the sensors in the west, it will know when this might or will happen.
Each building will also need its own sensor array, passing info into the cloud to allow other buildings the same information. We can see this is not difficult and certainly not expensive.
To put the data explosion in context, consider this. Every minute of every day we create
- More than 204 million email messages
- Over 2 million Google search queries
- 48 hours of new YouTube videos
- 684,000 bits of content shared on Facebook
- More than 100,000 tweets
- $272,000 spent on e-commerce
- 3,600 new photos shared on Instagram
- Nearly 350 new WordPress blog posts
You might say this is meaningless, but if I can watch the tweets and use an algorithm to detect weather chat, I can integrate that into the sensor data to get a better picture, add face book, and google search, I am getting a dam good picture of weather.
Change the algorithm, and I suddenly get traffic and bus info, and a google message telling me to get to the bus stop, its a little scary, but for me I welcome this its not big brother, its good use of sensor data