A Unified Frame Work to Integrate Hadoop and IOT to Resolve the Issues of Storage, Processing with Leveraging Capacity of Analytics

  • Authors

    • Gudapati Syam Prasad
    • P Rajesh
    • Sk Wasim Akram
    2018-05-31
    https://doi.org/10.14419/ijet.v7i2.32.15390
  • Hadoop, IOT, Analytics, Storage, Map Reduce.

  • The new trend in the research and real time applications is Internet of Things (IOT). The functional benefits of IOT are ranging from smart house to smart cities. The main purpose of IOT is to integrate various devices logically and interacting between the devices without human intervention. The current discussion mainly focuses on leveraging the capacity of analytics in IOT and resolves the storage issues of the bulk data generated by IOT. The proposed idea gives the usage of Hadoop platform to store the data and from that data performing analytics for the sake of better utilization of IOT communications. The importance is explained with some real time scenarios where there is perfect blend of Hadoop platform and IOT. To store the various categories of the data Hadoop Distributed File System (HDFS) can be used, and to ingest the data from external platforms we can make use of Sqoop or Flume. The data available in HDFS can be used to process with the usage of Map Reduce (MR)technique. Once the data is available in  HDFS the analytics can be performed with Hive, Pig or R in the context of Machine learning or data mining techniques. The outcome of the proposed idea is integration of Hadoop and IOT platforms with a unified frame work which accommodates the integration of Hadoop and IOT, storage provisions to handle bulk data, processing of the stored data and applying analytics so as to effectively serve various stake holders.

     

     


     

  • References

    1. [1] Umapavankumar.K, Dr.B.Lakshmareddy ,†Various Computing models in Hadoop eco system along with the perspective of analytics using R and Machine learning†Vol. 14 CIC 2016 Special Issue International Journal of Computer Science and Information Security (IJCSIS) https://sites.google.com/site/ijcsis/ ISSN 1947-5500.

      [2] www.cloudera.com

      [3] www. https://kontakt.io

      [4] S. Lohr, “The age of big data,†N. Y. Times, vol. 11, 2012.

      [5] S. Madden, “From Databases to Big Data.,†IEEE Internet Comput., vol. 16, no. 3, 2012.

      [6] P. Zikopoulos, C. Eaton, and others, Understanding big data: Analytics for enterprise class hadoop and streaming data. McGraw-Hill Osborne Media, 2011.

      [7] A. McAfee, E. Brynjolfsson, T. H. Davenport, D. J. Patil, and D. Barton, “Big data,†Manag. Revolut.Harv. Bus Rev, vol. 90, no. 10, pp. 61–67, 2012.

      [8] R. Appuswamy, C. Gkantsidis, D. Narayanan, O. Hodson, and A. Rowstron, “Scale-up vs Scale-out for Hadoop: Time to rethink?,†in Proceedings of the 4th annual Symposium on Cloud Computing, 2013, p. 20.

      [9] A. S. Tanenbaum and M. Van Steen, Distributed systems.Prentice-Hall, 2007.[7] C. P. Chen and C.-Y. Zhang, “Dataintensive applications, challenges, techniques and technologies: A survey on Big Data,†Inf. Sci., vol. 275, pp. 314–347, 2014.

      [10] T. B. Murdoch and A. S. Detsky, “The inevitable application of big data to health care,†Jama, vol. 309, no. 13, pp. 1351– 1352, 2013.

      [11] Dr.B.LakshmaReddy,Umapavankumar.K,†Big data techniques and analytics in Ecommerce business†International Conference at Pondicherry University, on October 2016.

      [12] www.safaribooksonline.com

      [13] J. Dean and S. Ghemawat, “MapReduce: simplified data processing on large clusters,†Commun. ACM, vol. 51, no. 1, pp. 107–113, 2008.

      [14] J. Y. Monteith, J. D. McGregor, and J. E. Ingram, “Hadoop and its Evolving Ecosystem.,†in IWSECO@ ICSOB, 2013, pp. 57–68.

      [15] K. Ting and J. J. Cecho, Apache Sqoop Cookbook. O’Reilly Media, Inc., 2013. [14] S. Hoffman, Apache Flume: Distributed Log Collection for Hadoop. Packt Publishing Ltd, 2013.

      [16] S. Haloi, Apache ZooKeeper Essentials. Packt Publishing Ltd, 2015.

      [17] M. K. Islam and A. Srinivasan, Apache Oozie: The Workflow Scheduler for Hadoop. O’Reilly Media, Inc., 2015.

      [18] C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins, “Pig latin: a notsoforeign language for data processing,†in Proceedings of the 2008 ACM SIGMOD international conference on Management of data, 2008, pp. 1099–1110.

      [19] H. Bansal, S. Mehrotra, and S. Chauhan, Apache Hive cookbook.Packt Publ., 2016.

      [20] E. Alpaydin, Introduction to machine learning (adaptive computation and machine learning series). The MIT Press Cambridge, 2004.

  • Downloads

  • How to Cite

    Syam Prasad, G., Rajesh, P., & Wasim Akram, S. (2018). A Unified Frame Work to Integrate Hadoop and IOT to Resolve the Issues of Storage, Processing with Leveraging Capacity of Analytics. International Journal of Engineering & Technology, 7(2.32), 147-149. https://doi.org/10.14419/ijet.v7i2.32.15390