A Library for Articulating the Measurement Streams with Columnar Data

  • Authors

    • Mario Diván
    • María Laura Sánchez Reynoso
    2018-12-09
    https://doi.org/10.14419/ijet.v7i4.31.23373
  • Columnar Data, Data Interchanging, Data Stream Processing, Measurement, Evaluation.

  • The CINCAMI/Measurement Interchange Schema (MIS) organizes jointly data and metadata generated from the heterogeneous measurement devices under the same data stream. The Processing Architecture based on Measurement Metadata (PAbMM) is a data stream engine which processes the data streams organized under the CINCAMI/MIS schema. PAbMM is able to replicate in real-time each measurement stream but limited to C-INCAMI/MIS. This constitutes a use limitation of CINCAMI/MIS because the reading, using and writing of the measures was previously performed just by PAbMM. The CINCAMImisConversor library extracts this functionality with the aim of fostering the using along with any measurement project who need it. The functionality extraction was guided by PAbMM´s internal behavior, which is documented through SPEM metamodel. This allowed defining to the CincamimisConversor’s object model, its implementation and carry forward a discrete simulation for having an associated time reference. As a contribution, the library allows the data format conversion from the CINCAMI/MIS streams to the columnar organization. The library could translate 4900 measures in approximately 11 ms.

     

     

  • References

    1. [1] Cohn T, Global Political Economy: Theory and Practice, 7th ed., New York: Routledge, 2016.

      [2] von Bertalanffy L & Hofkirchner W, General System Theory: Foundations, Development, Applications, 1st ed., New York: George Braziller Inc., 2015.

      [3] Kang J, Lee S, Kim H, Kim S, Culler D, Jung P, Choi T, Jo K & Shim J, "High-fidelity Environmental Monitoring Using Wireless Sensor Networks," in Proceedings of the 11th ACM Conference on Embedded Networked Sensor Systems, Roma, 2013.

      [4] Diván M, "Processing Architecture based on Measurement Metadata," in International Conference on Reliability, Infocom Technologies and Optimization (ICRITO), Noida, 2016.

      [5] Olsina L, Papa F & Molina H, "How to Measure and Evaluate Web Applications in a Consistent Way," in Web Engineering: Modelling and Implementing Web Applications, Rossi G, Pastor O, Schwabe D & Olsina L, Eds., London, Springer-Verlag, 2007, pp. 385-420.

      [6] Molina H & Olsina L, "Towards the Support of Contextual Information to a Measurement and Evaluation Framework," in Quality of Information and Communications Technology (QUATIC), Lisbon, 2007.

      [7] Becker P, Lew P & Olsina L, "Strategy to Improve Quality for Software Applications: A Process View," in Conference on Software and Systems Process, Waikiki, Honolulu, 2011.

      [8] Diván M & Martín M, "Towards a Consistent Measurement Stream Processing from Heterogeneous Data Sources," International Journal of Electrical and Computer Engineering (IJECE), vol. 7, no. 6, pp. 3164-3175, December 2017.

      [9] Diván M & Sánchez Reynoso ML, "A Library for Measurement Interchanging from Heterogeneous Data Sources," in Under reviewing in The XLIV Latin-American Computer Conference (CLEI), Sao Paulo, Brazil, 2018.

      [10] Apache Storm Core Team, Apache Storm, Wakefield, MA: Apache Software Foundation, 2018.

      [11] Apache HBase Core Team, Apache HBase, Wakefield, MA: Apache Software Foundation, 2018.

      [12] R Core Team, R: A Language and Environment for Statistical Computing, Vienna: R Foundation for Statistical Computing, 2017.

      [13] Esteves D, Moussallem D, Neto C, Soru T, Usbeck R, Ackermann M & Lehmann J, "MEX Vocabulary: A Lightweight Interchange Format for Machine Learning Experiments," in 11th International Conference on Semantic Systems, Vienna, Austria, 2015.

      [14] J. Huang, C. Lange and S. Auer, "Streaming Transformation of XML to RDF Using XPath-based Mappings," in 11th International Conference on Semantic Systems, Vienna, Austria, 2015.

      [15] Sun L, Franklin M, Wang J & Wu E, «Skipping-oriented Partitioning for Columnar Layouts, » VLDB Endowment, vol. 10, nº 4, pp. 421-432, 2016.

      [16] Object Management Group (OMG), "Software and Systems Process Engineering Meta-Model Specification (SPEM)," Object Management Group (OMG), 2008.

      [17] Diván M & Olsina L, "Process View for a Data Stream Processing Strategy based on Measurement Metadata (Extended Version)," Electronic Journal of Informatics and Operations Research, vol. 13, no. 1, pp. 16-34, June 2014.

      [18] Diván M & Martín M, "Towards the feedback in the data stream processing based on an organizational memory," in National Conference on Informatic Engineering / Information System, Córdoba, 2013.

      [19] Apache Kafka Core Team, Apache Kafka. A distributed streaming platform, Wakefield, MA: Apache Software Foundation, 2018.

      [20] Diván M & Martín M, "A New Storm Topology for Synopsis Management in the Processing Architecture," in XLIII Latin American Computer Conference (CLEI), Córdoba, 2017.

      [21] Redis Labs Core Team, Redis, Mountain View, CA: Redis Labs, 2018.

      [22] Bifet A, Holmes G, Kirkby R & Pfahringer B, "MOA: Massive Online Analysis," The Journal of Machine Learning Research, vol. 11, pp. 1601-1604, August 2010.

      [23] Dalton L, "Optimal ROC-Based Classification and Performance Analysis Under Bayesian Uncertainty Models," IEEE/ACM Trans. Comput. Biol. Bioinformatics, vol. 13, no. 4, pp. 719--729, 2016.

      [24] Diván M & Sánchez Reynoso ML, "Behavioural Similarity Analysis for Supporting the Recommendation in PAbMM," in 1st International Conference on Infocom Technologies and Unmanned Systems (ICTUS), Dubai, 2017.

      [25] Open Geospatial Consortium and International Standard Organization, "ISO 19136:2007 Geographic Information - Geography Markup Language," International Standard Organization, Geneva, 2007.

      [26] Google, "google-gson: A Java serialization/deserialization library to convert Java Objects into JSON and back," 26 March 2018. [Online]. Available: https://github.com/google/gson. Last visit: 26.03.2018.

      [27] Kopff G, "gkopff/gson-javatime-serialisers: A set of GSON serialiser/deserialisers for dealing with Java 8 java.time entities.," 26 March 2018. [Online]. Available: https://github.com/gkopff/gson-javatime-serialisers. Last visit: 26.03.2018.

      [28] Garofalakis M, Gehrke J & Rastogi R, Eds., Data Stream Management: Processing High-Speed Data Streams, Berlin: Springer-Verlag Berlin Heidelberg, 2016.

  • Downloads

  • How to Cite

    Diván, M., & Laura Sánchez Reynoso, M. (2018). A Library for Articulating the Measurement Streams with Columnar Data. International Journal of Engineering & Technology, 7(4.31), 234-241. https://doi.org/10.14419/ijet.v7i4.31.23373