{"id":93,"date":"2022-12-14T10:29:15","date_gmt":"2022-12-14T10:29:15","guid":{"rendered":"https:\/\/demo.mekshq.com\/awsm-wp\/?p=93"},"modified":"2024-08-03T14:49:27","modified_gmt":"2024-08-03T14:49:27","slug":"time-series-database-and-analytics-using-azure-data-explorer","status":"publish","type":"post","link":"https:\/\/dipankar.me\/yogesh\/time-series-database-and-analytics-using-azure-data-explorer\/","title":{"rendered":"Spiro & Ola Electric"},"content":{"rendered":"\n
\n \n \n \n

\n In this article, I will be giving a brief summary of the Time Series Database and how it can be used for Advanced Analytics, we will also cover Azure Data Explorer : A Powerful time series database, this is one of the first article in the series of Timeseries: ADX articles that I am planning to publish, so lets start with these basic 3 questions <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

\n

  1. <\/span>What is a Time series Database?<\/li>
  2. Why Time-series DB? <\/span><\/li>
  3. How to do Advanced Analytics with ADX: Time Series DB<\/li><\/ol>\n\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

    \n 1. What is a Time Series Database?\n <\/h2>\n\n \n\n\n\n\n\n\n \n \n

    \n A Time series database is a database that is capable of storing high-frequency, streaming data in time series, it stores information or data part of a \u201ctime series,\u201d which is a set of data points that are associated with timestamps. Due to this, the time series database size grows very fast in a short amount of time. We will talk more about its characteristics further, for now, let’s look at popular TimeSeries Databases <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

    \n

    1. Influxdb <\/span><\/li>
    2. Apache Druid <\/span><\/li>
    3. Azure Data Explorer ( ADX \/Kusto)<\/li>
    4. TimescaleDB<\/li><\/ol>\n\n <\/p>\n\n \n\n\n\n\n\n\n\n \n
      \n
      \n \n
      \n \n
      \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
      \n Time Series Database\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

      \n 2. Why Time Series?\n <\/h2>\n\n \n\n\n\n\n\n\n \n \n

      \n The data coming from Sensors, Scooters, Cars, and Devices running for long hours is usually coming in milliseconds, or nanoseconds, so to store and fetch this data from relational databases or NOSQL databases is not a perfect match, storing and optimising this on <\/span>timestamp<\/strong> <\/span>till nanoseconds level and ability to <\/span>fetch, partition the data on timestamp<\/strong> <\/span>is inbuilt to time series database <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

      \n If you see the characteristic of the data here, it is a stream of data coming at a very high frequency, like every second.  Time dimension takes a crucial role here. A time series database helps to partition this data on time. <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

      \n The best way to analyze the things which are constantly fluctuating i.e., <\/span>Non-Stationary Data<\/span> <\/span>over time is by using Time Series\n <\/blockquote>\n\n \n\n\n\n\n\n\n \n \n

      \n Partitioning data on time helps sort the out-of-order data and ordered data separately. This immensely improves the analytical queries, data engineering jobs, and data scientist models running on it. Also, the data is indexed on time, so fetching data based on time is fast. It also supports time-based joins across multiple tables for effective analytics. <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n\n \n

      \n
      \n \n
      \n \n
      \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
      \n Fetching data from the devices within milliseconds\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

      \n A few of the things that a time series database is good at for the live streaming data is <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

      \n a) <\/span>Low latency<\/strong> <\/span>for data ingestion from the sensor to the database <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

      \n b) <\/span>High write throughput <\/span><\/strong>as the time series data is live, write-heavy, and highly concurrent <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

      \n c) <\/span>High read throughput<\/strong> <\/span>as the time series data is to be read and instantly presented on BI Tools <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

      \n d) <\/span>High data compression <\/span><\/strong>within a day millions of records are added, these should be compressed while ingestion or while storage to avoid extremely large disk size consumption <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

      \n Azure Data Explorer is a fully managed PaaS service from Azure, it comes as a very good option for Time Series Database it fits in all the above requirements and it also has a great computing power to process the live streaming data and historical data stored in ADX cluster <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

      \n Time Series can be created and analysed within a few seconds in Azure Data Explorer\n <\/blockquote>\n\n \n\n\n\n\n\n\n \n \n

      \n ADX has the ability to generate real-time alerts based on the data that is ingested into ADX, it can run rules against the data in flight and generate alerts, emails <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

      \n ADX also has very good integration with all Azure Services like Logic App, Event Hub, Blob Storage, and Application insights, in fact, the journey of ADX ( Kusto ) started at Microsoft as a log aggregator, it is being used as a log engine of all Azure services for last 10 years, in 2021 it was publicly launched to be used as Time Series Database <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

      \n It has some great machine learning features, trends, correlations, and models can be executed on the fly and visualisations can be generated using UI\/UX and BI tools <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

      \n Other striking features of ADX <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

      \n

      1. It is used as a logging solution for all Microsoft Services in the background for the last 10 years <\/span><\/li>
      2. So it’s a proven solution, as per Microsoft documentation daily 100 Petabytes of data is ingested in ADX, a very powerful data ingestion system <\/span><\/li>
      3. Use intuitive and powerful KQL ( Kusto Query Language ), Azure Log Analytics queries are actually KQL queries so upskilling for Azure resources should not be a big challenge <\/span><\/li>
      4. For SQL users there will be a bit of learning but should be easily manageable, they can start with TSQL and gradually mode to <\/span><\/li>
      5. Direct integration point with all Azure services and multiple other third-party services <\/span><\/li>
      6. Plotting trends and applying toner machine learning algorithms becomes easy using ADX features <\/span><\/li><\/ol>\n\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

        \n 3. How to do Advanced Analytics with ADX: Time Series DB?\n <\/h2>\n\n \n\n\n\n\n\n\n \n \n

        \n ADX for Time Series Analytics <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

        \n

        1. Inbuilt support for TimeSeries creation <\/span><\/li>
        2. Inbuilt functions for anomaly detection and forecasting <\/span><\/li>
        3. Inbuilt support for ML algorithm integration <\/span><\/li><\/ol>\n\n <\/p>\n\n \n\n\n\n\n\n\n\n \n
          \n
          \n \n
          \n \n
          \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
          \n Advanced Analytics with ADX\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

          \n In the next few articles, I am going to cover different aspects of the ADX Timeseries database and do a deep dive into it <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

          \n Let me know what role you play: Are you a Data Engineer, Data Scientist, or Data Analyst, and are you already using any Timeseries database, also let me know what aspect of the Timeseries database you would like to know more – about so that I cover that in my next article, also if you are interested in learning ADX: Time Series database architecture and implementation, I will be happy to share <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

          \n Hope you enjoyed the article, thanks for reading <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

          \n Cheers\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

          \n Yogesh\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

          \n Happy reading, happy learning …\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

          \n Follow me here <\/span>hashtag<\/span>#<\/span>DigitalDipankar<\/span><\/a>\n <\/p>\n\n \n\n\n\n\n\n\n \n <\/div>\n","protected":false},"excerpt":{"rendered":"

          In this article, I will be giving a brief summary of the Time Series Database and how it can be used for Advanced Analytics, we will also cover Azure Data Explorer : A Powerful time series database, this is…<\/p>\n","protected":false},"author":1,"featured_media":604,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[28],"tags":[31],"class_list":["post-93","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cognizant-technology-solutions","tag-jan-2007-dec-2014"],"_links":{"self":[{"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/posts\/93"}],"collection":[{"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/comments?post=93"}],"version-history":[{"count":4,"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/posts\/93\/revisions"}],"predecessor-version":[{"id":624,"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/posts\/93\/revisions\/624"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/media\/604"}],"wp:attachment":[{"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/media?parent=93"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/categories?post=93"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/tags?post=93"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}