In the ever-evolving landscape of business intelligence (BI), the need for seamless interaction with data is paramount. Imagine a world where you could effortlessly pose natural language questions to your datasets and receive insightful answers in return. Welcome to the future of BI, where the power of conversational interfaces meets the robust capabilities of Domo. This blog post serves as your comprehensive guide to implementing a BI ChatBot within the Domo platform, a revolutionary step towards making data exploration and analysis more intuitive and accessible than ever before. Gone are the days of wrestling with complex queries or navigating through intricate dashboards. With the BI ChatBot in Domo, users can now simply articulate their questions in plain language and navigate through datasets with unprecedented ease. Join us on this journey as we break down the process into manageable steps, allowing you to harness the full potential of BI ChatBot integration within the Domo ecosystem. Whether you're a seasoned data analyst or a business professional seeking data-driven insights, this guide will empower you to unlock the true value of your data through natural language interactions. Get ready to elevate your BI experience and transform the way you interact with your datasets. Let's dive into the future of business intelligence with the implementation of a BI ChatBot in Domo. Prerequisites: ChatGPT API Key: Prepare for the integration of natural language to SQL conversion by obtaining a ChatGPT API Key. This key will empower your system to seamlessly translate user queries in natural language into SQL commands. DOMO Access: Ensure that you have the necessary access rights to create a new application within the Domo platform. This step is crucial for configuring and deploying the BI ChatBot effectively within your Domo environment. 1: Integrate the HTML Easy Bricks App. Begin the process by incorporating the HTML Easy Bricks App into your project. Navigate to the AppStore and add the HTML Easy Bricks to your collection. Save it to your dashboard for easy access. Upon opening the App for the first time, it will have a default appearance. To enhance its visual appeal and functionality, customize it by incorporating the HTML and CSS code. This transformation will result in the refined look illustrated below. Image 1: DOMO HTML Easy Brick UI 2: Map/Connect the Dataset to the Card. In this phase, establish a connection between the dataset and the card where users will pose their inquiries. Refer to the image below, where the "Key" dataset is linked to "dataset0." Extend this mapping to accommodate up to three datasets. If your project involves more datasets, consider using the DDX-TEN-DATASETS App instead of HTML Easy Bricks for a more scalable solution. This ensures seamless integration and accessibility for users interacting with various datasets within your Domo environment. Image 2: Attach Dataset With Card 3: Execute the Query on the Dataset for Results. In this phase, you'll implement the code to execute a query on the dataset, fetching the desired results. Before this, initiate a call to the ChatGPT API to dynamically generate an SQL query based on the user's natural language question. It's essential to note that the below code is designed to only accept valid column names in the query, adhering strictly to MySQL syntax. To facilitate accurate query generation from ChatGPT, create a prompt that includes the dataset schema and provides clear guidance for obtaining precise SQL queries. Here is a call to the ChatGPT API to get SQL Query. VAR GPTKEY = 'key' VAR Prompt = 'Write effective prompt' $.ajax({ url: 'https://api.openai.com/v1/chat/completions', headers: { 'Authorization': 'Bearer ' + GPTKEY, 'Content-Type': 'application/json' }, method: 'POST', data: JSON.stringify({ model: 'gpt-3.5-turbo', messages: Prompt, max_tokens: 100, temperature: 0.5, top_p: 1.0, frequency_penalty: 0.0, presence_penalty: 0.0 }), success: function (response) { //Write code to store the Query into the variable } }); Refer to the code snippet below for executing the query on Domo and retrieving the results. var domo = window.domo; var datasets = window.datasets; domo.post('/sql/v1/'+ 'dataset0', SQLQuery, {contentType: 'text/plain'}).then(function(data) { //Write your Java or JQuery code to print data. }); The above code will accept the SQL queries generated by ChatGPT. It's important to highlight that, in the code, there is a hardcoded specification that every query will be applied to the dataset mapped as 'dataset0'. It's advisable to customize this part based on user selection. The code is designed to accept datasets with names such as 'dataset0', 'dataset1', and so forth. Ensure that any modifications align with the chosen dataset for optimal functionality, you can also use the domo.get method to get data for more information visit here. The outcome will be presented in JSON format, offering flexibility for further processing. You can seamlessly transfer this data to a table format and display or print it as needed. Conclusion Incorporating a BI ChatBot in Domo revolutionizes data interaction, seamlessly translating natural language queries into actionable insights. The guide's step-by-step approach simplifies integration, offering both analysts and business professionals an intuitive and accessible data exploration experience. As datasets effortlessly respond to user inquiries, this transformative synergy between ChatGPT and Domo reshapes how we extract value from data, heralding a future of conversational and insightful business intelligence. Dive into this dynamic integration to propel your decision-making processes into a new era of efficiency and accessibility.
In this blog, I will be sharing insights on how to effectively manage Conditional Authorization and Swagger Customization. Case 1 I'm currently working on a problem our QA team found while testing our website. Specifically, there's an issue with one of the features in the application that uses an API. In the QA environment, we need to allow access without authentication, but in the production environment, authentication is required. To fix this, I added a feature called Conditional Authorize Attribute with help of Environment Variable. This feature lets us control access to the API based on the environment. It allows anonymous access when necessary. In my situation, I've added a environment variable setting called "ASPNETCORE_ENVIRONMENT" to "QA" in the testing site's pipeline. Because of this, I can use the API on the QA server without requiring authentication. This method also helps specific authorization rules for the API based on the environment. Case 2 Additionally, I've added Swagger requests into a value object to meet specific requirements on swagger. By extending the Swashbuckle Swagger IOperationFilter, I integrated logic tailored to our needs. This approach allows us to customize requests in Swagger for all APIs directly. Furthermore, I've implemented a middleware designed to handle responses and here's how it works. In my case, there are three kinds of response class in my code that specify the response type (like ApiErrorResponse, ValidatorResponse, ResponseModel). According to the requirements, when we get a 200-status code with the correct response class model, I need to wrap the response object in a value format. I created a middleware for this. It figures out which endpoint we're dealing with through the HttpContext. Using that endpoint, I grab the metadata related to the ProducesResponseTypeAttribute class and check for a status code of OK (Metadata Extraction). If I manage to get the metadata with a status code of 200, I include that response in value format. Otherwise, I stick with the same model response. This helps you to modify the response as per needed outcome. These implementations provide a flexible solution for conditionally authorizing API access and wrapping request/response in an object according to specified requirements.
Hello, here I am going to explain what are the ways to publish a Power BI report on the Power BI Service which is the Cloud solution provided by Power BI and on the on-premise solution, keep reading this blog and, you will be able to publish the report on the Power BI Service as well as in on-premise. Power BI Service(Cloud Solution) Prerequisites: Power BI Service Account:- You must have a Power BI Service Account in order to publish the report to the service. Power BI Desktop:- Install the Power BI desktop on your system. Note:- Before publishing the report, login in power bi desktop with your Power BI service account. There are simply three types: Publish from Power BI Desktop. Publish from Power BI Server. Publish using Power BI Rest API. 1: Publish from Power BI Desktop Publish from the Power BI Desktop is the easiest way to publish a Power BI report, First, open the report that you want to publish in Power BI Desktop then click on the publish icon. Figure 1: Power BI Desktop When you click on it, it will ask you to select the workspace that you want to publish on. Then click on the select button, Bingo!! You successfully published the report. Figure 2: Power BI Desktop 2: Publish from Power BI Service This is a second method to publish a Power BI Report, For that first log into the Power BI service and, open the workspace where you want to publish the report. After, Click on the Upload icon then browse your report, That’s it for the second method. Figure 3: Power BI Service 3: Publish Using Power BI Rest API Here, We are using a PowerShell script that will work based on Power BI Rest API to publish the Power BI report. It is really simple to use Power BI Rest API with PowerShell script to deploy the report. To use Power BI API in PowerShell Script we need to install the MicrosoftPowerBIMgmt module. Install-Module -Name MicrosoftPowerBIMgmt Here are the steps to deploy the Power BI report using PowerShell Script. Step 1: Create a required parameter Before publishing the report we need the following things that we will assign as parameter PBI Username:- Assign the Username of the Power BI service Account. PBI Password:- Assign Password of Power BI service Account. WorkspaceName:- Assign workspace name. Report Name:- Assign Report name. Power BI File path:- Provide full Power BI File path where is located Workspace object:- In This variable, we assign the workspace id from the workspace name Step 2: Make the password secure and create a credential variable. $SecurePassword= ConvertTo-SecureString -String $PBIPassword -AsPlainText -Force $Creds = New-Object Management.Automation.PSCredential -ArgumentList ($PBIUserName, $SecurePassword) Step 3: Connect to Power BI Service Connect-PowerBIServiceAccount -Credential $Creds Step 4: Publish the Report This is the final and very important step in this step we will publish the report. New-PowerBIReport -Path $pbixPath -Name $ReportName -Workspace $workspaceObject -ConflictAction CreateOrOverwrite On-Premise Solution Prerequisites: Power BI report server:- You must have a Power BI report server that is installed in your system with fully configured and, it must be set up with a Database, web URL, and web portal. Power BI desktop report server:- Install the Power BI desktop report server in your system. Note:- This is different than normal Power BI desktop There are simply two types: Using Power BI Desktop. Using Power BI Report Server. 1: Using Power BI Desktop. Open the report that you want to deploy then Open the “File” navigation option(situated in the upper-left corner) and, click on “save as”. Figure 4- Power BI Desktop File panel Note:- When you click on “Power BI Report Server” you will get the below message where you have to give the web portal URL URL from the step of configuring the report server. Figure 5:- Adding Report Server Address(Source:- Radacad) After successful deployment, you will see a message with a link to the report. Figure 6:- Successfully published report(Source: Radacad). 2: Using Power BI Report Server. This is a different way to deploy the report so, first you have to open Power BI Report Server. Figure 7:- Power BI Report Server(Source: Radacad) Click on “Upload” as mentioned in the above picture and upload Power BI Report from your files Conclusion I Hope I’ve Shown You How Easy It Is To deploy Power BI reports on the Report server as well as on-premise, So far we have learned the step-by-step process of it.
Here, I will explain how to restrict users from using expired tokens in a .NET Core application. Token expiration checks are crucial for ensuring the security of your application. Here's a general outline of how you can achieve this: 1. Configure Token Expiration: When generating a token, such as a JWT, set an expiration time for the token. This is typically done during token creation. For example, when using JWTs, you can specify the expiration claim: var tokenDescriptor = new SecurityTokenDescriptor { Expires = DateTime.Now.AddMinutes(30) // Set expiration time }; 2. Token Validation Middleware: Create middleware in your application to validate the token on each request. This middleware should verify the token's expiration time. You can configure this middleware in the startup or program file on the .NET side. public void Configure(IApplicationBuilder app, IHostingEnvironment env) { app.UseMiddleware<TokenExpirationMiddleware>(); } 3. Token Expiration Middleware: Develop middleware to validate the token's expiration time. Take note of the following points: ValidateIssuerSigningKey: Set to true, indicating that the system should validate the issuer signing key. IssuerSigningKey: The byte array represents the secret key used for both signing and verifying the JWT token. ValidateIssuer and ValidateAudience: Set to false, indicating that validation of the issuer and audience is skipped. By setting ClockSkew to TimeSpan.Zero, you specify no tolerance for clock differences. If the current time on the server or client is not precisely within the token's validity period, the token is considered expired. public class TokenExpirationMiddleware { private readonly RequestDelegate _next; public TokenExpirationMiddleware(RequestDelegate next) { _next = next; } public async Task Invoke(HttpContext context) { // Check if the request has a valid token var token = context.Request.Headers["Authorization"].FirstOrDefault()?.Split(" ").Last(); if (token != null) { var tokenHandler = new JwtSecurityTokenHandler(); var key = Encoding.ASCII.GetBytes("YourSecretKey"); // Replace with your actual secret key of Issuer var tokenValidationParameters = new TokenValidationParameters { ValidateIssuerSigningKey = true, IssuerSigningKey = new SymmetricSecurityKey(key), ValidateIssuer = false, ValidateAudience = false, ClockSkew = TimeSpan.Zero }; try { // Validate the token var principal = tokenHandler.ValidateToken(token, tokenValidationParameters, out var securityToken); // Check if the token is expired if (securityToken is JwtSecurityToken jwtSecurityToken) { if (jwtSecurityToken.ValidTo < DateTime.Now) { // Token is expired context.Response.StatusCode = (int)HttpStatusCode.Unauthorized; return; } } } catch (SecurityTokenException) { // Token validation failed context.Response.StatusCode = (int)HttpStatusCode.Unauthorized; return; } } await _next(context); } } Working fine with proper token time. Here is an example: I am providing an expired token, and it will result in a 401 Unauthorized status. You can also check the token in https://jwt.io/ for time expired (exp) . By following these steps, you can effectively implement checks to ensure that users are not able to use expired tokens within your .NET Core application.
In this blog, we are going to spread some awareness of how AI tools are replacing software engineers. Using these tools, we make our tasks much easier and smoother, but we don’t know how we are losing our dependency on the IT industry. Overview of AI Tools - AI tools, or artificial intelligence tools, offer numerous advantages across various domains. They streamline processes, enhance efficiency, and contribute to innovation. Perks include automation of repetitive tasks, data analysis for insights, improved decision-making, and the ability to handle complex computations at scale. AI tools also have applications in healthcare, finance, customer service, and more, making them invaluable for addressing challenges and driving progress. Cons. of AI Tools - While AI tools offer significant benefits, they also come with potential drawbacks. Concerns include less confidence for decision making, security for our code and other data, and Early replacement of your job. Ethics and morality are important human features that can be difficult to incorporate into an AI. The rapid progress of AI has raised several concerns that one day, AI will grow uncontrollably. And eventually, wipe out humanity. This moment is referred to as the AI singularity. The equivalent of 300 million full-time jobs could be lost to automation according to an April 2023 Report from Goldman Sachs Research. The author also estimates “that roughly two-thirds of U.S. occupations are exposed to some degree of automation by AI.” The story is complicated and tough. Economists and researchers have said many jobs will be eliminated by AI soon. These AI Tools are directly harmful to the economy of any country that is populated by the IT industry, for example, India’s IT industry has $255 billion of contribution to GDP and It could decrease now because of AI Tools and the recession that employees have to face. And it can make a country’s growth slower than other countries. Drawbacks of AI Tools Lack of creativeness. Privacy. Ethical Concerns. Lack of interpersonal skills. Unemployment. Job displacement. Increasing human laziness. Solution/Awareness to reduce usage of AI Tools Let’s search for one scenario in the AI tool and also search for that scenario in other websites which are not consist of AI Tools. Here I am using one AI Tool to develop one HTML Form with validation and design. I asked for it and it gave me the whole code which consists of all my requirements which I needed. It also provides design and script in one result. In short, it just has 2 steps copy the code and execute it, It fulfills all the requirements and sometimes it also increases some more features which shows a design that how you need and it can also give you these codes in separate files as per their usage. Here I analyze the requirements and research according to the requirements and I found these results which have all the things step-by-step. Firstly, it will explain the initial step for the HTML Form. Then it will explain the usage of all the tags that are going to be used in this process. This Image shows that how to implement the checkbox and drop-down to the HTML Form. This is the final Image, and this image shows the final result of the form with all the required things. In short, AI Tools are harmful for humans and to replace AI Tools, it just requires some observation and analysis on the particular task/project. I hope you find this blog informative and interesting so please get connected to us for more blogs like this.
Achievers Night, a highly anticipated annual event, took place recently, leaving attendees in awe with its grandeur and celebration of outstanding accomplishments. This blog aims to provide an in-depth review of this remarkable evening, highlighting the captivating performances, inspirational speeches, and overall atmosphere that made it an unforgettable experience. Inspiring Keynote Speeches: The event featured notable achievers from various fields who delivered thought-provoking and inspiring keynote speeches. These accomplished individuals shared their personal journeys, highlighting the challenges they faced and the perseverance that led to their success. Their words of wisdom and valuable insights resonated with the audience, igniting a sense of motivation and determination to pursue their own dreams and goals. Recognition of Excellence: Achievers Night served as a platform to acknowledge and honor exceptional individuals who have made significant contributions in their respective domains. The event presented awards in various categories to recognize the remarkable achievements of these deserving individuals. The atmosphere was filled with excitement and admiration as the achievers took the stage, sharing their gratitude, and inspiring others with their stories. Unforgettable Performances: The stage was set ablaze with mesmerizing performances by talented individuals, showcasing their skills and entertaining the audience throughout the night. From amazing and funny dance performances to a hilarious stand-up comedy, each act was meticulously curated. The performers demonstrated their unwavering passion, leaving the audience in awe of their incredible talents. Elegance and Ambiance: Achievers Night was not just about recognizing accomplishments; it was an enchanting experience from start to finish. The venue was adorned with elegant decor, creating a sophisticated and celebratory ambiance. From the beautifully arranged seating to the impeccable lighting and sound systems, every detail was meticulously planned to ensure a memorable evening for all attendees. Conclusion: Achievers Night surpassed all expectations, providing an enchanting and inspiring experience for everyone involved. From outstanding performances to insightful speeches and the recognition of excellence, the event celebrated the achievements of extraordinary individuals and ignited a spark of ambition within each attendee. It was an evening that left a lasting impression, reminding us that with passion, perseverance, and dedication, we can all become achievers in our own right.
In this blog, we will explore Azure Databricks, a cloud-based analytics platform, and how it can be used to parse a CSV file from Azure storage and then store the data in a database. Additionally, we will also learn how to process stream data and use Databricks notebook in Azure Data Pipeline. Azure Databricks Overview Azure Databricks is an Apache Spark-based analytics platform that provides a collaborative workspace for data scientists, data engineers, and business analysts. It is a cloud-based service that is designed to handle big data and allows users to process data at scale. Databricks also provides tools for data analysis, machine learning, and visualization. With its integration with Azure Storage, Azure Data Factory, and other Azure services, Azure Databricks can be used to build end-to-end data processing pipelines. Parsing CSV File from Azure BlobStorage to Database using Azure Databricks Azure Databricks can be used to parse CSV files from Azure Storage and then store the data in a database. Here are the steps to accomplish this: Configure Various Azure Components 1. Create Azure Resource Group Image 1 2. Create Azure DataBricks Resource Image 2 3. Create SQL Server Resource Image 3 4. Create SQL Database Resource Image 4 5. Create Azure Storage Account Image 5 6. Create Azure DataFactory Resource Image 6 7. Launch Databricks Resource Workspace Image 7 8. Create Computing Cluster Image 8 9. Create New Notebook Image 9 Parsing CSV File from Azure Storage to Database using Azure Databricks Azure Databricks can be used to parse CSV files from Azure Storage and then store the data in a database. Here are the steps to accomplish this: 1. Create a cluster: First, create a cluster in Azure Databricks as above. A cluster is a group of nodes that work together to process data. 2. Import all the necessary models in the databricks notebook %python from datetime import datetime, timedelta from azure.storage.blob import BlobServiceClient, generate_blob_sas, BlobSasPermissions import pandas as pd import pymssql import pyspark.sql Code 1 3. Mount Azure Storage: Next, mount the Azure Storage account in Databricks as follows #Configure Blob Connection storage_account_name = "storage" storage_account_access_key="***********************************" blob_container = "blob-container" Code 2 4. Establish The DataBase Connection #DB connection conn = pymssql.connect(server='****************.database.windows.net', user='*****', password='*****', database='DataBricksDB') cursor = conn.cursor() Code 3 5. Parse CSV file: Once the storage account is mounted, you can parse the CSV file using the following code #get a list of all blob from the container blob_list = [] for blob_i in container_client.list_blobs(): blob_list.append(blob_i.name) # print(blob_list) df_list = [] #Generate SAS key for each file and load to the dataframe for blob_i in blob_list: print(blob_i) sas_i = generate_blob_sas(account_name = storage_account_name, container_name = blob_container, blob_name = blob_i, account_key = storage_account_access_key, permission = BlobSasPermissions(read=True), expiry = datetime.utcnow() + timedelta(hours=12)) sas_url = 'https://' + storage_account_name +'.blob.core.windows.net/' + blob_container + '/' +blob_i print(sas_url) df=pd.read_csv(sas_url) df_list.append(df) Code 4 6. Transform and Store data in a database: Finally, you can store the data in a database using the following code #Truncate Table Sales Truncate_Query = "IF EXISTS (SELECT * FROM sysobjects WHERE name='sales' and xtype='U') truncate table sales" cursor.execute(Truncate_Query) conn.commit() # SQL Query For Table Creation create_table_query = "IF NOT EXISTS (SELECT * FROM sysobjects WHERE name='sales' and xtype='U') CREATE TABLE sales (REGION varchar(max),COUNTRY varchar(max),ITEMTYPE varchar(max),SALESCHANNEL varchar(max),ORDERPRIORITY varchar(max),ORDERDATE varchar(max),ORDERID varchar(max),SHIPDATE varchar(max),UNITSSOLD varchar(max),UNITPRICE varchar(max),UNITCOST varchar(max),TOTALREVENUE varchar(max),TOTALCOST varchar(max),TOTALPROFIT varchar(max))IF NOT EXISTS (SELECT * FROM sysobjects WHERE name='sales' and xtype='U') CREATE TABLE sales (REGION varchar(max),COUNTRY varchar(max),ITEMTYPE varchar(max),SALESCHANNEL varchar(max),ORDERPRIORITY varchar(max),ORDERDATE varchar(max),ORDERID varchar(max),SHIPDATE varchar(max),UNITSSOLD varchar(max),UNITPRICE varchar(max),UNITCOST varchar(max),TOTALREVENUE varchar(max),TOTALCOST varchar(max),TOTALPROFIT varchar(max))" cursor.execute(create_table_query) conn.commit() #Insert Data From Main DataFrame for rows in df_combined.itertuples(index=False,name=None): row = str(list(rows)) row_data = row[1:-1] row_data = row_data.replace("nan","''") row_data = row_data.replace("None","''") insert_query = "insert into sales (REGION,COUNTRY,ITEMTYPE,SALESCHANNEL,ORDERPRIORITY,ORDERDATE,ORDERID,SHIPDATE,UNITSSOLD,UNITPRICE,UNITCOST,TOTALREVENUE,TOTALCOST,TOTALPROFIT) values ("+row_data+")" print(insert_query) cursor.execute(insert_query) conn.commit() Code 5 As, Shown here The data from all the files is loaded to the SQL server Table Image 10 Azure Databricks notebook can be used to process stream data in Azure Data Pipeline. Here are the steps to accomplish this: 1. Create a Databricks notebook: First, create a Databricks notebook in Azure Databricks. A notebook is a web-based interface for working with code and data. 2. Create a job: Next, create a job in Azure Data Factory to execute the notebook. A job is a collection of tasks that can be scheduled and run automatically. 3. Configure the job: In the job settings, specify the Azure Databricks cluster and notebook that you want to use. Also, specify the input and output datasets. 4. Write the code: In the Databricks notebook, write the code to process the stream data. Here is an example code: #from pyspark.sql.functions import window stream_data = spark.readStream \ .format("csv") \ .option("header", "true") \ .schema("<schema>") \ .load("/mnt/<mount-name>/<file-name>.csv") stream_data = stream_data \ .withWatermark("timestamp", "10 minutes") \ .groupBy(window("timestamp", "10 Code 6 How To Use Azure Databrick notebook in Azure Data Factory pipeline and configure the DataFlow Pipeline Using it. Image 11 1. Create ADF Pipeline Image 12 2. Configure Data Pipeline Image 13 3. Add Trigger To the PipeLine Image 14 4. Configure the trigger Image 15 These capabilities make Azure Databricks an ideal platform for building real-time data processing solutions. Overall, Azure Databricks provides a scalable and flexible solution for data processing and analytics, and it's definitely worth exploring if you're working with big data on the Azure platform. With its powerful tools and easy-to-use interface, Azure Databricks is a valuable addition to any data analytics toolkit.
What is Docker? In this article, you will learn to build Docker image from scratch, and deploy and run your application as a Docker container using Dockerfile.Docker allows developers to build, test, and deploy applications quickly and efficiently using isolated and portable containers that run anywhere. How to Create Docker File In order to build the container image, you’ll need to use a Dockerfile. A Dockerfile is simply a text-based file with no file extension. A Dockerfile contains a script of instructions that Docker uses to create a container image.In a Dockerfile Everything on left is INSTRUCTION, and on right is an ARGUMENT to those instructions. Remember that the file name is "Dockerfile" without any extension. To create Docker file from visual studio - Open project folder in visual studio - Right click on project folder - Go to add - Go to docker support - There are two options to build docker file : windows and Linux - Select one of the given option and it will create the docker file Here we have selected windows operating system, so it will create docker file for windows image but if you select Linux operating system then it will create docker file for Linux image and rest of the docker commands will be same for windows as well as Linux How to Build Docker Image We will build our image using the Docker command. The below command will build the image using Dockerfile from the same directory. docker build -t demoimage:1.0 . - t is for tagging the image. - demoimage is the name of the image. - 1.0 is the tag name. If you don’t add any tag, it defaults to the tag named latest. - . means, we are referring to the Dockerfile location as the docker build context. After the image build output will look like below. Now, we can list the images by using this command. docker images Test the Docker Image Now after building the image we will run the Docker image. The command will be, docker run -d -p 4000:80 --name democontainer2 demoimage:1.0 - d flag is for running the container in detached mode. - p flag flag for the port number, the format is local-port:container-port. - --name for the container name, democontainer2 in our case. Docker started our container in the background and printed the Container ID on the terminal. We can check the running container by using the below command. docker ps In a web browser, access http://localhost:4000 and we can see the index page which displays the content in the custom HTML page we added to the docker image. After creating Docker image we can see all the local images in Docker windows Desktop. Go to the images tab and we can see all the images. Go to the containers tab and we can see all the containers also. Here we are using docker desktop for windows,so we have selected switch to windows option,if you are using docker desktop for Linux then select switch to Linux option. Push Docker Image to Docker Hub Docker Hub is a registry service on the cloud that allows you to download Docker images that are built by other communities. You can also upload your own Docker built images to Docker hub.To push our Docker image to the Docker hub, we need to create an account in the Docker hub. After that, execute the below command to log in from the terminal. It will ask for a username and password (if you are login for the first time). Provide the Docker hub credentials. docker login After login, we now need to tag our image with the docker username as shown below. docker tag demoimage:1.0 <username>/<image-name>:tag For example, here hiral1 is the docker hub username. docker tag demoimage:1.0 hiral1/demoimage:1.0 Run docker images command again and check the tagged image will be there. Now we can push our images to the Docker hub using the below command. docker push hiral1/demoimage:1.0 Now we can check this image will be available in our Docker Hub account. We can inspect a container by following command. docker inspect <container-id> We can view Docker logs in a Docker container by following command. docker logs <container-id> And we can stop the running container by following command. docker stop <container-id> Pull and Run Docker Image from Docker Hub To pull image from docker hub use the following command. docker pull <imagename:tag> Docker checks if the image already exists or not, if it is then it does not download further.In our case it is already there.So we need to remove existing image. To remove the docker image use the following command. docker rmi <imagename:tag> Now pull the docker image from docker hub and check for the list of images. To run the pulled image use the below command docker run -d -p 4000:80 --name windowscontainer hiral1/demoimage:1.0 - d flag is for running the container in detached mode. - p flag flag for the port number, the format is local-port:container-port. - --name for the container name, windowscontainer in our case. In a web browser, access http://localhost:4000 and we can see the index page which displays the content in the custom HTML page we added to the docker image. Below is the example of how to pull and run docker image on Linux VM. Here is the output of the docker image on the browser. Rename/Tag Docker Image To rename the docker image tag the image as follows. docker tag <oldimagename:tag> <newimagename:tag> In our case the old image name is hiral1/demoimage , old tag is 1.0 and new image name is newimage/latest.We have not provided a new tag to newimage, so it gives tag - latest by default. Remove Docker Image To remove the docker image use the following command. docker rmi <imagename:tag>
Abstract This article describes a TSQL JSON parser and provides the source. It is also designed to illustrate a number of string manipulation techniques and also eliminate the issues while dealing with the JSON document containing special symbols like (“/” , ”-”....) in T-SQL. With it you can do things like this to extract the data from a JSON file or document which contains noise and complexities. Summary For Implementation The code for the JSON Parser will run in SQL Server 2005, and even in SQL Server 2000 (note: some modifications are necessary). First the function stores all strings in the temporary table, even the name of the elements, since they are 'escapes' in a different way, and may contain, unescaped, brackets, Special Characters which denote objects or lists. These are replaced in the json string by tokens which represent the strings. After this fetch all the json keywords and values for further processing by using the regular expressions, various string functions and a list of SQL queries and variables to store the values for a particular object. And at the last function will return a whole table which contains rows and columns with no noise in the values as the other tables in the particular database. Figure 1:- Json Input Figure 2:- Function Output Background TSQL isn’t really designed for doing complex string parsing which contains special characters and particularly where strings represent nested data structures such as XML, JSON, or XHTML. You can do it but it is not a pretty sight; but If you ever want to do it anyway ? (note You can now do this rather more easily using SQL Server 2016’s built-in JSON support.) But If the SQL Server version is older or not compatible with the built-in JSON support then you can use this customized function to get the desired output by parsing any type of json document. There is so much stuff behind that all happens to you. For example, it could be that DBA doesn’t allow a CLR, or you lack the necessary skills with procedural scripting. Sometimes, there isn’t any application, or you want to run code unobtrusively across databases or servers. The Traditional way for dealing with data like this is to let a separate business layer parse a JSON ‘document’ into some meaningful structure(Like Tree) and then update the database by making a series of calls and lots of sql procedures. This is pretty, but can get more complicated and headache if you need to ensure that the updates to the database are wrapped into one transaction so that if anything goes wrong or any issues occur, then the whole transaction can be rolled back. This is why a TSQL approach has advantages. Adjacency list tables have the same structure whatever the data in them. This means that you can define a single Table-Valued Type and pass data structures around between stored procedures. Converting the data to Hierarchical table form will be different for each application, but is easy with a TSQL. You can, alternatively, convert the hierarchical table into JSON and interrogate that with SQL. JSON format JSON is one of the most popular lightweight markup languages, and is probably the best choice for transfer of object data from a web page. JSON is designed to be as lightweight as possible and so it has only two structures. The first, delimited by curly brackets, is a collection of Key/value pairs, separated by commas. The key is followed by a colon. The first snag for TSQL is that the curly or square brackets are not ‘escaped’ within a string, so that there is no way of partitioning a JSON ‘document’ simply. It is difficult to differentiate a bracket used as the delimiter of an array or structure, and one that is within a string. The second complication is that, unlike YAML, the datatypes of values can’t be explicitly declared. You have to pass them out from applying the rules from the JSON Specification. Implementation The JSON outputter is a great deal simpler, since one can be sure of the input, but essentially it does the reverse process, working from the root of the json document to the leaves. The only complication is working out the indent of the formatted output string. In the implementation, you’ll see a fairly heavy use of PATINDEX.This uses a RegEx. However, it is all we have, and can be pressed into service by chopping the string it is searching (if only it had an optional third parameter like CHARINDEX that specified the index of the start position of the search!). The STUFF function is also important for this sort of string-manipulation work. CREATE FUNCTION [Platform].[parseJSON] (@JSON NVARCHAR(MAX)) RETURNS @hierarchy TABLE ( Element_ID INT IDENTITY(1, 1) NOT NULL /* internal surrogate primary key gives the order of parsing and the list order */ ,SequenceNo [int] NULL /* the place in the sequence for the element */ ,Parent_ID INT NULL /* if the element has a parent then it is in this column. The document is the ultimate parent, so you can get the structure from recursing from the document */ ,[Object_ID] INT NULL /* each list or object has an object id. This ties all elements to a parent. Lists are treated as objects here */ ,[Name] NVARCHAR(2000) NULL /* the Name of the object */ ,StringValue NVARCHAR(MAX) NOT NULL /*the string representation of the value of the element. */ ,ValueType VARCHAR(10) NOT NULL /* the declared type of the value represented as a string in StringValue*/ ) AS BEGIN DECLARE @FirstObject INT --the index of the first open bracket found in the JSON string ,@OpenDelimiter INT --the index of the next open bracket found in the JSON string ,@NextOpenDelimiter INT --the index of subsequent open bracket found in the JSON string ,@NextCloseDelimiter INT --the index of subsequent close bracket found in the JSON string ,@Type NVARCHAR(10) --whether it denotes an object or an array ,@NextCloseDelimiterChar CHAR(1) --either a '}' or a ']' ,@Contents NVARCHAR(MAX) --the unparsed contents of the bracketed expression ,@Start INT --index of the start of the token that you are parsing ,@end INT --index of the end of the token that you are parsing ,@param INT --the parameter at the end of the next Object/Array token ,@EndOfName INT --the index of the start of the parameter at end of Object/Array token ,@token NVARCHAR(200) --either a string or object ,@value NVARCHAR(MAX) -- the value as a string ,@SequenceNo INT -- the sequence number within a list ,@Name NVARCHAR(200) --the Name as a string ,@Parent_ID INT --the next parent ID to allocate ,@lenJSON INT --the current length of the JSON String ,@characters NCHAR(36) --used to convert hex to decimal ,@result BIGINT --the value of the hex symbol being parsed ,@index SMALLINT --used for parsing the hex value ,@Escape INT --the index of the next escape character /* in this temporary table we keep all strings, even the Names of the elements, since they are 'escaped' in a different way, and may contain, unescaped, brackets denoting objects or lists. These are replaced in the JSON string by tokens representing the string */ DECLARE @Strings TABLE ( String_ID INT IDENTITY(1, 1) ,StringValue NVARCHAR(MAX) ) IF ISNULL(@JSON, '') = '' RETURN SELECT @characters = '0123456789abcdefghijklmnopqrstuvwxyz' --initialise the characters to convert hex to ascii ,@SequenceNo = 0 --set the sequence no. to something sensible. ,@Parent_ID = 0; /* firstly we process all strings. This is done because [{} and ] aren't escaped in strings, which complicates an iterative parse. */ WHILE 1 = 1 --forever until there is nothing more to do BEGIN SELECT @start = PATINDEX('%[^a-zA-Z]["]%', @json collate SQL_Latin1_General_CP850_Bin);--next delimited string IF @start = 0 BREAK --no more so drop through the WHILE loop IF SUBSTRING(@json, @start + 1, 1) = '"' BEGIN --Delimited Name SET @start = @Start + 1; SET @end = PATINDEX('%[^\]["]%', RIGHT(@json, LEN(@json + '|') - @start) collate SQL_Latin1_General_CP850_Bin); END IF @end = 0 --either the end or no end delimiter to last string BEGIN -- check if ending with a double slash... SET @end = PATINDEX('%[\][\]["]%', RIGHT(@json, LEN(@json + '|') - @start) collate SQL_Latin1_General_CP850_Bin); IF @end = 0 --we really have reached the end BEGIN BREAK --assume all tokens found END END SELECT @token = SUBSTRING(@json, @start + 1, @end - 1) --now put in the escaped control characters SELECT @token = REPLACE(@token, FromString, ToString) FROM ( SELECT '\b' ,CHAR(08) UNION ALL SELECT '\f' ,CHAR(12) UNION ALL SELECT '\n' ,CHAR(10) UNION ALL SELECT '\r' ,CHAR(13) UNION ALL SELECT '\t' ,CHAR(09) UNION ALL SELECT '\"' ,'"' UNION ALL SELECT '\/' ,'/' ) substitutions(FromString, ToString) SELECT @token = Replace(@token, '\\', '\') SELECT @result = 0 ,@escape = 1 --Begin to take out any hex escape codes WHILE @escape > 0 BEGIN SELECT @index = 0 --find the next hex escape sequence ,@escape = PATINDEX('%\x[0-9a-f][0-9a-f][0-9a-f][0-9a-f]%', @token collate SQL_Latin1_General_CP850_Bin) IF @escape > 0 --if there is one BEGIN WHILE @index < 4 --there are always four digits to a \x sequence BEGIN SELECT --determine its value @result = @result + POWER(16, @index) * (CHARINDEX(SUBSTRING(@token, @escape + 2 + 3 - @index, 1), @characters) - 1) ,@index = @index + 1; END -- and replace the hex sequence by its unicode value SELECT @token = STUFF(@token, @escape, 6, NCHAR(@result)) END END --now store the string away INSERT INTO @Strings (StringValue) SELECT @token -- and replace the string with a token SELECT @JSON = STUFF(@json, @start, @end + 1, '@string' + CONVERT(NCHAR(5), @@identity)) END -- all strings are now removed. Now we find the first leaf. WHILE 1 = 1 --forever until there is nothing more to do BEGIN SELECT @Parent_ID = @Parent_ID + 1 --find the first object or list by looking for the open bracket SELECT @FirstObject = PATINDEX('%[{[[]%', @json collate SQL_Latin1_General_CP850_Bin) --object or array IF @FirstObject = 0 BREAK IF (SUBSTRING(@json, @FirstObject, 1) = '{') SELECT @NextCloseDelimiterChar = '}' ,@type = 'object' ELSE SELECT @NextCloseDelimiterChar = ']' ,@type = 'array' SELECT @OpenDelimiter = @firstObject WHILE 1 = 1 --find the innermost object or list... BEGIN SELECT @lenJSON = LEN(@JSON + '|') - 1 --find the matching close-delimiter proceeding after the open-delimiter SELECT @NextCloseDelimiter = CHARINDEX(@NextCloseDelimiterChar, @json, @OpenDelimiter + 1) --is there an intervening open-delimiter of either type SELECT @NextOpenDelimiter = PATINDEX('%[{[[]%', RIGHT(@json, @lenJSON - @OpenDelimiter) collate SQL_Latin1_General_CP850_Bin) --object IF @NextOpenDelimiter = 0 BREAK SELECT @NextOpenDelimiter = @NextOpenDelimiter + @OpenDelimiter IF @NextCloseDelimiter < @NextOpenDelimiter BREAK IF SUBSTRING(@json, @NextOpenDelimiter, 1) = '{' SELECT @NextCloseDelimiterChar = '}' ,@type = 'object' ELSE SELECT @NextCloseDelimiterChar = ']' ,@type = 'array' SELECT @OpenDelimiter = @NextOpenDelimiter END ---and parse out the list or Name/value pairs SELECT @contents = SUBSTRING(@json, @OpenDelimiter + 1, @NextCloseDelimiter - @OpenDelimiter - 1) SELECT @JSON = STUFF(@json, @OpenDelimiter, @NextCloseDelimiter - @OpenDelimiter + 1, '@' + @type + CONVERT(NCHAR(5), @Parent_ID)) WHILE (PATINDEX('%[A-Za-z0-9@+.e]%', @contents collate SQL_Latin1_General_CP850_Bin)) <> 0 BEGIN IF @Type = 'object' --it will be a 0-n list containing a string followed by a string, number,boolean, or null BEGIN SELECT @SequenceNo = 0 ,@end = CHARINDEX(':', ' ' + @contents) --if there is anything, it will be a string-based Name. SELECT @start = PATINDEX('%[^A-Za-z@][@]%', ' ' + @contents collate SQL_Latin1_General_CP850_Bin) --AAAAAAAA SELECT @token = RTrim(Substring(' ' + @contents, @start + 1, @End - @Start - 1)) ,@endofName = PATINDEX('%[0-9]%', @token collate SQL_Latin1_General_CP850_Bin) ,@param = RIGHT(@token, LEN(@token) - @endofName + 1) SELECT @token = LEFT(@token, @endofName - 1) ,@Contents = RIGHT(' ' + @contents, LEN(' ' + @contents + '|') - @end - 1) SELECT @Name = StringValue FROM @strings WHERE string_id = @param --fetch the Name END ELSE SELECT @Name = NULL ,@SequenceNo = @SequenceNo + 1 SELECT @end = CHARINDEX(',', @contents) -- a string-token, object-token, list-token, number,boolean, or null IF @end = 0 --HR Engineering notation bugfix start IF ISNUMERIC(@contents) = 1 SELECT @end = LEN(@contents) + 1 ELSE --HR Engineering notation bugfix end SELECT @end = PATINDEX('%[A-Za-z0-9@+.e][^A-Za-z0-9@+.e]%', @contents + ' ' collate SQL_Latin1_General_CP850_Bin) + 1 SELECT @start = PATINDEX('%[^A-Za-z0-9@+.e][-A-Za-z0-9@+.e]%', ' ' + @contents collate SQL_Latin1_General_CP850_Bin) --select @start,@end, LEN(@contents+'|'), @contents SELECT @Value = RTRIM(SUBSTRING(@contents, @start, @End - @Start)) ,@Contents = RIGHT(@contents + ' ', LEN(@contents + '|') - @end) IF SUBSTRING(@value, 1, 7) = '@object' INSERT INTO @hierarchy ( [Name] ,SequenceNo ,Parent_ID ,StringValue ,[Object_ID] ,ValueType ) SELECT @Name ,@SequenceNo ,@Parent_ID ,SUBSTRING(@value, 8, 5) ,SUBSTRING(@value, 8, 5) ,'object' ELSE IF SUBSTRING(@value, 1, 6) = '@array' INSERT INTO @hierarchy ( [Name] ,SequenceNo ,Parent_ID ,StringValue ,[Object_ID] ,ValueType ) SELECT @Name ,@SequenceNo ,@Parent_ID ,SUBSTRING(@value, 7, 5) ,SUBSTRING(@value, 7, 5) ,'array' ELSE IF SUBSTRING(@value, 1, 7) = '@string' INSERT INTO @hierarchy ( [Name] ,SequenceNo ,Parent_ID ,StringValue ,ValueType ) SELECT @Name ,@SequenceNo ,@Parent_ID ,StringValue ,'string' FROM @strings WHERE string_id = SUBSTRING(@value, 8, 5) ELSE IF @value IN ('true', 'false') INSERT INTO @hierarchy ( [Name] ,SequenceNo ,Parent_ID ,StringValue ,ValueType ) SELECT @Name ,@SequenceNo ,@Parent_ID ,@value ,'boolean' ELSE IF @value = 'null' INSERT INTO @hierarchy ( [Name] ,SequenceNo ,Parent_ID ,StringValue ,ValueType ) SELECT @Name ,@SequenceNo ,@Parent_ID ,@value ,'null' ELSE IF PATINDEX('%[^0-9-]%', @value collate SQL_Latin1_General_CP850_Bin) > 0 INSERT INTO @hierarchy ( [Name] ,SequenceNo ,Parent_ID ,StringValue ,ValueType ) SELECT @Name ,@SequenceNo ,@Parent_ID ,@value ,'real' ELSE INSERT INTO @hierarchy ( [Name] ,SequenceNo ,Parent_ID ,StringValue ,ValueType ) SELECT @Name ,@SequenceNo ,@Parent_ID ,@value ,'int' IF @Contents = ' ' SELECT @SequenceNo = 0 END END INSERT INTO @hierarchy ( [Name] ,SequenceNo ,Parent_ID ,StringValue ,[Object_ID] ,ValueType ) SELECT '-' ,1 ,NULL ,'' ,@Parent_ID - 1 ,@type RETURN END Code Snippet 1:- ParseJson Function Closure The so-called ‘impedance-mismatch’ between applications and databases is an illusion. if the developer has understood the data correctly then there is less complexity while processing it. But has been trickier with other formats such as JSON. By using techniques like this, it should be possible to liberate the application or website from having to do the mapping from the object model to the relational, and spraying the database with ad-hoc T-SQL that uses the fact/dimension tables or updateable views. If the database can be provided with the JSON, or the Table-Valued parameter, then there is a better chance of maintaining full transactional integrity for the more complex updates. The database developer already has the tools to do the work with XML, but why not the simpler, and more practical JSON? I hope these routines get you started with experimenting with all this for your requirements.
