{"id":13365,"date":"2017-07-06T20:34:00","date_gmt":"2017-07-06T20:34:00","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/premier_developer\/?p=13365"},"modified":"2019-03-05T12:44:39","modified_gmt":"2019-03-05T19:44:39","slug":"computer-vision-made-easy-with-cognitive-services","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/premier-developer\/computer-vision-made-easy-with-cognitive-services\/","title":{"rendered":"Computer Vision Made Easy with Cognitive Services"},"content":{"rendered":"<p>This post is provide by App Dev Manager, <a href=\"https:\/\/www.linkedin.com\/in\/andrewkanieski\/\">Andrew Kanieski<\/a>, who takes K9 obedience training to a new level with computer vision and Cognitive Services.<\/p>\n<hr \/>\n<p>Computer vision is truly amazing technology! It can be used to distill rich information from images that can help you breathe life into your applications. Microsoft&#8217;s Cognitive Services &#8211; Computer Vision API gives developers the power to develop their software with the ability to truly see and comprehend the world around it.<\/p>\n<p>When I come upon a new technology I try to make application in my life. I ask myself how could I see myself using this? There are many ways in which we can use Computer Vision to make our day to day lives easier!<\/p>\n<h3><strong>Context<\/strong>: <em>Enter the family dog<\/em><\/h3>\n<p>One problem in my home we always struggle with is keeping our family dog\n<em>out<\/em> of our bedrooms. As much as we train him, he always sneaks in to nap on our beds. How can we write software to help us train our family friend?<\/p>\n<h3><strong>Problem<\/strong>: <em>I need to be alerted when our dog enters a room<\/em><\/h3>\n<p>This is where I applied my new found knowledge of the Computer Vision API. What if every time my dog entered our bedroom I was alerted via SMS message?<\/p>\n<h5>Prerequisites<\/h5>\n<ul>\n<li>IP Camera (or other means of collecting images) &#8211; I am using an\n<a href=\"https:\/\/amcrest.com\/amcrest-1080p-wifi-video-security-ip-camera-pt.html\">Amcrest IP2M-841B<\/a><\/li>\n<li>.NET Core Installed &#8211; Install Instructions\n<a href=\"https:\/\/www.microsoft.com\/net\/core#windowscmd\">Here<\/a><\/li>\n<li>VSCode (or other editor) &#8211; Install Instructions\n<a href=\"https:\/\/code.visualstudio.com\/\">Here<\/a><\/li>\n<li>Twilio Account (Trial or Subscriber) &#8211; More Info on Twilio + Azure\n<a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/twilio-dotnet-how-to-use-for-voice-sms\">Here<\/a><\/li>\n<\/ul>\n<h3>Assuming you have .NET Core installed, let\u2019s start by creating a new .NET Core Console Application:<\/h3>\n<p>From console:<\/p>\n<pre class=\"lang:default decode:true\">dotnet new console\r\ndotnet restore<\/pre>\n<p>This will create our initial application with a Program.cs and a dog-watcher.csproj<\/p>\n<p>In Program.cs I added the needed variables that will be passed into our application via Environment. Other means of storing and working with configuration are also fine. I just happen to prefer environment variables.<\/p>\n<pre class=\"lang:default decode:true \">static string IP_CAMERA_SNAPSHOT_URL = System.Environment.GetEnvironmentVariable(\"IP_CAMERA_SNAPSHOT_URL\");\r\nstatic string IP_CAMERA_USER = System.Environment.GetEnvironmentVariable(\"IP_CAMERA_USER\");\r\nstatic string IP_CAMERA_PASSWORD = System.Environment.GetEnvironmentVariable(\"IP_CAMERA_PASSWORD\");\r\n<\/pre>\n<p>In this case I am using an IP based camera that provides you with a REST API for getting a snapshot from the camera via HTTP.<\/p>\n<p>Now that we have our needed configuration collected, let\u2019s begin by creating logic to fetch our snapshots. Our IP camera uses HTTP GET with basic auth.<\/p>\n<p>First, lets add System.Net.Http;<\/p>\n<pre class=\"lang:default decode:true \">dotnet add package System.Net.Http<\/pre>\n<p>Now you can go ahead and use the System.Net.Http namespace.<\/p>\n<p>Program.cs<\/p>\n<pre class=\"lang:default decode:true \">static byte[] GetSnapshotFromIPCamera(string url) {\r\n        Client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue(\"Basic\", System.Convert.ToBase64String(System.Text.ASCIIEncoding.ASCII.GetBytes(IP_CAMERA_USER + \":\" + IP_CAMERA_PASSWORD)));\r\n        return Client.GetByteArrayAsync(url).Result;\r\n    }\r\n<\/pre>\n<p>Now that we have logic to collect images, let\u2019s create some classes that match with the response from the Computer Vision API. This is only mapping Tags since that is the data element we are working with. In this case tags will provide us with a list of descriptors that computer vision has identified and with what level of confidence it feels those descriptors are identified.<\/p>\n<pre class=\"lang:default decode:true\">using Newtonsoft.Json;\r\n \r\nnamespace Vision {\r\n    public class Response {\r\n        [JsonProperty(\"tags\")]\r\n        public Tag[] Tags {get;set;}\r\n    }\r\n    public class Tag {\r\n        [JsonProperty(\"name\")]\r\n        public string Name {get;set;}\r\n        [JsonProperty(\"confidence\")]\r\n        public float Confidence {get;set;}\r\n    }\r\n}\r\n<\/pre>\n<p>Now let\u2019s post those images to Cognitive Services for analysis. Notice the use of the variable AZURE_VISION_URL and AZURE_VISION_KEY these are coming directly from the Azure Portal after we\u2019ve gone and added our Coginitive Services subscription.<\/p>\n<pre class=\"lang:default decode:true\">static string AZURE_VISION_URL = System.Environment.GetEnvironmentVariable(\"AZURE_VISION_URL\");\r\nstatic string AZURE_VISION_KEY = System.Environment.GetEnvironmentVariable(\"AZURE_VISION_KEY\");\r\n       \r\nstatic Vision.Response GetVisionAnalysisResponse(byte[] image) {\r\n    Client.DefaultRequestHeaders.Add(\"Ocp-Apim-Subscription-Key\", AZURE_VISION_KEY);\r\n    using (var content = new System.Net.Http.ByteArrayContent(image))\r\n    {\r\n        content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue(\"application\/octet-stream\");\r\n        string res = Client.PostAsync(AZURE_VISION_URL + @\"\/analyze?visualFeatures=Tags, Description\", content).Result.Content.ReadAsStringAsync().Result;\r\n        return JsonConvert.DeserializeObject&lt;Vision.Response&gt;(res);\r\n    }\r\n}\r\n<\/pre>\n<p>Now let\u2019s wire up the loop that watches our room.<\/p>\n<pre class=\"lang:default decode:true \">static int INTERVAL = String.IsNullOrEmpty(System.Environment.GetEnvironmentVariable(\"INTERVAL\")) ? 3000 : Convert.ToInt32(System.Environment.GetEnvironmentVariable(\"INTERVAL\"));\r\n \r\nstatic void Main(string[] args)\r\n{\r\n    while(true) {\r\n        try {\r\n            var res = GetVisionAnalysisResponse(GetSnapshotFromIPCamera(IP_CAMERA_SNAPSHOT_URL));\r\n \r\n            \/\/ Only trigger alert logic when CV is 90% sure a dog has been spotted!\r\n            if (res.Tags\r\n                    .Where(t =&gt; t.Name.ToUpper().Equals(\"DOG\") &amp;&amp; t.Confidence &gt; 0.90)\r\n                    .Count() &gt; 0) {\r\n                \/\/ TODO: ADD LOGIC FOR WHEN DOG IS FOUND IN THE ROOM\r\n            }\r\n \r\n        } catch (Exception ex) {\r\n            Console.WriteLine(ex);\r\n        }\r\n        Console.WriteLine(\"Waiting for next interval\");\r\n        System.Threading.Thread.Sleep(INTERVAL);\r\n    }\r\n}\r\n<\/pre>\n<p><em>Note<\/em>: that trial Twilio accounts can only send SMS messages to verified devices.<\/p>\n<pre class=\"lang:default decode:true \">dotnet add package Twilio<\/pre>\n<p><strong>Program.cs<\/strong><\/p>\n<pre class=\"lang:default decode:true \">static string TWILIO_API_KEY = System.Environment.GetEnvironmentVariable(\"TWILIO_API_KEY\");\r\nstatic string TWILIO_API_TOKEN = System.Environment.GetEnvironmentVariable(\"TWILIO_API_TOKEN\");\r\nstatic string TWILIO_API_PHONE = System.Environment.GetEnvironmentVariable(\"TWILIO_API_PHONE\");\r\n \r\nstatic void Main(string[] args)\r\n{\r\n    while(true) {\r\n        try {\r\n            var res = GetVisionAnalysisResponse(GetSnapshotFromIPCamera(IP_CAMERA_SNAPSHOT_URL));\r\n \r\n            \/\/ Only trigger alert logic when CV is 90% sure a dog has been spotted!\r\n            if (res.Tags\r\n                    .Where(t =&gt; t.Name.ToUpper().Equals(\"DOG\") &amp;&amp; t.Confidence &gt; 0.90)\r\n                    .Count() &gt; 0) {\r\n                Twilio.TwilioClient.Init(TWILIO_API_KEY, TWILIO_API_TOKEN);\r\n                Twilio.Rest.Api.V2010.Account.MessageResource.Create(\r\n                    new Twilio.Types.PhoneNumber(TARGET_PHONE), \r\n                    from: new Twilio.Types.PhoneNumber(TWILIO_API_PHONE), \r\n                    body: \"It looks like Jack has made his way into your room!\"\r\n                );\r\n            }\r\n \r\n        } catch (Exception ex) {\r\n            Console.WriteLine(ex);\r\n        }\r\n        Console.WriteLine(\"Waiting for next interval\");\r\n        System.Threading.Thread.Sleep(INTERVAL);\r\n    }\r\n}\r\n<\/pre>\n<p>Now we can simply adjust our camera for ideal viewing angle and run our application.<\/p>\n<h3>Conclusion<\/h3>\n<p>Working with Microsoft\u2019s Cognitive Services is an easy way to build intelligent applications. Additionally, Azure provides a seamless hosting environment for applications that interact with Cognitive Services.<\/p>\n<hr \/>\n<p><a href=\"https:\/\/blogs.msdn.com\/b\/premier_developer\/archive\/2014\/09\/15\/welcome.aspx\"><strong>Premier Support for Developers<\/strong><\/a> provides strategic technology guidance, critical support coverage, and a range of essential services to help teams optimize development lifecycles and improve software quality.\u00a0 Contact your Application Development Manager (ADM) or <a href=\"https:\/\/blogs.msdn.microsoft.com\/premier_developer\/contact-us\/\">email us<\/a><b><\/b> to learn more about what we can do for you.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This post is provide by App Dev Manager, Andrew Kanieski, who takes K9 obedience training to a new level with computer vision and Cognitive Services. Computer vision is truly amazing technology! It can be used to distill rich information from images that can help you breathe life into your applications. Microsoft&#8217;s Cognitive Services &#8211; Computer [&hellip;]<\/p>\n","protected":false},"author":582,"featured_media":37840,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[25],"tags":[24,72,3],"class_list":["post-13365","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure","tag-azure","tag-cognitive-services","tag-team"],"acf":[],"blog_post_summary":"<p>This post is provide by App Dev Manager, Andrew Kanieski, who takes K9 obedience training to a new level with computer vision and Cognitive Services. Computer vision is truly amazing technology! It can be used to distill rich information from images that can help you breathe life into your applications. Microsoft&#8217;s Cognitive Services &#8211; Computer [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/posts\/13365","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/users\/582"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/comments?post=13365"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/posts\/13365\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/media\/37840"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/media?parent=13365"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/categories?post=13365"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/tags?post=13365"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}