Wednesday, November 19, 2014

Machine Learning Week 1 (Notes)

(Here are my notes from Week 1 of Stanford's Machine learning class by Andrew Ng)


Definition:    ”Algorithms for inferring unknowns from knowns”


  • Database Mining
    • study large data sets like web click data, medical records, biology and engineering data
  • Application which can’t be program by hand
    • autonomous helicopters, handwriting recognition, natural language processing, computer vision
  • Self Customizing programs
    • Amazon and Netflix recommendations
  • Understanding human learning
    • Real Artificial intelligence

Tom Mitchell provides a more modern definition: "A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E."

Example: playing checkers.

  • E = the experience of playing many games of checkers
  • T = the task of playing checkers.
  • P = the probability that the program will win the next game.

Supervised Learning

  1. Regression


    Size in feet is a feature.

  2. Classification



    Tumor size is a feature but there can be multiple other features like clump thickness, uniformity of cell size etc…
    How to create an algorithm which can deal with “infinite” number of features ???

Unsupervised Learning


  • No structure is given
  • We need to cluster the Data, examples:
    • Google news
    • Organize computing clusters
    • social network analysis
    • market segmentation
    • astronomical data analysis
    • gene classification for individuals


Separate out the noises from two voices….


Linear Regression With One Variable Model


Hypothesis is the prediction of the value y based on x, given the theta.

How to choose the theta ????

Cost Function


Cost function is also known as the “Squared Error function”

Idea : Choose theta 0, theta 1 so that h theta ( x ) is close to y for our training examples (x , y)


Minimize the J (theta 0, theta 1) so that our answer is close to y.


Another way of presenting the hypothesis:
Minimize the J (theta 0, theta 1) so that our answer is close to y.




Gradient Descent Algorithm


Graphical representation:


One of the side effects of using this algorithm is that if we change the starting point slightly than we end up with a different “Local Optima”


Use this “gradient descent” algorithm to minimize the “cost function” in linear regression model.



  1. It is important to perform a “Simultaneous Update” on the j 0 and j 1 theta’s.
  2. Alpha = learning rate.
  3. If Alpha is too small, gradient descent can be slow
  4. If Alpha is too large, gradient descent can overshoot the minimum. It may fail to converge or even diverge.
  5. There can be multiplelocal optima
  6. There is only 1global optima”. which is the minimum from the entire set
  7. Cost function is “bowl” shape so it only has global optima.
  8. Gradient descent can converge to a local minimum, even with the learning rate α fixed.


To calculate for multiple features, convert the X and Y into Matrix and Vector:


Wednesday, November 12, 2014

.Net on OS X and Linux


Today at the Connect() developer event Microsoft announced something amazing. Here are the 2 blog posts from 2 Scott’s explaining it with more details:

Announcing .NET 2015 - .NET as Open Source, .NET on Mac and Linux, and Visual Studio Community

Announcing Open Source of .NET Core Framework, .NET Core Distribution for Linux/OSX, and Free Visual Studio Community Edition

Indeed, it is an exciting time to be a developer….

Sunday, October 05, 2014

Playing with Light Sensors (Smart Night Lamp)

After the Avengers Emergency Helpline, here is another Hardware App. I call it “Smart Night Lamp”.
It uses photoresistors and Pulse Width Modulation (PWM) to switch on & off the RGB LED based on surrounding light.

Circuit Picture:


Click here to see the video of it working….



Thursday, July 24, 2014

Free Online Computer Science courses

Recently I came across this excellent post on aGupieWare’s blog listing down all the free online computer science course materials.
Here is the complete list copied AS-IS from the post:

Introductory Courses

Intro to Computer Science:

Mathematics: Programming: Theory of Computation: Data Structures and Algorithms:

Core Courses

Theory: Algorithms and Data Structures: Mathematics: Operating Systems: Computer Programming: Software Engineering: Computer Architecture: Data Management: Networking and Data Communications: Cryptography and Security: Artificial Intelligence:

Intermediate and Advanced Courses

Algorithms and Data Structures: Systems: Programming: Software Engineering: Mobile App Development: Web Development: Databases and Data Management: Security: Cryptography: Artificial Intelligence and Machine Learning: Natural Language Processing: Digital Media: Networking and Communications: Statistics and Probability:

I went through the Harvard CS course before few years and it was lot of fun.

Enjoy Learning…

Wednesday, July 02, 2014

Patterns for Cloud Apps on Azure

I have been working on a lot of new Azure projects and here are some really good articles, posters and video's that I found very useful around best practices/ patterns.

Azure Rocks !!!

Tuesday, May 13, 2014

Tuesday, April 15, 2014

Web Essentials and Visual Studio 2013 Update 2 (RC)

Here are some of the cool new features in the Web Essentials and VS 2013 Update 2 (RC):

  • Create Azure Websites and Virtual Machine from Visual Studio

  • JavaScript code and style analysis via JsHint and JSCS

  • Manage Virtual Machines from Server Explorer

  • Image Previews

  • Create / Update  Image Sprites

  • Minify/Bundle JavaScript & HTML


There are ton of new exciting features. I would recommend to watch this “Fun” filled session presented by Scott Hanselman and Scott Hunter to get a good overview of what is in the release and what’s coming….

Thursday, April 03, 2014

Build 2014 - Day 2 (Notes)



  • 57% of fortune 500 companies are using Azure. More than 300 million active directory users. Over 1 Million SQL Databases.
  • Azure powered the NDC Sochi olympics live media streaming
  • Support for Puppet and Chef infrastructure configuration tools
  • New features on Azure IaaS
    • Create and Manage VM’s from Visual Studio
    • Enable Debugging from VS and attach it to the VM
    • Azure portal now has plugins and agents for Puppet and Chef integrated in the VM Creation wizard
    • Puppet Enterprise Console demo (load a class for microsoft-sysinternals)
    • Speech from CEO of Getty Images. They use puppet and azure heavily.
    • Autocale feature is now available
  • New features on Azure Paas
    • Web
      • Azure websites will now support “Java”
      • Azure websites “Staging” feature
      • Traffic Manager support. Intelligent customer routing. It will work with websties as well virtual machines.
      • New options on the VS web project creation screen to create websites directly from there
      • Sync CSS changes from IE DOM Explorer to Visual Studio CSS file. This works in Chrome and IE
      • Sync the editing of HTML tag values with HTML files in Visual Studio. (LIVE !!!!)
      • JsHint integrated for VS JavaScript debugging
      • WebJobs can be simple console applications which can listen to events fro Queue and Blob updates
      • Free SSL certificates for Websites
      • CDN generally available now
    • Mobile
      • Mobile services now support Active Directory
      • New VS Project Template for creating Mobile Service
      • Support for Local and Remote debugging for Mobile Services
      • New Mobile Services .Net API’s which can be used in all apps.
      • Demo of win 8 app with portable library + sharepoint document upload + xamarian to port the app to iOS
      • Demo of “DocuSign” app on iPhone. Built using ,Net and SQL Server.
      • “Offline Data Sync” features for Mobile Services
      • Azure AD Premium offerings
      • Kindle support for Notification Hub
    • Data
      • SQL Azure Database now supports 500 GB + 99.95% SLA
      • Self Service restore feature ( for emergenices like DELETE from tablename Smile) is now builtin
      • Active geo replication feature. Primary and Secondary region data replication.
      • Updates to HDInsight. Supports Hadoop 2.2
      • YARN support
  • Programming Languages & Tools
  • The New Azure Management Portal ( )
    • Reimagined modern design with live breadcrumbs to navigate easily
    • Extensible design
    • Integrated TFS Work Items, Check-in views, Code views, Application Insights.
    • Integrated Code Editor for complete DevOps lifecycle
    • Resource Groups (group of resources which are running an application)
    • New Azure Powershell cmdlets to publish resource templates
    • Export the entire script of the resource group in a ,json file. Similar to infrastructure scripts
    • Azure Resource Manager preview available
    • Visual Studio Online GA
    • Screenshots:



  • Part 2 of Keynote
    • Key developer concerns/questions
      • Support Existing Investments
        • Classic Desktop apps. Demo of WPF connected Office 365 Calendar API’s
        • Demo of old VB6 apps transformed to a Web App using WebMap
      • Mobile and Cloud First
        • Demo of geofencing component for foursquare
        • Demo of Heapsylon Sensoria capturning the foot sensor data and showing live feed on a windows phone
        • Launch of new “App Studio” to generate Windows 8 apps from Websites.
        • New Offline support using the HTML5 Index DB to cache the browsed data

Building services for Connected Devices

  • Key Aspects
    • End to end asynchronous data flow
    • Direct HTTP based connectivity and inbound messaging
    • Queue drien message processing
  • General Purpose IoT Architecture
    • Device
    • Protocol Gateway (Ingest)
    • Messaging and Eventing
      • Embedded Devices + Mobile Devices
      • Message Processing
      • Bulk Storage & Latent Analytics
      • More Stuff
  • Sample fast food outlet application
    • Updates published to central server every minute (~5Kb XML files)
  • Solution –> Azure web role as ingest and Message Queue + Azure workerrole
  • Ingest and Protocol Management
    • Device almost always has to initiate connections
      • Firewalls, NATs, dynamic IP, general network weirdness
    • Socket Management, Hold Sockets open, keep alives – latency on push notifications
    • Protocol – HTTP, AMWP, MQTT, TCP, custom (bluetooth etc)
    • Direct connectivity vs concentrator/local gateway
      • Smart Grid / Power meters
    • Message response
  • VS Code demo
    • ServicePointManager configurations for improving latency and concurrent connections with Storage
      • UseNagleAlgorithm
      • DefaultConnectionLimit
      • Expect100Continue
    • Queue “CreateIfNotExistsAsync” method should only be called once. Use factory method which initializes this
    • Unbound concurrency can bring down the system. E.g. Task.Run in a for loop
    • Use a Task.WaitAll outside the for loop. Inside the for loop just add Tasks to a task list.
    • In memory queue –> .GetConsumingEnumerable() API is a friend
    • Managing configuration settings between Worker Role and Web Role can cause issues. Instead of using app.config and web.config use the Role configuration to store the settings.

High-performance web platform

  • The Physics of FAST (Physical factors in performance )
    • Network Utilization and Limits  (bandwidth in mbps)
      • 2G  (0.1)
      • 3G  (1.8)
      • 4G HSPA+ (3.4)
      • WiFi (6.1)
      • 4G LTE  (7.5)
    • We can increase the bandwidth but latency might still be the issue (clap and response clap demo)
    • Speed and Responsiveness
      • CPU Utilization  / Elapsed Page Load
      • Real and Perceived performance
    • Memory Utilization
      • important when targeting low-cost devices with your sites and apps
    • Power Consumption
      • Let it rest !
      • CPU Utilization, GPU Utilization and VSync
      • Thermo mitigation (won’t charge since the device is too hot)
      • 5 Watt for Phone devices
    • Targets Devices Matter
      • Priortize those
  • Web Runtime Architecture
    • Networking cache
    • Parsers
      • creates internal data structures
    • DOM Tree (DOM API, JavaScript, CSS Cascade)
      • JS is sent to a sandbox
    • Formatting
    • Layout
    • Display Tree
    • Painting
      • DirectX services
    • Compositing
      • Handing over to GPU
    • Input / Hit Testing / DMANIP
  • Event Tracing for Windows (ETW)
  • Measuring Performance
    • F12 Developer Tools (UI Responsiveness uses the ETW events)
    • Windows Performance Toolkit
  • Performance in the Real World
    • Network Optimization
      • Navigation Timing (F12 Console –> window.performance.timing)
      • Average Payload
        • 93 resources, 16 hosts
        • 6 requests per host
        • 1.7 MB total size, 46% cacheable (on Mobiles)
        • Majority of the size is around “Images”
      • How to optimize Images
        • Right-size your images
        • Reduce # of downloads (use sprites and minified images)
        • Use a CDN
    • Speed and Responsiveness
      • Defer Script execution (scripts in bottom)
      • Reduce # of Frameworks (single frameworks or libraries)
      • Coaleasce style and layout changes
        • jittering of a control as we scroll
        • instead of updating the screen on every mouse event
        • just use css to paint it after the event is done
      • Use CSS Animation
    • Memory
      • Download size vs Memory Size
        • 557KB image –--->  Memory = width * height * 4
        • Process
          • Download Image
          • Decode
          • GPU Copy  (here the CPU and CPU is not doing anything)
          • GPU render 
        • Image Formats Matter (for both memory and CPU utilization)
          • JPEG is encoded to YCbCr and then decoded by CPU to JPEG for rendering (IE10)
          • GPU’s can render YCbCr so decoding is not needed for JPEG. (IE11 uses GPU for image rendering)
          • Use JPEG when possible instead of PNG (as they are much faster)

Building Services for Mobile Apps using Web API 2.1

  • Mysteries of building backend services for mobile apps
    • SOAP ? REST ? ODATA ?
    • which framework ?
    • hosting?
    • security ?
    • consuming ?
    • offline ? notifications ? etc..
  • SOAP is not mobile friendly (If you need SOAP then use WCF)
  • HTTP and mobile – made for each other
    • Ubiquitous
    • Interoperable
    • Scalable
    • Flexible
    • Mature
    • Simple
  • Many faces of HTTP frameworks in .NET
    • WCF Web HTTP (underlying it sends XML so its kind off SOAP)
    • WCF Data Services
    • MVC
    • Web API
      • A framework for creating HTTP services that can reach a broad range of clients
  • Why Web API
    • First class modern HTTP programming model
    • Easily map resources to URI’s and implement the uniform interface of HTTP
    • Rich support for formats and HTTP content negotiation
    • Request validation
    • Enable hypermedia with link generation
    • Separate out cross cutting concerns
      • authorization
      • caching
      • message header
    • Help page generation
    • Flexible hosting
      • can be hosted outside IIS within an EXE
    • Light-weight, testable, scales
    • 2.0 features
      • OWIN Integration
      • Request batching
    • 2.1 Features
      • Global error handler
      • attribute routing improvements
      • BSON formatter
      • Portable query building and parsing
  • Samples and Docs
  • Code at
  • REST and OData
    • REST is an Architectural style for distributed sytems
      • Defined by set of constraints (client-server, stateless, cacheable, layered, uniform interface)
      • Induces properties: performance, scalability, evolvability, reliability, transparency
      • Do a the Web does
    • OData
      • Is a protocol for interacting with REST-based data services
      • Solves common issues (querying, paging etc)
  • Web API 2.2 #NEW# features
  • Build client logic as “Portable Libraries” (share code between Desktop, Phone and Windows)
  • Security for Web API’s
    • User Authorization filters (ex [Authorize])
    • Implement your authorization logic
      • derive from authorizationfilterattribute or implement IAuthorizationFilter
  • Make Web API’s secure with OAuth 2.0
    • Authorize requests using OAuth 2.0 Bearer tokens
  • Get tokens from a trusted authz server
    • Host your own
      • MS OWIN for simple server implementation
      • Use thinktecture full server implementation
    • Use Existing one
      • Azure active directory
      • ADFS on windows server 2012
      • Azure mobile services

Windows App Studio (Beta)

Wednesday, April 02, 2014

Build 2014 – Day 1 (Notes)



  • Mobile First / Cloud First vision and strategy
  • Microsoft announced Windows Phone 8.1 and Windows 8.1 Update
  • New personal assistant “Cortana” – Microsoft’s answer to ‘Siri’ comes on Windows Phone 8.1
    • Remind me to ask about puppy when I talk to “blah” next. (reminds when the call arrives)
  • Many new features and optimizations on Windows Phone 8.1
    • Weekly calendar views
    • Swipe keyboard (World records of fastest phone keyboard)
  • New Universal App development approach
    • Share code between ALL Win 8 platforms (Phone, Desktops, Tablets, XBox One) easily with special Visual studio templates
  • WinJS is now Open Source
  • Visual Studio 2013 Update 2 RC available now
  • Windows 8.1 Update available now on MSDN
  • New Nokia 930 and other lumia devices announced


Single Page Application with and Angular.js

  • Sample to get started
    • include angular.min.js
    • create a controller js file and include it
    • add the ng-app attribute on the html tag
    • bind the controller to body
      <body ng-controller=”testController”>
    • Sample repeater control
      <div ng-repeat=”card in cards”>
      Card {{}}
    • testController is just a function
      var testController = function ($scope) {
         $ = [ { id:1 }, { id:2 } ];
  • Bundling and minification may not work as angular needs the name $scope
  • Use either nuget or cdn to get the angular js files
  • keep you app controller js files separate
  • Angular Items
    • Modules
      var mainApp = angular.module(‘mainApp’,[‘testController’]);
      var testController = angular.module(‘testController’,[]);
    • Controllers
    • Dependency Injection
    • $http
      Lightweight API for AJAX requests
    • $resources
    • Filters
    • routing

What’s new in WinJS

  • WinJS 2.1 for Phone
  • WinJS vision is to create a cross platform library for all devices
  • WinJS is now Open Source and accepts contributions from community
  • WinJS can now be used in cross platform Web sites and apps
  • New Pivot Control for Phone
  • Improvements and Support of WinJS controls on Phone
    • Appbar on Phone
    • ListView on Phone
    • Semantic Zoom on Phone  (e.g. grouped listview by alphabets. contacts list)
    • New Animations on Phone
    • User Themes on Phone
  • App bar code automatically adjusts between Phone and PC
  • WinJS Ratecontrol works with Angular and Knockout via 2 way binding

Large Scale JavaScript with TypeScript

  • Project “Monaco” – full code editor in browser for azurewebsites.
  • JS Code base roadmap for “Monaco”
    • Small – 50 KLOC  (Modules, classes, promises) – 10% typescript
    • Medium – 100 KLOC (AMD, lazy loaded, contributions) – 50% typescript
    • Large – 200 KLOC  (components, API, dependency injection) – 100% typescript
  • Concerns with Javascript
    • organizing a large and growing code base
    • refatoring javascript code is difficult
    • describing API’s
  • TypeScript to rescue
    • Optional and Structural typing
    • Classes, Modules
    • Interfaces (very powerful)
      • interfaces named object types of describing the shape of JavaScript objects
  • Typescript has Generics
  • Typescript Type Definitions project on github (‘DefinitelyTyped’ respository’)
  • Code Organization Pains
    • Modules are open
      • undisciplined name space contributions
    • You can have a cyclic dependencies
      • without noticing
    • Name spaces have no relationship to the code organization on disk
      • renaming files/name spaces is no fun
    • Growing code pains
      • requires ordering of js files
      • dependencies between files
      • eager loading (may load lots of stuff which is not needed)
  • TypeScript External modules for rescue
    • Separately loaded code referenced using external module names
      • implemented in separate source files
      • entities are private unless exported
    • Module Pattern
      • CommonJS
      • Asynchronous Module Definition (AMD) e.g. requireJS
  • Monaco
    • Server uses node.js
    • Client
    • Common module syntax enables code sharing between client and server
  • AMD migration was very good. Self contained modules.
  • AMD Applied
    • Lazy Loading
  • Componentization
    • reuse typescript code as ‘binary’ JS components with a declarations file
    • e.g. 
      • tsc – declarations –out typescriptservices.js typesript.ts
      • generates typescriptservices.d.ts

Monday, November 11, 2013

Cross Domain AJAX Calls and SharePoint Apps on Office 365

To give a bit of background, I have been exploring architectures and various components needed to create a multi-tenant SharePoint App. So far I have settled on using SharePoint only, as a frontend layer and using external services to drive the entire backend, including storage.

Given this high level design the first interesting item that I came across was while posting data to the backend Service from a SharePoint Hosted App. The classic client side Cross-domain security issue. Here are some of the solutions out there for this issue:

  • Using the JSONP hack
    • Due to its nature, this only works for ‘GET’ requests and I wanted to do a ‘POST.
  • Using the default SharePoint 2013 Web Proxy
    • I was able to make the ‘GET’ requests work with this approach but not the ‘POST’ requests.
  • Creating a Custom Proxy
    • Worked but involved quite a few hoops and a remote web page, which I did not want to create.
  • Enabling Cross Origin Resource Sharing (CORS) in WCF Services
    • Worked perfectly !!!

For simple requests we can enable CORS by setting some response headers like Access-Control-Allow-Origin, Access-Control-Allow-Methods and Access-Control-Allow-Headers with the value ‘*’. 
But all requests from Office 365 contain special headers and hence they are ‘Preflighted’, which means they would first send an http request with “OPTIONS” method to check the domain and then send the actual request. (More details on Preflighted requests here)
For Preflighted requests we need to write some extra code along with adding the response headers. Here is a good article with sample code that explains this and below is the snap shot of the code:




And here is the snap shot of the sample app working on Office 365:


Additional Resources on CORS and Multitenant Applications with Azure:

Monday, November 04, 2013

Visual Studio 2013 is cool !!!

Over the weekend I upgraded to Windows 8.1 Enterprise and also installed Visual Studio 2013. There are quite a few editor features that I absolutely love in VS 2013. Here is a quick preview of 3 editor features that I like the most:

Peek Definition (Alt+F12)


Code Lens (References + Unit Tests summary above all Methods)



For more details on the new features or to download Visual Studio 2013, visit :

Happy Coding !!!

Sunday, October 20, 2013

Avengers Emergency Helpline - Connecting Things to Internet

Avengers Emergency Helpline site. I will explain how I did this in my next post.

Enjoy playing with real things !!!….

Wednesday, October 16, 2013

Beginning of the Internet of Things…

Just unwrapped my Arduino Kit……


WP_20131016_006    WP_20131016_007



I have been watching this new wave of “Internet of Things” and I have decided to ride on it. This is my first experience with hardware projects. So far I have been breaking software builds but now I will break some real sh*t Smile. Although there is ton of material out there so I am at least safe in the beginning…

For someone who wants to ride this wave like me, here are few links that helped me a lot and hope it would help you too:

Enjoy playing with real things !!!….

SharePoint Apps vs. SharePoint Solutions

Last few weeks I have been working on a project which required some research on whether a developer should create an App or a traditional WSP Solution for the SharePoint 2013. The research will soon be transformed into a web app and I will post that when its ready but meanwhile here are few links which I found very helpful for this topic.

Friday, March 08, 2013

Authentication and Authorization with remote apps in Office 365 and SharePoint Online (Part 1)

This post is detailing about how you perform authentication and authorization from a remote app in SharePoint Online.
Especially, when the remote apps are running on a Non .Net technology platforms. Which means we can’t use the OOTB ‘TokenHelper’ class.
The entire flow needs to only use simple HttpRequests.
I am going to break this into 3 parts:
  1. Register a Remote App in SharePoint
  2. Get the 'AccessToken’ via the Azure ACS and SharePoint dance
  3. Call SharePoint REST Service with the AccessToken
Right now I am using the .Net HttpRequest class to perform this entire example and understand the entire flow. I am going to convert this into a JavaScript library soon so that it can be easily consumed by any external platforms. Ok, so lets get started.

Register a Remote App in SharePoint

There is some good guidance around registering an app for SharePoint but in our case we just want to register an app to perform the OAuth from a remote application so the only good option is to register it via ‘/_layouts/15/appregnew.aspx’.

There are 3 pieces of information that we need from the app registration:
  • client_id          = App Id
  • client_secret   = App Secret
  • redirect_uri     = Redirect URI

Get the 'AccessToken’ via the Azure ACS and SharePoint dance

There are 3 steps to this dance:
Step 1: Get the Request token
Getting the request token just requires a well formed Url with all the 3 pieces of information that we collected during the app registration.

(See this app permissions section for all the Scope and Rights available in SharePoint Online.)
This Url would redirect to the msonline login screen and after you enter the credentials if you prompt you with the trust screen:



Once you trust the app, it would redirect back to the ‘redirect_uri’ configured during the app registration along with the request token in the querystring


Step 2: Get the Realm

(This method is taken directly from TokenHelper class. The targetApplicationUri is the SharePoint Online url)
Step 3: And finally, Get the Access Token
Now that we have the requestToken and realm, we need to create a POST request to ACS to get back the access token


Call SharePoint REST Service with the AccessToken

The only thing to remember before calling the SharePoint REST API’s is to make sure that we requested the correct Scope and Rights while generating the access token. In the code above I request ‘AllProfile.Manage’ as my scope so I can call the User Profile REST API’s.
That’s it. Once we have the access token we can call all the SharePoint REST API’s that fetches the data. For creating, updating and deleting we need to get 1 more piece of data which is RequestDigest. I will cover this in my next post on uploading documents to SkyDrive Pro using REST API.

Reference Links: 

Tips and FAQs: OAuth and remote apps for SharePoint 2013
OAuth authentication and authorization flow for apps that ask for access permissions on the fly in SharePoint 2013 (advanced topic)
Get started with the SharePoint 2013 REST service
Using the SharePoint 2013 REST service

Download the Code from