Adventure Scientists - Timber tracking

 

Goal: Identify Black Walnut trees and extract core, leaf, and twig samples

Some notes:

1. Make sure you don't sign up to sample from private land. Use the GaiaGPS app's shading feature to determine if a piece of land is private or public.
2. Identifying Black Walnut trees is pretty hard; use the iNaturalist app to determine if there are any previous sightings of Black Walnut trees - you need to cross-verify if those sightings are unsampled.
3. Extracting the leaves and twigs is very hard; extracting the core is probably the easiest thing to do. Make sure they send you a borer tool.

Polar Codes

Just read a really nice article in the WIRED magazine about the inventor of the "Polar Codes", Erdal Arikan and Huawei.. they spend an equal amount of time praising Erdal Arikan and dissing Huawei... 

First they talk about how Erdal invented the "Polar Codes". His obsession with the Shannon limit in Information Theory. This is the technical upper limit on how much information you can pack in a channel, factoring in the noise and redundancy. It was a very hard problem and even his mentor in MIT came close to a solution and gives up. Erdal goes back to Turkey and helps set up the engineering college in a University - Bilkent, which he heads. This helps him to work on this problem over 20 years, while other people in US have to work on small problems for the sake of tenure.

Simultaneously, they talk about Huawei's rise through the support of the Chinese Government and by stealing intellectual property. Apparently, Huawei screws Nortel, and when Nortel has to file for Bankruptcy, Huawei takes over their research team. It is the Chinese born head of the Nortel team who identifies Erdal's polar codes and starts working on them.

Now, even if one company develops the technology, the standards have to be agreed by a lot of companies, the governing body is called 3GPP. Huawei has a lot of leverage there, because it holds most of patents and it has been able to push the Polar Codes based technology. US doesn't even have an equivalent company - Europe has Ericsson, Japan has a few. And all the Chinese companies are working together in pushing the Polar Codes based standard including ZTE and Lenovo. They've succeeded :)

Now to claim legitimacy, Huawei is honoring the inventor Erdal - They start the article by saying the ceremony and settings were corny and cold-war style :)

https://youtu.be/BE5HuqEg0oY

Deep Learning - Coursera

 

This course was introduction to Neural Networks, I'll try to summarize this in as simple a manner as possible.

Logistic regression can be viewed as a simple, single layer neural network. Similarly, a neural network can be viewed as multiple layers of logistic regression.

The difference being that, logistic regression can detect linear patterns only i.e. just do line fitting through the training dataset.

Neural networks can detect non-linear patterns. This is because the each layer of the neural network has a non-linear activation function.

Logistic Regression with Gradient Descent

The goal of Logistic Regression is to train a model or create a model which can make predictions, more specifically True or False predictions.

The input is anything that can be represented as a matrix, say X, then we need a matrix W and a vector b such that:

W * X + b  = a.   

When an activation function say Ω is applied to a, we get the True or False result.

Rounding Ω(a) to an integer gives the True or False result.

Gradient Descent Algorithm

 The training algorithm proceeds as follows: 

We take a large number of training examples and We start with a zero matrix and zero vector for W and b.

For each training example X and training result Y, 

1. we calculate  a = Ω (W * X + b) 
2. Next, we calculate the cost Y vs. a
3. Next, we adjust W and b based on the cost.

We do this until there is no difference in cost across different iterations i.e there is no gradient descent.

Neural Network 

The Neural Network goal is similar, but goal is to train multiple layers and we start with random value matrices

The neural network algorithm is similar to the gradient descent algorithm, but once again it works across multiple layers.

1. As an equivalent of step 1 in logistic regression, we have "Forward Propagation"
2. Once again we calculate cost by comparing with intermediate step's result with the training result.
3. As an equivalent of step 3 in Logistic regression, we have backward propogation, which adjusts the weights across all the layers.

Other Stuff

Hyper-parameters

Things like the number of layers, the learning rate etc. Tuning these for optimal efficiency is a course of its own

Vectorization

This is a computation optimization where we avoid explicit for-loops in the code and instead use Python and Numpy's inbuilt features such as broadcasting

Learning Tip

Do the course with a friend, makes it much more easier and fun..

Learning Scala online

1. Programming courses take a lot more time than any other course

2. Do all of the code covered in the videos, not just the assignments

3. This one took almost a year

4. Key concept learnt was "Pattern matching - applied to lists, objects etc."
https://github.com/gany-c/ScalaCoursera

Mining millions of reviews

Study Notes from http://users.eecs.northwestern.edu/~choudhar/Publications/MiningMillionsofReviewsATechniqueToRankProductsBasedOnImportanceofReviews.pdf

Abstract

1. Mine Reviews
2. Get Score – present a product ranking model that applies weights to product review factors to calculate a products ranking score. Rank reviews and products based on score.
3. Sort reviews and products based on score.

Introduction

1. Too many reviews 
2. Reviews may contain –ve feedback about the seller and not the product.  Those need to be filtered.
3. A Review's credibility can be based on -
1. Date or Age of the review
2. Number of Helpful votes/ number of votes
4. Ratings provided can have a personal bias.

Methodology

1. Summary: Sentiment analysis for each relevant sentence
2. While calculating the product ranking scores, reviews should not be equally weighted, they should be mined and given proportional weights.
3. 3 stages are proposed in evaluating a review's weight
1. filter out irrelevant sentences.
2. use helpfulness votes and age to derive the review's weight.
3. calculate the product's ranking score based on the review weights.

Filtering Irrelevant sentences

1. This is treated as a binary classification problem
2. Use Support Vector Machines (SVM) to train a hypothesis function h. - sentence gives h(sentence)
3. The sentence is translated to a vector X
4. The SVM uses linear regression – h(X) = Beta-Transpose * X + b (Just like you used gradient descent to implement linear regression, use SVM here)
5. 1000 sentences are collected manually and used as the training set
6. 10 fold cross validation is used.

Calculating the Review's weight

1. Helpfulness Vote - H
1. bare minimum, it is X out of Y people found it useful.
2. to beat the bias in ratings do the following
1. ignore reviews with less than 10 votes
2. use  simple X out of Y ratio for items with 10 – 200 reviews
3. if number of items is greater than 200, multiply by a gaining factor, > 1
2. Age of review and durability - T
1. Younger reviews have a greater weight.
1. younger reviews will naturally have less number of votes
2. newer versions of the product will match with newer reviews.
2. Calculating the age based metric = e^ Decay rate * ( time of review – time of product release) + initializing factor.
3. Sentence splitter and part of speech tagging:-
1. Split reviews into sentences, using MXTERMINATOR
2. Assign positive or negative sentiments to sentences.
1. Use part of speech tagger to do assess sentiment.
2. Sentences are saved with PART-OF-SPEECH tags.

Product Ranking Score Function:-

1.  Calculate the sentiment or the polarity of the review. Use polarity, H and T to calculate the product ranking score.

2.  Calculating review sentiment

1. Manually pick a set of common adjectives/ adverbs as a seed list.

2. Augment it with synonyms and antonyms

3. If a sentence has an adjective or adverb from a positive set then it is positive.

4. Negative sentences are handled similarly

5. If a sentence has more than 1 sentiment, then you Polarity = Sum of positives + sum of negatives.

3. Final Score of a product is – For all of its reviews, Sum of all ( Polarity * H * T) / Sum of all H * Sum of all T

Evaluation and Analysis

1. Compare review rank with sales rank (Amazon specific) - how well the product is sold within its category
2. Mean Average Precision or MAP
1. Spearman's coefficient for a set of products between both human ranking and the above algorithm.
3. Results
1. Filtering out irrelevant sentences improves performance
2. giving weights to reviews is an additional improvement
3. weights + age of reviews is even better

Effects of Individual Features

1. To get the features that contribute the most to the ranking, the correlation between the ranking by a feature and overall ranking can be calculated.

Future work, Consider these additional attributes

1. Reviewer Credibility
2. Prioritizing features
3. Look for Sarcasm :)
4. Filter out Spam.
5. Data from other sources.

Performance Testing

This is a rough outline of my post, will be improved..

Goal: Performance test a Java application - Rest API, load balanced on linux nodes.

1. 1. Generate requests i.e. the load for performance test through JMeter or Silk (simulating 1000s of client requests)
2. 2. Take one or 2 server nodes as sample and extrapolate.
3. 3. Monitoring tools can be  - Jconsole (comes with JDK) and vmstat - available in Linux.
4. 4. Metrics to be measured are - 
1. CPU usage
2. Number of threads
3. Memory saw tooth
5. Example of unnecessary high CPU usage in code 

while(true)

{

 if(some-externally-flag){

}

}

This will cause an extraordinary amount of churn.

Architect - enroute

I'm trying to transition from a senior level engineer to a Software Architect. I believe the key to that transition is designing applications that can scale to take enterprise level performance loads. That in turn boils down to the following questions:-

1. Most production systems are distributed, i.e. several nodes sit behind a domain address and share the work load as determined by a load balancer.
1. How to do Capacity Planning for such a system:-
1. number of nodes
2. amount of memory and processors on each node
3. How do you performance test such a system.
2. If such a load balanced cluster exists at a middleware level, What is the best way of running a batch job over that cluster?
3. How does such a system maintain state or sessions?
2. How to determine the best technology stack for a given application?
3. When would you use an asynchronous channel of communication? JMS or Kafka?
4. How would you determine if a regular RDBMS such as Oracle or a NoSQL system such as Cassandra needs to be used?
5. When would you use object caching systems like MemcacheD or CouchBase? 
6. How to trace requests that flow across several clusters e.g. an incoming HTTP request for a page view could flow across a front server cluster and then proceed to a middleware cluster before hitting the DB and then back..

I hope to find the answers to all these questions and come up with blog posts for each of these.. Architect en route.. :)

Machine Learning - Stanford@Coursera

So, I just got done with another course on Coursera; this time it is Andrew Ng's Machine Learning.  I have tried to summarize is what I've learnt over the past 10 weeks. This post may end up being a bit too technical and one dimensional; and that's because the course goes depth wise into Machine Learning, but it still just a baby step into that field. 

To start, What is Machine Learning? There are several definitions of Machine Learning available online and I'm not going repeat one of them here. Instead, let me tell you what Machine Learning is not..  Machine Learning is certainly not Big Data.. It precedes MapReduce and NoSQL by a couple of decades. Machine Learning can work without terabytes of data. The programming assignments given out in this course covered a wide range of application such as Handwriting recognition, predicting house prices, Image compression etc and all of them used nominal amounts of training data. Big data, in turn, is not something that is solely used for predictive analysis (Machine Learning) problems. It can be used for simple applications like searching log files for keywords that occur most often etc. But, Machine Learning is supposed to work well with large amounts of data and so there is a considerable overlap between the 2 fields. Technologies like Apache Mahout operate in this space.

From a coding perspective, I could say that Machine Learning is Matrix Manipulation :) or at least, it boils down to that often. The matrix could represent anything, say the features of a product or it could be a pixel map of a photograph or a vector representation of an audio clip. Matrix manipulations are computationally intensive operations i.e. take a lot of time and memory. Writing efficient algorithms for matrix operations is more of a mathematician's job and than an engineer's job. Fortunately, prepackaged solutions exist in tools such as Matlab or Octave. So, if you are an engineer working on a Machine Learning problem, most likely you wouldn't be coming up with an algorithmic solution; instead you would be doing the following:-

1. Building the data representation and choosing the most appropriate features (columns of the Matrix)

2. Reduce the number of dimensions i.e. columns (compression)

2. Choosing the best algorithm (inbuilt function) or pipeline of algorithms, most relevant to the problem.

3. Fine tuning the parameters.

This seemed a little counter intuitive to me at first.

And next, for those of you who are curious specifically about this course, I can broadly classify its syllabus into 3 categories:-

1. The concept is lucid and so is the underlying mathematics: - Linear Regression, Gradient Descent for Linear Regression, Feature scaling and Normalization, Regularization to prevent overfitting a training set, K means Clustering and Collaborative Filtering.

Collaborative Filtering is about building recommender systems, i.e. "People who bought this also bought", "You may like". If you are in e-commerce, this might interest you.. 

2. Concept is  relatively simple to implement with Octave's inbuilt functions, but whose underlying mathematics is beyond the scope of the course and beyond me as well :)  - Normal Equations for Linear Regression, Sigmoid function and Logistic Regression, Dimensionality reduction with  SVD, Anomaly Detection using Gaussian distribution.

3. And finally, the part that evaporated as soon as the video lecture stopped playing :) - Neural Networks, Support Vector Machines

After the course, what next? I have my eyes set on this competition in Kaggle http://www.kaggle.com/c/galaxy-zoo-the-galaxy-challenge It is about classifying galaxies. If time permits I would like to do at least one non trivial submission. Anyone, who is interested in teaming up, please do reach out :) The Prize money is $10,000 :)

Startup Engineering - Stanford@Coursera

So, this is my first successfully completed online course. The last one I took on Coursera was Astronomy from Duke University; flunked that one. I had better luck this time.

So here is what the past 12 weeks have taught me:

1. I got to know the latest technology stack for building a web application:-

• Node.js is the new "in thing" and Javascript as such, is getting into everything. JS is now running on the server side of web applications, Android may integrate with ChromeOS and so the system calls that are now in Java may be replaced by JS. JS is even getting into the database.
• Linux is the one unchanging thing, when everything else is so meteoric. The preferred option for development of applications and deployment of live web servers is still Linux. So, it is a good place to dig deep and be strong at the fundamental level.
• Git and GitHub are now the #1 option for version control. GitHub isn't just a platform for hosting source code, it has become a community by itself. A prospective employer may want to look at your public GitHub repository; here's mine https://github.com/gany-c

2. The course has been a primer into enterprise cloud services like Amazon's Web Services and Heroku. Now, I know what they offer and to a certain extent, how to use them.

3. If you are going to commit to spending several years of your life on a PhD or a Startup, you might want to pick the project with the largest market potential.  This lecture gives a good introduction to analyzing market potential - https://d396qusza40orc.cloudfront.net/startup/lecture_slides%2Flecture5-market-wireframing-design.pdf. Some of the freely available tools include Google's Keyword Planner and Facebook's advertiser tools.

4. Bitcoin - I didn't know that money could work this way.. everyone just knows how much everyone else has.. interesting stuff.. guess if we can do that for everything, then we wouldn't need Governments anymore.

5. Regulations - Founding a startup doesn't necessarily mean you solve technical challenges all the time. You may well end up spending your time fighting harassment from competitors and government agencies.

All in all, I'm really impressed that we live in a time and age where such an incredible piece of education from a University like Stanford is available for free and from the comfort of our homes. I'm glad that I made use of it and completed the course successfully. I plan to take the machine learning course taught by Andrew Ng of Stanford, in the coming months.

So, who wants to cofound a company with me? I do have a few ideas ;)

My masters work

Here are the 2 most important pieces of work that at I did at University of Kentucky almost 10 years back. Any feedback is welcome:

Frequent Item-sets Generation using Progressive Sampling

Requirements Tracing using LSI-based Information Retrieval

A small eclipse Tip..

Annoying special characters suddenly appear in your eclipse source code files?

: Go to Window > Preferences > General > Editors > Text Editors
Click on the link "whitespace characters" to fine tune what should be shown.

http://stackoverflow.com/questions/364168/show-some-invisible-whitespace-characters-in-eclipse

My first Chrome extension

Here is my first chrome extension.. it show the first 20 pictures in Flickr photostream..

theskymind.crx

Open the link in Chrome to install the extension..

Thread Sleep in Java

There are quite a few articles online that deal with the sleep method in Threads. They deal with the granularity of the time that the thread is supposed to sleep i.e. with the lower limit of the sleep time. One such good post is this one:

http://www.javamex.com/tutorials/threads/sleep_issues.shtml

But I noticed that there weren't many places that talk about a possible upper limit on the sleep time (if there is one).  So I tried it on the following the operating systems:


Red Hat Enterprise Linux Server release 5.6 (Tikanga)

Microsoft Windows XP professional, Version 2002 service pack 3 using  
Eclipse Java EE IDE for Web Developers.Version: Helios Release,Build id: 20100617-1415


My code was pretty simple, I put a thread to sleep for 30 minutes and had a logging statement above and below the sleep statement. The thread recovered fine.

To conclude, I couldn't notice an upper limit on the sleep time.

SMTP vs IMAP vs POP3

Now I know the difference :)

A lecture is worth a 1000 books

Here are some good video tutorials:

Database by the IITs:
http://www.youtube.com/user/nptelhrd#g/c/9426FE14B809CC41

Unix by Stanford:
http://openclassroom.stanford.edu/MainFolder/CoursePage.php?course=PracticalUnix

Hadoop by CloudEra:

http://sonnygill.net/nosql/hadoop-training-cloudera/

Choice of Technology Stack

I was looking for the ideal stack and here is what I found online

Should you build the whole web solution with Java or should you implement real-time algorithms in Java and web site in Ruby or Python?


Actually this is a question about keeping your technology homogeneous or heterogeneous. Our quick suggestion is: the more homogeneous your technology stack is, the better. In the end it is easier and cheaper to maintain and develop.

Simple rule of thumb  is: the more your team is experienced with a particular technology, the better the chances that this technology is a good fit for your project. For example if you have a Java team with 3-5 years of experience, it will definitely develop more rapidly and efficiently than a team of junior Ruby engineers with 1-2 years of experience

Eclipse and the JDK

I thought I knew the basic relationship between the JDK, the JRE and the JVM. But then the Eclipse IDE raised a question in my mind. I want to know that if I have eclipse installed on my system, do I still need to install JDK separately to be able to compile my Java programs? Here is what I found online.

Eclipse doesn't come with a jdk but a compiler and runtime environment built in. Technically therefore you won't need to install one, though if you want to run or compile from outside the IDE (highly advisable) or want to test against the reference implementation (even better, the built-in compiler isn't 100% compatible) you should.

My new Ford Fusion and Microsoft Sync

I own a Ford Fusion hybrid which has the Microsoft Sync entertainment management system, with speech recognition. There are 20+ commands and learning them while driving can be quite a challenge. Here is what I have learned so far..

1. There are 3 attached devices:
1. Phone
2. Bluetooth Audio (through the same phone in our case)
3. USB link to Ipod
2. Voice commands to these switch to these devices:
1. PHONE
2. BLUETOOTH AUDIO
3. USB
4. LINE IN or USER DEVICE is the super set for all 3.
3. Simple Voice commands are
1. PLAY
2. PAUSE
3. NEXT TRACK
4. PREVIOUS TRACK
5. SHUFFLE
6. PLAY ALL: use when

                                                              i.      Player gets stuck in one song or set of songs.

                                                            ii.      While starting up and on PLAY says “current selection cannot be played”.

                                                          iii.      Sadly works only with USB link. ”PLAY ALL” Won’t with Bluetooth link.

7. AUTOPLAY ON: Theoretically, should resume play from wherever the song stopped. If it doesn’t work, use PLAY ALL to start over
8. USB if system went looking for another device i.e. Bluetooth
9. Big round button on top right: PLAY/PAUSE, same as 3.a, 3.b
3. Phone
1. Answering a call – phone key on steering.
2. .Rejecting or Disconnecting a call – hold phone key down.

another interview

Given an M*N matrix calculate the number of paths between (0,0) and (m,n); no going up or left:

Solution: recursive function => double calll

Question 2: what is the order of complexity of your solution?
3: What is closure in javascript?

A Phone screen..

Had a phone screen with a techie firm from the bay area, Here are some of the questions:

1. Explain a 3 tier web application?
2. What is an n-tier web appliaction?
3. Explain the MVC pattern..
4. How do you implement a login functionality with an MVC?
5. How does the browser back button or refresh affect the login functionality?
6. How do you secure the password? (get or post?)
7. Describe a situation where get would be better.
8. Explain the principles of Ajax?..
9. Which browser object is responsible for getting and posting the request in Ajax?
10. Which html tag would you use to create a list of numbers on a html page?
11. What is the difference between an HTML div and an HTML span tag.
12. Implement a function that returns 1 when 2 is passed and vice versa.
13. Implement the above with divison or bit shift
14. Which one would be most efficient in terms of memory and performance for Q.12: If-else or Array or HashMap or 13?
15. What is SQL injection? How to protect the final query from becoming ill formed due to use of keywords in the form input?