Week 7: Getting started with Filter-RKHS

-

 This week I revisited the RKHS method to confirm that we are on the right track and got started with implementation of the Filtered-RKHS method. Here is a day-wise summary of my progress throughout the week:

Monday & Tuesday:

Analysed the RFF code to reduce the time taken for calculations. Implemented an RFF calculation method according to the formula:

As described in this article where z(x)T z(y) is the equivalent calculation to k(x,y), the basic kernel function. The kernel techniques we employ use an iterative approach instead being performed through matrix operations. As a result of this, it involves calling the k(x,y) function repeatedly over the length of the dataset to produce results. I could not figure out how replacing the RFF calculations instead of RKHS kernel calculations would speed up the calculations. If anything, it is going to be slower because while the k(x,y) function is a straightforward calculation, the RFF method iterates over R random selected features as the formula shown above indicates. I must have gone wrong in my calculations somewhere or misunderstood some concepts. I will be revisiting this method again to figure it out after discussion with Roger Sir.

Wednesday:

To  reinforce that the RKHS method we have been using so far is reliable, I tested it with different models other than the Y= X2 model which we have been using for weeks now. Here are some of those results

Y = sinh(X)
Y = exp(X)
Y = tanh(X)
Y = tanh(X) + X2 

As seen, the RKHS method stays superior to the Probspace method across different curves and therefore we are on the right track.

Thursday & Friday:


Worked on implementing the hybrid Multi-Conditional probability method, Which is sort of a combination of how Probspace handles multiple conditionals and the RKHS method. The way Probspace calculates P(Z| Y , X = x) is that it basically filters the whole dataset and returns only the data points where X = x and then calculates P(Z|Y) on these data points.

 Since this may sometimes leave us with only a few data points left, there is an algorithm to search within close vicinity of the value x i.e, for some small σ, instead of Returning data points where X =x, it returns all data points where the value of X lies within range (x - σ ,  x + σ).
This provides us with more data to work with. However the tradeoff is that the σ value must not be so large that it makes the results inaccurate.
What I have implemented is that, after this filtration stage, I feed the filtered data points to the RKHS kernel and calculate P(Z | Y) using kernel methods instead of the ProbSPace methods. 
The model I used is:
X = logistic(0,1)
Y = logistic(1,1)
Z = math.sin(X)+math.tanh(Y)

Where X and Y are independent variables and Z depends on X and Y.
Here are some results:

The ProbSpace Z|Y,X=0  is the curve where data points have been filtered through the Probspace method and then the RKHS method has been applied to calculate P(Z|Y) on it.

As seen, the curve resembles a tanh() curve, which is what Z is when the influence of X has been removed.

Similarly, in this curve we are calculating P(Z|X,Y=0) i.e, we filter based on value of Y and then calculate P(Z|Y). The obtained curve resembles a sin() curve, which is nothing but Z with the influence of Y removed.


These results are excellent, however the obtained curves are only so accurate because of the high availability of data points i.e, since X=0 and Y=0 are around the mean of the dataset, we were able to filter a relatively large amount of datapoints in close vicinity to 0. However, as we condition on points farther away from the mean, X = 5 or Y = 3 for example, there are fewer data points available, even when the σ value is increased. This leads to poorer results. The graphs and  Average Error table is shown below:



Probability Calculation

Number of filtered data points

Average Error

P(Z|Y,X=0)

472

0.027500236786978147

P(Z|Y,X=3)

178

0.12472792815940457

P(Z|Y,X=5)

146

0.18335539958562355




P(Z|X,Y=0)

351

0.3696281399936332

P(Z|X,Y=1)

223

0.8753761183932961

P(Z|X,Y=3)

185

1.0922776215075785



Upcoming Week Plans:

Revisit the RFF method. Possibly work with Roger sir’s implementation of JointProb i.e, implementation of joint probability distributions through RKHS kernel.

Comments

Post a Comment

Popular posts from this blog

Week 12: Final Week

-
  These past 12 weeks have flown by so fast! It still feels like yesterday I started working on this project. There were highs and lows but at the end of the day we overcame our obstacles to create something to be proud of. It was a great learning experience and I had fun working with the team.  Monday: Implemented the prototype of the K decrement algorithm to reduce the number of filter variables in case of shortage of data points upon filtration. Here are some results: K decrement- condP() K = 100 rf = 0.8, minmin = 20 dims, datSize, tries, K =  3 1000 1 100 Average R2: JP, UP, PS =  0.67025996194      0.726423475099       0.7434199290171855 K=100 , rf = 0.8, minmin = 5 dims, datSize, tries, K =  3 1000 5 100 Average R2: JP, UP, PS =  0.70784384556      0.87296540935         0.764876980843549 K=100 , rf = 0.8, minmin = 4 dims, datSize, tries, K =  4 1000 1 100 Average R2: JP, UP, PS =...
Read more

Week 0: An Introduction

-
Image
What is Causality?      Causality (also referred to as causation, or cause and effect) is influence by which one event, process, state or object (a cause) contributes to the production of another event, process, state or object (an effect) where the cause is partly responsible for the effect, and the effect is partly dependent on the cause. That is the definition according to wikipedia. In simpler terms though, if a variable b is dependant on another variable a , then we can say a   causes   b  or that a   is a causal factor to b . Why Study Causality?      Traditional statistics only deals with pure values of variables and attributes. It does not tell us of the causal relationships between factors in a large dataset. ML algorithms in their current state can be biased, suffer from a relative lack of explainability, and are limited in their ability to generalize the patterns they find in a training data set for multiple applications. ...
Read more

Week 5: Random Features and Multiple Conditionals

-
Image
       This week I've digested multiple papers and they are all not necessarily related but I believe it was still required to gain understanding on how to move forward with the project. Here is a day-wise summary of my progress throughout the week: Monday: Studied the Fast Nonparametric Conditional Density Estimation paper . The paper details a method to calculate the ideal bandwidths for the Kernel methods(sigma in case of the Gaussian Kernel). They have made use of Dual-tree recursion to reduce the time complexity of the operation to O(n2). The previous best method, calculating integrated squared errors(ISE) was of O(n3) so this method saves a lot of time without compromising accuracy when dealing with large datasets. However, calculating bandwidths is not a priority right now, so after discussion with Roger sir, I have shifted my attention towards extending our kernel technique to condition on multiple variables. Tuesday:      Revisited the Nonpa...
Read more