Week 8: Filter-RKHS and revisiting RFF

-

 This week I polished the Filter-RKHS technique from last week. We find this method to be faster and more accurate than the existing ProbSpace. I also revisited the RFF method and found a way of making it faster than the RKHS method through matrix calculations. 

Monday :

Discussed the results of the Filter-RKHS method I’d implemented last week with Roger sir. Found out why there appeared to be an offset in some of the graphs. Started working on comparing it with the Probspace method both in terms of time taken and accuracies in the form of average error and a new metric, r2 (r squared) which gives us a value from 0 to 1 based on how well the predicted values fit the ideal curve i.e, if  r2 = 1, it’s a perfect fit and vice versa.


Tuesday & Wednesday:

Here is a comprehensive report about the Filter RKHS method, in comparison to Probspace.  

Model Used:

X = logistic(0,1)

Y = logistic(1,1)

Z = math.sin(X)+math.tanh(Y)

Some General Observations:

  • When calculating P(Z|X,Y=y) we observed a falling-off in the Filtered-RKHS curve after about 6-σ  like shown:
            

Firstly, while the Probspace curve also falls off after this point, it is not as bad as the RKHS curve(which just flatlines). This skews the r2 calculations (like shown):


Filter-RKHS     

Average Error: 0.3696281399936332       r2: 0.18229232980738574

ProbSpace       

Average Error: 0.3343384323726664       r2: 0.4050433191928714


So for this specific calculation (P(Z|X,Y=y)) I will be limiting the X-axis range in (-6,6) going further in my testing. 


  • The Probspace technique is not far off from the Filtered-RKHS when the conditional value is around the mean i.e, when we are conditioning on X = 0 or Y = 0, the accuracies are very close (shown below): 
  
(The errors table for these graphs is after the next section. )

  • Where the Filtered RKHS method really shines in comparison is when we are conditioning X and Y on variables farther away from the mean, with lesser data points i.e, say X=3,5 or Y=3,5 for example. This is shown below:


  • Here is the Errors Comparison Table:

Probability Calculation

Number of filtered data points

R2 - FIltered RKHS method

R2  - Probspace method

P(Z|Y,X=0)

472

0.9988389090959336

0.9961056711545451

P(Z|Y,X=3)

178

0.9911884697222485

0.9281245034430583

P(Z|Y,X=5)

146

0.9813956008171265

0.02164428475009683





P(Z|X,Y=0)

351

0.9898016016934746

0.9706109520204094

P(Z|X,Y=3)

223

0.9697153921182456

0.9700721245115337

P(Z|X,Y=5)

185

0.8155517594997833

0.6721837012086203


Time taken:


Regarding the runtimes comparison, it depends on the value of the conditional i.e, for a X=0 , there is a higher number of datapoints available upon filtration and hence longer calculation times and vice versa. The FIlter-RKHS method takes between 0.07 to 0.2 seconds while the Probspace method takes between 8.5 to 10 seconds.


Thursday & Friday:

I Revisited the RFF method, after going through all of the theory again and understanding the code implementation, I was able to figure out a way to make it faster than the RKHS method, as it is supposed to be.

Turns out the right method of doing it was through matrix calculations of the kernel approximation matrix, which is different from the iterative approach we have been using for our 

RKHS kernel methods. There is an overhead time associated with the RFF method (about 4 seconds) which is required to obtain the approximation matrix. However the time taken to predict is magnitudes lesser than the RKHS method and the ProbSpace method. Here is a comparison of time-taken and errors, for different number of datapoints:


 

No. of TestPoints

Method

Time Taken

R2

Avg. Error



200

RFF

0.4371540546417

0.9781275444993528

1.976777971455252

RKHS

2.3733172416687

0.9902684201562044

1.163594525239619

ProbSpace

8.2712295055389       

0.5438537959871682

10.91937570677275




2000

RFF

3.4921276569366455

0.9792308212576896

1.906275141082829

RKHS

25.28713369369507

0.9910955762929684

1.120838399663066

ProbSPace

83.62622618675232

0.5645116007884847

10.55194800387150



Note: The Overhead time of RFF (4.01 and 4.39 seconds respectively) has not been included in the table above, time taken is measured strictly for prediction. 


With this, the RFF method is at a stage that we were initially hopeful about. It produces results comparable to the RKHS method at a faster rate.





Comments

Post a Comment

Popular posts from this blog

Week 10: Implementing UPROB

-
Image
Week 10 Report This week, I developed the UPROB module (you can check out the code here ). Since this was the main focus throughout the week and I spent all my time developing, bug-fixing , testing and documenting results, I feel that a day-wise summary would not be the most effective method to show results. Instead I have opted to explain the functionalities, working and usage of this module. This will also serve as documentation for future reference. I have listed the results and conclusion towards the end of this report. UPROB Init: Initializing a UPROB Object is very similar to that of rkhsMV except for the final argument k , which denotes the % of variables to be filtered before applying JPROB methods on the remaining ones.  R1 and R2 are two separate rkhsMV instances, one for each of the UPROB methods, condE() and condP(). The idea is to store R1 and R2 such that if we are repeatedly performing operations on the same dataset, we need not initialize a new kernel each time thus sa
Read more

Week 1: RKHS Kernel Implementation

-
Image
      My first long term goal is to improve the existing conditional probability calculation module by making use of RKHS mapping to get rid of the curse of dimensionality problem that has been haunting the traditional method. Here is a day-wise summary of my progress throughout the week: Monday: My Internship officially started on this day, I had a 1 on 1 meeting with Roger Sir regarding this week’s assignment which would be to implement an RKHS kernel for the purpose of predicting a probability distribution function, given just a set of data points ( 1000 in my case) belonging to the said distribution. This would bolster my understanding of RKHS working and will come in use when implementing more complex kernels in the future. Tuesday: Implemented the RKHS kernel and the evaluation function. Dataset for the arbitrary function X = logistic(-2,1) if choice([0,1]) else logistic(2,1) was created using the synthDataGen.py script and used as the basis for the evaluation function. Made
Read more

Week 12: Final Week

-
  These past 12 weeks have flown by so fast! It still feels like yesterday I started working on this project. There were highs and lows but at the end of the day we overcame our obstacles to create something to be proud of. It was a great learning experience and I had fun working with the team.  Monday: Implemented the prototype of the K decrement algorithm to reduce the number of filter variables in case of shortage of data points upon filtration. Here are some results: K decrement- condP() K = 100 rf = 0.8, minmin = 20 dims, datSize, tries, K =  3 1000 1 100 Average R2: JP, UP, PS =  0.67025996194      0.726423475099       0.7434199290171855 K=100 , rf = 0.8, minmin = 5 dims, datSize, tries, K =  3 1000 5 100 Average R2: JP, UP, PS =  0.70784384556      0.87296540935         0.764876980843549 K=100 , rf = 0.8, minmin = 4 dims, datSize, tries, K =  4 1000 1 100 Average R2: JP, UP, PS =  0.70162883565      0.80123074862         0.6089856810665888  K=100 in condP() function means that w
Read more