Localized Regression

[hrothgar]

They gave me the main blog at TMW to play with for a week

 

Thought folks might find it amusing to see the fun part of my work...

 

http://blogs.mathworks.com/loren/2011/01/13/data-driven-fitting/

 

(I'm quite serious about the "fun" part)

[PassedOut]

Thanks, Richard. That does look like fun, and your posting was clear and well written.

[matmat]

Hm. thought you were going to link us to some study of IQ amongst elected officials in washington.

[Cyberyeti]

And I thought from the title you were talking about another phenomenon where creatures evolve something, and then unevolve it because of local conditions.

 

eg Santa Catalina island rattlesnake which has lost its rattle because there are no large mammals to warn off.

 

Interesting article.

[whereagles]

LOL

 

On a more on-topic note, the subject is interesting. I learned some curve-fitting when I was undergraduate, but, since it was physics, we always had some underlying theory for the main effect.

[helene_t]

Can we have Lo(w)ess with two or more independent variables? I know it used to be popular among people who correct images for background signal but not sure what software they used.

[hrothgar]

MATLAB has a localized regression implementation that works with two independent variables.

 

In theory, you should be able to extend the technique to cover N independent variables.

In practice, LOWESS requires a KNN search operation and KNN search slows down a lot with large numbers of dimensions.

 

Most people seem to switch over to boosted or bagged decision trees for large numbers of N.

[rogerclee]

I thought the article was interesting, thanks.

[Echognome]

Richard - As you know, econometrics is a huge field in economics. Does the localized regression technique fit into any of the known econometrics techniques that have been discussed in academic papers? I was just surprised to see the blog without any reference to academic literature on the subject. If it is truly a novel technique, I am surprised that no one has attempted to publish it.

[hrothgar]

Hi Matt

 

I wouldn't characterize anything in the blog post as novel. I was simply trying to provide a useful illustration of localized regression and the bootstrap.

 

It's pretty easy to find references about the application of this techniques within Economics for low dimensional spaces.

(Once you hit higher numbers of dimensions boosted and bagged regression trees are preferred).

 

There's also a lot of debate about the relative merits of localized regression versus smoothing splines in low dimensional spaces.

 

For example, the following piece preferred localized regression

http://www.cs.pitt.edu/~chang/265/sispaper/sisref/3.pdf

 

while

http://www.polisci.ohio-state.edu/faculty/lkeele/TOC.pdf

 

argued that smoothing splines are preferrable

[helene_t]

no its not a new idea, it has been around for at least a couple of decades.

 

Lo(w)ess is linear (or quadratic) regression where one model is fitted for each value of the independent variable, in that the points used for regression are weighted on the basis of their distance to the point of interest, using a dynamic-bandwidth kernel. It is some weird kernel the motivation of which I don't quite understand but I suppose the inventors had their reasons :)

[helene_t]

btw with many covariates my first try would be with a some kind of penalized regression. What about penalized lowess? I think I will implement that and see if it works.

 

Last week we had a guest speaker from New Zealand who had developped kernel smoothing specifically for estimation of local odds, i.e. you have two processes X and Y which you could fit by Lowess or such. Rather than fitting the two independently and then compute odds, he fitted the local log-odds directly. I found that interesting although of course a very specific thing.

[kenberg]

I don't know whether this is relevant but possibly it is of interest. Someone referred me to a letter by Brad Efron in the NYTimes. Apparently there was a research article on ESP that has provoked discussion about the use of statistics in the social sciences.

 

The link to Efron:

http://www.nytimes.com/2011/01/18/science/18letters-PROCRUSTEANC_LETTERS.html?_r=1&ref=science

 

His letter links to an article in the Times:

http://www.nytimes.com/2011/01/11/science/11esp.html

 

In the "small world" department, I wrote a term paper on esp when I was in high school (when Rhine at Duke was a big cheese), and I sort of knew Brad Efron then. He went to St. Paul Central, I went to St. Paul Monroe. My friend Joe Auslander pointed me to Efron's letter because he knew of this (weak) high school link. Richard's blog mentions resampling, and that relates back to Efron.

 

I haven't the time right now, but I am interested in Richard's blog and intend to get to it.