It's hard to find a "kernel" to do this.

Perhaps the easiest way to get such effect is  combining several images with various level of shift and rotation of source image.

Dear Cc Zhou,

if I correctly understand your question, you would like to get images that are more blurred in one direction (the direction of camera pan) than blurred in another direction. Is this correct?

You could simply apply a kernel (a filter mask) that is more elongated in the direction of the camera pan. One of the simplest kernels could be (1 1 1)/3, which computes the average of three adjacent pixels in horizontal direction (some blur) and leaves the vertical direction unchanged. If you seek for arbitrary direction, you have to create a corresponding kernel that could be square like ( 2 1 1 0 0; 1 1 2 1 0; 0 0 1 1 1; 0 0 0 1 2; 0 0 0 0 1) / 16, which is rather arbitrary.

This, however, does not simulate the effect that appears if the camera is following a driving car. In this case, the background will heavily be blurred, but the car will not.

Regards,

Thanks for your all answers. yeah, I admit it is hard to mimic the real-life camera shake blurs. A lot of research used simple artificial blurry images by convolving a sharp image with one simple line motion blur(like Tilo Strutz suggested). And some record and playback camera shake by using robot platform with a fixed distance between scene and camera (see paper: Recording and playback of camera shake: benchmarking blind deconvolution with a real-word database). A CVPR 2016 paper, A Comparative Study for Single Image Blind Deblurring researched the gap between synthetically blurred images and real blurred images. 

It is difficult that using a handheld camera takes a blurry photo, and records the relevant groundtruth image at the same time. What I am doing now is using cnn to deblur blurred images. I am gonna start with simple artificial blurred data. If any of you have interest in it, we can discuss about it together.

Dear Cc Zhou, 

a very simple way to burr a digital image is applying a low-pass filter over this image. In the spatial domain, if you apply a mean or a median filter on the image (independent of the filtering orientation), by calculating these statistical measurements on a set of pixels with correlate spatial position and using unitary filtering weights, you will achieve this effect. 

You can also apply a FFT on the input image, convert its domain to the frequency one and apply a low-pass filtering algorithm, you will also have a blurred image in the end of the processing.

On the other hand, to threat blurred images, you can also think about high-pass filters, that evidence the high frequency components of the image (edges and transitions).

Hope it can bring you some insights.