@lubos: Hey after some more testing, I found out it was the "shading" actually affecting the physic.
When I used Smooth shading for softbody / cloth objects, the simulation went well.
When I used Flat shading, those objects just exploded or broke.
It is kind of weird that shading will affect the physics...this doesn't happen in blender's internal simulation. But at least I can use simulation now.
Edit: Also one more question about the simulation, is there a way to attach the simulating object to other object? I tried to use parent, armature deform and rigid body joint, but all failed.