This one is tricky. The main culprit actually lies in your ListView attribute android:layout_height. You probably set it to wrap_content. While the result may sometimes be harmless e.g. setting values to TextView and ImageView, try adding some threads within the getView() method and you will be surprised why it gets called many times.
This is because if you set the ListView height to wrap_content, every View in the list has to rescaled, so the method keeps getting called. Replace the height with a fill_parent to solve this issue.
A value of 0dip works as well provided you set a layout_weight value.
Moral lesson? Never use wrap_content as height value of a ListView widget.