Muhammad Abu Khusa, 82, was seized outside his home in Beit Lahiya, nearby residents said.
They said Abu Khusa tried to reach his family and told them he was in pain.
The former prisoner was first arrested in 1978 and served 36 years in prison.
In 2011, Hamas and other armed groups exchanged a captured Israeli soldier for 1,000 Palestinians in Israeli jails.
Since the kidnapping and killing of three teenagers in the occupied West Bank in June, Israel has begun rounding up those it freed in exchange for the soldier, Gilad Shalit.
Hamas has denounced the arrests as a violation of the agreement.