dtNavMeshQuery::closestPointOnPolyBoundary

Returns a point on the boundary closest to the source point if the source point is outside the polygon's xz-bounds.