Nice find.
In the first photo that you found you can see a retaining wall that is still there. It is the wall for Bertino Memorial Field at the Riverdale Country Day School. This Google street-view shows the same spot (note the wall on the left):
It looks like J.B. James beautiful childhood home was knocked down for the construction of the Henry Hudson Parkway–right around the time he wrote this memoir. Explains some of the nostalgia.