City staffers may have considered ditching the historic building designed by I.M. Pei earlier than previously reported.