Hydroxychloroquine, Palbociclib, and Letrozole Before Surgery in Treating Patients With Estrogen Receptor Positive, HER2 Negative Breast Cancer

Prior treatment

• Prior exposure to CDK 4/6 inhibitor therapy
• Treatment with any of the following medications within 4 weeks before the baseline diagnostic biopsy is taken: a. Oral estrogens, including hormone replacement therapy (but prior depot estrogen use not allowed). b. Investigational agents (or 5 half-lives, whichever is longer)
• Required concomitant use of any drug that is a strong CYP3A inhibitor or inducer
• Neo-adjuvant cohorts (Phase II): Prior therapy for breast cancer (medical, surgical or radiation therapy)

Other illness

• History of retinal disease or active visual disturbances (normal baseline study-specified retinal exam required)
• Acute illness, including infections requiring medical therapy, known bleeding diathesis or need for anticoagulation
• Psychological, familial, sociological or geographical conditions that do not permit compliance with the study protocol
• Neo-adjuvant cohorts (Phase II): Clinical T4 disease
• Neo-adjuvant cohorts (Phase II): Inoperable or metastatic breast cancer based on standard evaluation

Other

• Life expectancy of less than 6 months
• Pregnancy, lactation or planning to be pregnant.

• Signed written informed consent
• Diagnosis of estrogen positive breast cancer, estrogen receptor-positive and HER2-negative by American Society of Clinical Oncology (ASCO)/College of American Pathologists (CAP) criteria
• Eastern Cooperative Oncology Group (ECOG) performance status 0-1
• Postmenopausal defined by: a. Age \>= 55 years and 1 year or more of amenorrhea b. Age \< 55 years and 1 year or more of amenorrhea with luteinizing hormone (LH) and/or follicle stimulating hormone (FSH) levels in the postmenopausal range c. Age \< 55 with prior hysterectomy but intact ovaries with LH and/or FSH levels in the postmenopausal range d. Chemotherapy or medically induced ovarian suppression with 1 year or more of amenorrhea and with LH and/or FSH levels in the postmenopausal range e. Status after bilateral oophorectomy (\>= 28 days prior to first study treatment)
• Absolute neutrophil count (ANC) \>= 1500 cells/ul
• Platelet count \>= 100,000/ul
• Serum creatinine concentration \< 1.5 x upper limit of normal (ULN)
• Bilirubin level \< 1.5 x ULN
• Aspartate aminotransferase (AST) or alanine aminotransferase (ALT) \< 3 x ULN
• Alkaline phosphatase =\< 2.5 ULN
• Metastatic cohorts (Phase I): Diagnosis of stage IV estrogen positive breast cancer, estrogen receptor-positive and HER2-negative by ASCO/CAP criteria
• Metastatic cohorts (Phase I): Must be a candidate for treatment with CDK4/6 inhibitor and hormonal therapy with an aromatase inhibitor as standard of care
• Metastatic cohorts (Phase I): No prior exposure to CDK 4/6 inhibitors
• Neoadjuvant cohorts (Phase II): Diagnosis of stage I-III estrogen positive breast cancer, estrogen receptor-positive and HER2-negative by ASCO/CAP criteria. If stage I, clinical tumor size must be \>= 1.5 cm
• Neoadjuvant cohorts (Phase II): Baseline tumor Ki67 \> 5%
• Neoadjuvant cohorts (Phase II): Surgical candidate and appropriate for pre-operative endocrine therapy