The most significant arthropod-borne viral infections in modern history are caused by species of the Flavivirus (FV) genus due to their global transmission. While immunization is effective at decreasing the burden imposed by some FV, vaccines for other FV such as Dengue virus (DENV) and Zika virus (ZIKV) are still in development or not widely available. A better understanding of the protective immune response, specifically the antibody response, to DENV and ZIKV will advance the vaccine development endeavors for these viruses. Utilizing longitudinal samples from controlled human immunization/challenge studies and natural infections, we investigated the antibody response to these viruses in both the cellular and serological compartments. Primary DENV or ZIKV infection results in an antibody response specific to the infecting virus with little cross-reactivity to other related FV. This specificity profile is observed in the immune serum, and both the acute plasmablast and convalescent memory B cell sub-populations. Neutralizing antibodies were identified in these compartments of the antibody response, suggesting their functional association with homologous immunity to the infecting virus. In contrast to primary infection, the antibody response induced by immunization with the tetravalent live attenuated vaccine TV003 is cross-reactive in both the serological and cellular compartments. Interpretation of these findings reveals an intimate and complex relationship between B cell sub-populations and serum antibodies, and contributes to our knowledge of various differentiation routes amongst these B cells. Further analysis using monoclonal antibodies reveals structural vulnerabilities of FV. We identified domain III of the viral envelope glycoprotein to be one of the immunodominant and neutralizing targets for DENV- and ZIKV-specific antibodies. Further studies will continue to elucidate the connections between different B cell sub-populations engaged in FV infections, and identify specific subsets that either confer protection or pose a disease risk in the secondary antibody response invoked by multiple infections. Characterization of antibodies produced by these B cell subsets will further define the determinants of antibody-mediated protection. Application of these findings will benefit the endeavor to combat the burden of FV. Better understanding of the structural-functional relationship between viral epitopes and neutralizing antibodies will create avenues for antibody-based therapies as well as vaccine design. Leveraging virus-specific antibodies induced by primary exposures will improve the specificity of DENV or ZIKV diagnosis to differentiate infection by each of these viral species or serotypes. Accurate diagnoses will lead to proper treatments to improve disease outcomes. Furthermore, accurate diagnoses will aid in sero-surveillance efforts to ultimately inform vaccine strategies for endemic populations