Searches for the Rare Decay of the Higgs Boson to Two Muons and for New Physics in Dilepton Final States with the ATLAS Experiment

This thesis presents two topics: a search for rare dimuon decay of the Standard Model Higgs boson and a search for new physics with non-resonant phenomena at the TeV mass scale with dilepton final states. Both studies investigate data recorded by the ATLAS experiment from sqrt(s)=13 TeV proton-proton collisions produced during the second of the Large Hadron Collider at CERN. This data corresponds to an integrated luminosity of 139 ifb. The first study is the search for the Standard Model Higgs boson decaying to two muons. This decay is used to study the Higgs boson Yukawa couplings to the second generation fermions as part of a field-wide effort to study the properties of the Higgs boson. Identifying events produced by this decay is complicated by the small Higgs boson branching fraction to muons and the large irreducible background from other Standard Model processes. Previous studies selected events using criteria that targets Higgs production through gluon-gluon fusion and vector-boson fusion mechanisms. To increase sensitivity to Higgs produced events, new selections are added targeting vector boson associated (VH) production with a leptonic decay from the vector boson. Multivariate analysis methods are used to identify events for different categories. The limits set on signal production in the new phase spaces explored in this analysis are the first of their kind. The strongest expected (observed) limit on leptonic VH production excludes signals down to 13.2 (22.6) times the Standard Model prediction. The combination of VH with the other major production modes results in a signal significance of 2.0 sigma over the background hypothesis. The second study is a search for non-resonant phenomena in the dielectron and dimuon final states. One possible source for this is quark and lepton compositeness at energy scales beyond direct access at the LHC. These would lead to contact interactions that produce non-resonant enhancement in dilepton production at the TeV mass scale. This search introduces several novel methods including a background model derived from data and a formalism to parameterization of the associated uncertainties. In this search, no significant deviation in data is observed with respect to the expected background. Upper limits on the visible cross-section times branching ratio are set in this search. These, along with benchmark CI signal efficiencies, can be interpreted as limits in terms of a variety of signal models. The lower limits on the energy scale of CI, Lambda, reach 35.8 TeV, indicating the quarks and leptons are still point-like particles at 10e-20 m. These are the strongest limits on qqll contact-interactions to date.