Studies of the Higgs boson properties and search for new physics beyond the standard model in multilepton final states with the ATLAS detector at LHC

The measurement of the Higgs boson properties is one of the main focuses of the high energy physics. The Yukawa coupling between Higgs and top quark - which is the heaviest Standard Model (SM) fermion - is one of the most interesting constants which can be probed from the cross section measurement of associated production of a Higgs boson with a top-quark pair ($t\bar{t}H$). This $t\bar{t}H$ cross section measurement with proton-proton collisions at a center-of-mass energy of ${\sqrt{s}=13}$ TeV is presented in this thesis. The measurement was performed in multilepton final states using a dataset corresponding to an integrated luminosity of 79.9 fb$^{-1}$ recorded by the ATLAS experiment at the Large Hadron Collider (LHC). The selected data are categorized in six final states by the number and flavor of charged-lepton candidates to search for the $t\bar{t}H$ signal. The best-fit value obtained is $\mu_{t\bar{t}H}$ is $\hat{\mu}=0.58^{+0.36}_{-0.33}$, while the corresponding cross section for $t\bar{t}H$ production is $\sigma_{t\bar{t}H}=294^{+182}_{-162}$ fb, in agreement with the Standard Model prediction, $\mu_{t\bar{t}H}^{SM}=507^{+35}_{-50}$ fb. The observed (expected) significance as excess above the background-only hypothesis is 1.8$\sigma$ (3.1$\sigma$). The analysis to measure the $t\bar{t}H$ cross section with the full Run 2 ATLAS dataset corresponding to 139.0 fb$^{-1}$ integrated luminosity is in progress. The expectation results are presented following the method used in the previous analysis with only the two same sign leptons and no tau channel. The expected significance derived from the statistical calculation is 2.2$\sigma$. The last part of the thesis presents a search for a heavy neutral resonance $X$ in an $X\rightarrow WW \rightarrow e\nu\mu\nu$ channel using data collected by ATLAS in 2015 and 2016. The 36.1 fb$^{-1}$ integrated luminosity dataset was taken at a center-of-mass energy of ${\sqrt{s}=13}$ TeV. Seven hypotheses are tested in the analysis: two scenarios, the narrow width approximation and the large width assumptions, that consider the decay widths of a model-independent heavy Higgs boson, and five benchmark models, the two-Higgs-doublet models, the Georgi-Machacek model, the heavy vector triplet model with a spin-1 heavy boson, the bulk Randall-Sundrum model with a spin-2 graviton at $k/\bar{M}_{Pl}=0.5$ or $1$, and the effective Lagrangian model with a VBF spin-2 graviton. No significant excess was found beyond the Standard Model background prediction in a mass range between 200 GeV and 5 TeV. Upper limits at 95% CL are set on the resonance production cross section times branching fraction $X\rightarrow WW$.