Studies of the Higgs boson properties and search for new physics beyond the standard model in the top sector with the ATLAS detector

Proton-proton collisions at a center of mass energy of 13 TeV recorded during 2015-2017 with the ATLAS detector at LHC are analyzed in this thesis. The first part of the thesis details electron efficiency measurements with a Tag-and-Probe method in Z->ee events. The measured electron reconstruction efficiency varies between 97\% (15<$E_T$<25 GeV) up to 99\% at higher $E_T$ with an uncertainty of around 1\% and per-mile level, respectively. The electron efficiency of a new algorithm (PLI and PLV) designed to reduce the electrons originating from heavy flavor decays is found to be 60-70\% at low $E_T$ ($E_T$<20 GeV) increasing up to 95\% at higher $E_T$. The second part of the thesis covers a search for flavor-changing neutral currents (FCNC) in top decays to an up-type quark (u or c) and a Higgs boson (t->Hu or t->Hc) using the 2015-2016 data. These processes are strongly suppressed in the Standard Model (SM), however larger enhancements are predicted in new physics models. Multivariate analyses (MVA) are used to improve signal-background separation in multilepton final states. The measured branching ratio for the t->Hu (t->Hc) process is found to be compatible with zero and a 95\% confidence level upper limit is set at 0.19\% (0.16\%) with an expected limit of 0.15\% (0.15\%), the best limit in a single channel to date. The Higgs boson production in association with a top quark pair (ttH) allows direct access to the top Yukawa coupling. The ttH production mode was observed by the ATLAS and CMS experiments, independently, by combining several decay channels. Further efforts to measure ttH in the two leptons same-sign channel are discussed in the last part of the thesis.