Study of Higgs boson production in association with top quarks at the ATLAS experiment and sensitivity of new top quark observables

In this thesis, the search for the Standard Model (SM) Higgs boson produced in association with a top-quark pair, $t\bar{t}H$, was performed. A dedicated study of the dileptonic topology together with a Higgs boson decaying into a pair of $b$ quarks is presented, using $20.3\,{\rm{fb}}^{-1}$ of proton-proton collision data at a centre-of-mass energy of $\sqrt{s}=8\,\rm{TeV}$ collected with the ATLAS detector. Being mostly dominated by $t\bar{t}+\,\rm{jets}$ production, the events were categorised according to their jet and $b$-tagged jet multiplicities, which suppresses the SM backgrounds without loss of events, while maintaining the highest possible signal sensitivity. In order to search for the signal, a combined statistical fit to signal-rich and signal-depleted regions was performed. An improved background prediction was achieved by combining information from differently populated regions. Signal-depleted regions were used either as background control regions, allowing to match the simulation prediction to that of data, or as sources of background information to constrain the statistical fit. A neural network was used to discriminate between signal and background events in the signal-rich regions. No significant excess of events above the background expectation was found and an observed (expected) $95\%$ confidence level upper limit was set at $7.8$ ($4.5$) times the Standard Model cross-section. The ratio of the measured signal strength to the Standard Model expectation was found to be $\mu = 3.7 \pm 2.2$ assuming a Standard Model Higgs boson with a mass of $125\,\rm{GeV}$. The sensitivity of several observables to the CP nature of the top quark couplings in neutral and charged currents was also investigated in a phenomenological approach. The top quark Yukawa coupling to the Higgs boson as well as the top quark electroweak coupling to the $W$ boson were studied, considering different beyond the standard model (BSM) scenarios, while new observables were proposed to gauge their nature. The top quark-Higgs boson coupling is probed using dileptonic $t\bar{t}H/A$ events with either a scalar SM Higgs Boson, $H$, or a pure pseudoscalar CP-violating Higgs boson, $A$, considering also mixed states. Angular distributions of the decay products, as well as CP angular asymmetries, are explored in order to separate the scalar from the pseudo-scalar components of the Higgs boson and reduce the contribution from the dominant irreducible background $t\bar{t}+b\bar{b}\,\rm{jets}$. Significant differences between angular distributions and between asymmetries of the $t\bar{t}H$, $t\bar{t}A$ and $t\bar{t}+b\bar{b}\,\rm{jets}$ processes are observed. The nature of the top quark electroweak coupling to the $W$ boson is also probed, using top quark (antiquark) polarisations in single top quark (antiquark) production processes, where possible deviations from the SM are considered in terms of anomalous $Wtb$ couplings. Three orthogonal axes to investigate the top quark polarisation in the t-channel single top process were proposed and the expressions for the polarisation in these axes in terms of anomalous Wtb couplings were calculated. It was found that the polarisations in the two axes orthogonal to the spectator quark axis are very sensitive to an anomalous coupling involving a $\bar b_L \sigma^{\mu \nu} t_R$ dipole term. In particular, an asymmetry based on the polarisation normal to the production plane is more sensitive to the imaginary part of this coupling than previously studied observables. In addition, the energy and noise response of the ATLAS hadronic calorimeter (TileCal) was investigated, including inter-correlations among cells of the calorimeter and with special attention to the effect of simultaneous multiple collisions (pile-up) during the first LHC run. A full survey of the TileCal cells was performed as a function of the pseudorapidity, bunch position in a train and number of interactions per bunch crossing. Several periods of data collected by the ATLAS experiment without any selective trigger at a center-of-mass energy of $7\,\rm{TeV}$ was analysed and compared to Monte Carlo simulation reweighted to the pile-up conditions of the data. The closely monitored response of the TileCal cells in different conditions contributed to a proper characterisation of the TileCal performance.