(also nonabelian homological algebra)
An $A_\infty$-category is a kind of category in which the associativity condition on the composition of morphisms is relaxed “up to higher coherent homotopy”.
The “A” is for Associative and the “${}_\infty$” indicates that associativity is relaxed up to higher homotopies without bound on the degree of the homotopies.
In the most widespread use of the word $A_\infty$-categories are linear categories in that they have hom-objects that are chain complexes. These are really models/presentations for stable (∞,1)-categories.
If the composition in the linear $A_\infty$-category does happen to be strictly associative it becomes the same as a dg-category. In fact, every linear $A_\infty$-category is $A_\infty$-equivalent to a dg-category. In this way, we have that $A_\infty$-categories related to dg-categories as models for stable (∞,1)-categories in roughly the same way as quasi-categories relate to simplicially enriched categories as models for (∞,1)-categories: the former is the general incarnation, while the latter is a semi-strictified version.
In what is strictly speaking a restrictive sense – which is however widely and conventionally understood in homological algebra as the standard notion of $A_\infty$-category (see references below) – the hom-spaces of an $A_\infty$-category are taken to be linear spaces, i.e. modules over some ring or field, and in fact chain complexes of such modules.
Therefore an $A_\infty$-category in this standard sense of homological algebra is a category which is in some way homotopically enriched over a category of chain complexes $Ch$. Since a category which is enriched in the ordinary sense of enriched category theory is a dg-category, there is a close relation between $A_\infty$-categories and dg-categories.
$A_\infty$-categories in this linear sense are a horizontal categorification of the notion of A-infinity-algebra. As such they are to A-infinity-algebras as Lie infinity-algebroids are to L-infinity-algebras. For this point of view see Konsevich-Soibelman 08.
A category $C$ such that
for all $X,Y$ in $Ob(C)$ the Hom-sets $Hom_C(X,Y)$ are finite dimensional chain complexes of $\mathbf{Z}$-graded modules
for all objects $X_1,...,X_n$ in $Ob(C)$ there is a family of linear composition maps (the higher compositions) $m_n : Hom_C(X_0,X_1) \otimes Hom_C(X_1,X_2) \otimes \cdots \otimes Hom_C(X_{n-1},X_n) \to Hom_C(X_0,X_n)$ of degree $n-2$ (homological grading convention is used) for $n\geq1$
$m_1$ is the differential on the chain complex $Hom_C(X,Y)$
$m_n$ satisfy the quadratic $A_\infty$-associativity equation for all $n\geq0$.
$m_1$ and $m_2$ will be chain maps but the compositions $m_i$ of higher order are not chain maps, nevertheless they are Massey products.
The framework of dg-categories and dg-functors is too narrow for many problems, and it is preferable to consider the wider class of $A_\infty$-categories and $A_\infty$-functors. Many features of $A_\infty$-categories and $A_\infty$-functors come from the fact that they form a symmetric closed multicategory, which is revealed in the language of comonads.
From a higher dimensional perspective $A_\infty$-categories are weak $\omega$-categories with all morphisms invertible. $A_\infty$-categories can also be viewed as noncommutative formal dg-manifolds with a closed marked subscheme of objects.
Every dg-category may be regarded as a special case when there are no higher maps (trivial homotopies) of an $A_\infty$-category.
Every $A_\infty$-category is $A_\infty$-equivalent to a dg-category.
This is a corollary of the $A_\infty$-categorical Yoneda lemma.
beware that this statement does not imply that the notion of $A_\infty$-categories is obsolete (see section 1.8 in Bespalov et al.): in practice it is often easier to work with a given naturally arising $A_\infty$-category than constructing its equivalent dg-category
for instance when dealing with a Fukaya? $A_\infty$-category;
or when dealing with various constructions on dg-categories, for instance certain quotients,that naturally yield directly $A_\infty$-categories instead of dg-categories.
The path space of a topological space $X$
The Fukaya category $Fuk(X)$ of a topological space $X$ – a Calabi-Yau A-∞ category
$A_\infty$-algebras are the $A_\infty$-categories with one object.
In the widest sense, $A_\infty$-category may be used as a term for a category in which the composition operation constitutes an algebra over an operad which resolves in some sense the associative operad $Ass$.
One should be aware, though, that this use of the term is not understood by default in the large body of literature concerned with the above linear notion.
A less general but non-linear definition is fairly straight forward in any category in which there is a notion of homotopy with the usual properties.
An $A_\infty$-category is a category over the $A_\infty$-operad: e.g. the free resolution in the context of dg-operads of the linear associative operad.
also the classical notion of bicategory can be interpreted as an $A_\infty$-category in Cat for a suitable Cat-operad.
For a short and precise introduction see
B. Keller, Introduction to $A_\infty$-algebras and modules (dvi, ps) and Addendum (ps), Homology, Homotopy and Applications 3 (2001), 1-35;
B. Keller, $A_\infty$ algebras, modules and functor categories, (pdf, ps).
and for a Fukaya category-oriented introduction see chapter 1 in
A very detailed treatment of $A_\infty$-categories is a recent book
Yu. Bespalov, Volodymyr Lyubashenko, O. Manzyuk, Pretriangulated $A_\infty$-categories, Proceedings of the Institute of Mathematics of NAS of Ukraine, vol. 76, Institute of Mathematics of NAS of Ukraine, Kyiv, 2008, 598 (ps.gz)
Oleksandr Manzyk, A-infinity-bimodules and Serre A-infinity-functors, dissertation pdf, djvu; Serre $A_\infty$ functors, talk at Categories in geometry and math. physics, Split 2007, slides, pdf, work with Volodymyr Lyubashenko
The relation of $A_\infty$-categories to differential graded algebras is emphasized in the introduction of
which mostly discusses just A-infinity-algebras, but points out a generalizations to $A_\infty$-categories, see the overview on p. 3
Essentially the authors say that an $A_\infty$-category should be a non(-graded-)commutative dg-manifold/L-infinity-algebroid.
More category theory and homotopy theory of $A_\infty$-categories is discussed in
Kenji Lefèvre-Hasegawa, Sur les A-infini catégories (arXiv:math/0310337)
See also
If one understands $A_\infty$-category as “operadically defined higher category”, then relevant references would include:
With operads modeled by dendroidal sets, n-categories for low $n$ viewed as objects with an $A-\infty$-composition operation are discussed in section 5 of
and
See also the references at model structure on algebras over an operad.