Serre functors were introduced by Aleksei Bondal and Mikhail Kapranov to study admissible subcategories? of triangulated categories. A Serre functor on a triangulated category $\mathcal{A}$ is an exact functor such that for any objects $A$ and $B$, $\Hom(A,B) \simeq \Hom(B, S(A))^*$. It does not always exist, but when it does it is unique up to graded natural isomorphism.
The Serre functor is a powerful tool for working with the derived category of coherent sheaves on a variety.
In the original paper, the following definition was given.
Let $\mathcal{A}$ be a $k$-linear triangulated category with finite-dimensional Hom‘s and $k$ algebraically closed. A Serre functor $S : \mathcal{A} \to \mathcal{A}$ is an additive equivalence that commutes with the translation functor, with bi-functorial isomorphisms $\phi_{A,B} : \Hom_\mathcal{A}(A,B) \stackrel{\sim}{\to} \Hom_{\mathcal{A}}(B,S(A))^*$ for any objects $A$ and $B$, such that the composite
coincides with the isomorphism induced by $S$.
In fact, the last commutativity condition can be deduced from just the bi-functoriality of $\phi_{A,B}$, and commutativity with the translation functor also follows from a proposition below. Hence, the following definition is seen in later papers.
Let $\mathcal{A}$ be a $k$-linear category with finite-dimensional Hom‘s and $k$ an arbitrary field. A Serre functor $S : \mathcal{A} \to \mathcal{A}$ is an additive equivalence with bi-functorial isomorphisms $\phi_{A,B} : \Hom_\mathcal{A}(A,B) \stackrel{\sim}{\to} \Hom_{\mathcal{A}}(B,S(A))^*$ for any objects $A$ and $B$.
Of course, formally the definition could be used in categories enriched over a symmetric monoidal category with a sufficiently nice involution.
In the derived category of finite-dimensional vector spaces over $k$, the identity functor is a Serre functor.
In the derived category of coherent sheaves on a smooth projective variety $X$, the functor $(\cdot \otimes \omega_X)[n]$ is a Serre functor, in view of Serre-Grothendieck duality?, where $\omega_X$ is the canonical sheaf and $n$ is the dimension of $X$.
Any autoequivalence $F : \mathcal{A} \to \mathcal{A}$ commutes with a Serre functor: there is a natural graded isomorphism of functors $F \circ S \stackrel{\sim}{\to} S \circ F$.
Any Serre functor in a triangulated category is exact? (i.e. distinguished triangles are mapped to distinguished triangles).
Any two Serre functors are connected by a canonical graded functorial isomorphism that commutes with the isomorphisms $\phi_{A,B}$ in the definition of the Serre functor.
The original paper and English translation:
А. И. Бондал, М. М. Капранов, Представимые функторы, функторы Серра и перестройки, Изв. АН СССР. Сер. матем., 53:6 (1989), 1183–1205 pdf
Alexei I. Bondal, Mikhail M. Kapranov, Representable functors, Serre functors, and mutations, Izv. Akad. Nauk SSSR Ser. Mat. 53 (1989), no. 6, 1183–1205, 1337.
The following paper gives the corrected definition and also demonstrates the utility of the Serre functor as a tool for working with the derived category of coherent sheaves on a variety (c.f. Bondal-Orlov reconstruction theorem):
See also: