That’s a great question and one with an answer that could be an entire book in and of itself!
There are a few things to consider.
First, if you are in doubt, then keep things relatively simple. Readability is the most important thing! If it’s painful to read your book, very few people will bother to finish it. My friend Joel Friedlander (who has been a book designer for a few decades now) recommends typefaces such as Bembo, Bodoni, Caslon, Garamond, Janson, Granjon, and Sabon. I personally often go with Garamond as it looks good in a range of point sizes and I like the graceful, flowing style.
If you can find something that fits the material you are working with, that can often be a nice touch, but for straight non-fiction there’s usually less of a vibe that you are trying to convey as there can be with fiction as you don’t need to set up the world quite so much.
Whatever you do, use a font designed for print, and not one designed for display on screen which can be difficult to read in print and/or for long periods of time. Also, you should never need more than 2 fonts for the interior of your book without a good reason; one font for the text, and one for visual elements such as chapter headings. You may decide to use a third font if your book has regular cut-outs or captions but even then I’d tend to use the same font and just make it a different point-size.
When setting up your book, be careful not to specify specific fonts; use styles instead and then you can easily update your entire document at once and experiment with different combinations to decide what you like.