Do I have to go to court to challenge an IRS determination?

The simple answer is no, although the process is not simple. You can file administrative appeal which a complicated, technical and costly process and the opportunity to do it is governed by the law and IRS rules. If an appeal is unsuccessful, you may be able to file a case in the U.S. Tax Court. That's something that is best handled by an attorney. Here are excerpts from an informative article written by the Chief of the IRS Independent Office of Appeals that is posted on the IRS website. 
Appeals is an independent function within the IRS, completely separate from the compliance functions responsible for collecting and assessing taxes. Appeals provides an informal forum for taxpayers who disagree with an IRS determination. Our job is to resolve tax disputes without litigation, where possible, consider each case fairly and impartially and improve public confidence in the integrity and efficiency of the IRS.
 
In Appeals, we work with taxpayers informally to settle tax disputes without a formal court hearing and, in most cases, without the need to hire someone to represent them. We provide taxpayers a meaningful opportunity to be heard, regardless of their educational or economic status, background or English language proficiency. In fact, the Taxpayer First Act of 2019 gives taxpayers the right to come to Appeals to dispute most IRS Compliance determinations.
 
Appeals is unique within tax administration, because we have the authority to compromise the amount of tax owed to resolve a dispute. This means Appeals can offer taxpayers a fair settlement based on the probable outcome if their case were to go to court.  We refer to this as evaluating the "hazards of litigation." Here, I should note that not every dispute merits a compromise and some issues do not raise hazards of litigation. Here, I should note that not every dispute merits a compromise and some issues do not raise hazards of litigation. Appeals Officers and Settlement Officers are trained to carefully review the facts and law of each case to reach an impartial and fair settlement. You can improve the chance of a favorable settlement by providing as much evidence and supporting documentation as possible with your case. Though not required, taxpayers may have a representative handle their case. Taxpayers who are unable to pay a representative may be eligible for assistance for free through a Low Income Taxpayer Clinic.
 
Typically, appeal rights become available following a compliance action by IRS that could include an audit, penalty assessment or notice of a collection action. If you receive an IRS notice and your case is eligible for an appeal, the notice will explain your appeal rights. At that point, if you disagree with the IRS determination, you can request an appeal. The next step is to write down, either in a formal protest or simple statement, the issues with which you disagree and why. You can find out more about how to request an appeal and where to send it on IRS.gov. It’s important to remember that you should make your appeal request with the IRS compliance person who worked your case. That employee then will be able to send your appeal request, along with your case file, to Appeals. Taxpayers can also come to appeals after filing a petition in the United States Tax Court to dispute the IRS compliance action.
To evaluate your case, the Appeals Officer will fully consider your position and arguments along with the administrative case file from the IRS compliance person who worked your case. You may request to view the non-privileged part of the Compliance file prior to meeting with Appeals. One of the most helpful things you can do is provide all relevant facts, documents and other information supporting your position to the IRS compliance person working your case before it comes to Appeals. If you are unable to locate an important document that might help explain your position, please try to explain the document, why it isn’t available and what steps you took to try to obtain copies, etc. In this way, Appeals will have available all the information necessary to fully review your case. Appeals Officers and Settlement Officers try to resolve cases after holding a taxpayer conference or by correspondence. But, some complex cases may take more than one conference to resolve.
Also read I Have A Problem With The IRS. Do I Have To Deal With It Myself?