Symptoms
When the new change tracking stored procedure sp_cleanup_history_table is run in SQL Server 2017, Assertion failure may occur.
Status
Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section.
Resolution
This issue is fixed in the following cumulative update for SQL Server:
About cumulative updates for SQL Server
Each new cumulative update for SQL Server contains all the hotfixes and security fixes that were in the previous build. We recommend that you install the latest build for your version of SQL Server:
References
Learn about the terminology that Microsoft uses to describe software updates.