dealing with dipdup
When a Tezos chain re-org occurs the DipDup indexer backing Hicdex does not gracefully handle the situation for an index the size of Hicdex.
When DipDup sees a chain re-org its mitigation is to delete all indexed data and start anew. This works fine for most dapps that use only a few megabytes worth of data and really focus on real time needs. However this does not work very well for extremely large data sets what cover hundreds of thousands of on-chain entities and their related transactions.
Due to this, even after hardware was upgraded H=N continued to experience system failures from re-indexing. Integro Labs LLC (TezTools) created a custom mitigation that uses a dedicated index that simply exists to create backups every 100 objkts and publishes those backups. Should any of the live index systems have an issue they reload from that known-good backup.
Due to the large increase in Tezos on-chain transactions as of October 2021 the Tezos blockchain is seeing re-org events every few hours. Thanks to this new backup reload system H=Ns users should only experience minor issues when the index is reloading.
Pictured: Index Server Reloading from a recent backup on 10/1/2021
