A lot happens under the hood when sending a Bitcoin transaction. There are a number of aspects of hot wallet administration that you should understand in order to avoid getting into a situation where you have a large backlog of unconfirmed transactions.
TRANSACTION FEE CALCULATION
The first and most important aspect of sending Bitcoin transactions is to set an appropriate transaction fee. Many Bitcoin users are not utilizing BitGo's dynamic fee support logic. Many transactions are currently setting fees that are exact multiples of 0.0001 BTC. These are obviously hard coded fees, which is the wrong approach to take. Instead of using a flat fee of 0.000x BTC, wallets should target a fee rate based upon the data size of the transaction and the current fee market conditions. You can read more about BitGo's dynamic fees in our post about surge pricing. The main point to remember is that miners prioritize the highest fee rate transactions in order to maximize their revenue, thus your fees must be competitive.
Using a static fee will eventually result in you accidentally creating a large data size transaction with a flat fee that results in a very low fee rate that miners will choose not to confirm.
Network throughput fluctuations:
A surge in network transactions causes contention for block space to increase, thus other Bitcoin users increase their fees in order to get higher mining priority, making your fees no longer competitive. Thus miners won't confirm your transactions.
Allow BitGo to set the transaction fee dynamically based upon the data size and current market conditions. When creating a transaction via BitGo's API, don't set a "fee" parameter and we will automatically compute the appropriate fee.
UNSPENT OUTPUT SELECTION
Spending unconfirmed outputs is also a common mistake that can be made. This is dangerous because unconfirmed outputs are money that you don't actually have yet, from the perspective of the Bitcoin network.
When you spend an unconfirmed output, you're ensuring that your transaction won't confirm until after the parent transaction(s) is also confirmed, which might never happen for any number of reasons such as low fees on the parent, double spending of the parent, or transaction malleability of the parent.
When creating a transaction you can pass the parameter "minConfirms" with a value of 1 or greater, and then you can pass the parameter "enforceMinConfirmsForChange" with a value of "true." This will cause your transaction to only select unspents that have been confirmed
UNSPENT OUTPUT MANAGEMENT
High volume hot wallets can also end up creating long chains of unconfirmed transactions. Long unconfirmed chains are dangerous for several reasons.
Transaction chains create dependencies. If anything goes wrong with one transaction in the chain, all of the descendants will get stuck and can't be confirmed until the parent(s) are confirmed. This can occur due to sudden spikes in the transaction fee market or it could happen if a malleated version of the transaction gets confirmed, which would invalidate all of the descendants.
Transaction chain limitations:
As of Bitcoin Core 0.12, nodes will stop relaying long chains of transactions. Unconfirmed chains more than ~25 transactions long (depending upon their total data size) won't get relayed. This will slow down how quickly your transactions get confirmed, since miners can only know about 25 of your sends at a time before the network will allow more descendent transactions to propagate through it.
We've found that BitGo users who create long chains of transactions are doing so because they don't have enough unspent outputs in their wallet. When you make the API call to BitGo to send a transaction, you can add an optional parameter called "splitChangeSize" and we recommend setting the value of this parameter to "-1" so that it targets an ideal size for unspents based on the sizes of your typical sends. This parameter will automatically create more change outputs if your wallet falls below an optimal number of outputs. The benefits of having more available confirmed unspent outputs to choose from will be faster confirmation times and robustness against edge case issues described above.
High volume unspents:
On the opposite end of the spectrum, some BitGo wallets have too many low value unspent outputs in their wallet, which makes it a challenge to send higher value transactions because we don't allow for the creation of transactions that are larger than 100KB, which is considered nonstandard and won't be propagated over the network. This often occurs in wallets that are primarily used for receiving transactions without many sends being made from the wallet.
We developed an unspent output consolidation tool that our API users can configure to clean up their wallet on a regular basis. If your wallet gets to the point that it has over 1,000 unspent outputs, we recommend using the tool to consolidate them. For receive-only wallets, it's recommended that you create a cron job that runs daily and checks the total number of unspent outputs, running the consolidation if the number has exceeded the desirable limit. Please also refer to the article found here for more information on consolidating unspents.
Depending on the dynamics of a particular hot wallet, it may be necessary to enable either change-splitting, periodic consolidation, or both. Smartly managing the unspent output set is by far the most important factor in ensuring a smoothly functioning hot wallet.