In most modern financial markets, traders express their preferences for assets by making orders. These orders are either executed if a counterparty is willing to match them or collected in a priority queue, called a limit order book. Such markets are said to adopt an order-driven trading mechanism. A key question in this domain is to model and analyze the strategic behavior of market participants, in response to different definitions of the trading mechanism (e.g., the priority queue changed from the continuous double auctions to the frequent call market). The objective is to design financial markets where pernicious behavior is minimized.The complex dynamics of market activities are typically studied via agent-based modeling (ABM) methods, enriched by Empirical Game-Theoretic Analysis (EGTA) to compute equilibria amongst market players and highlight the market behavior (also known as market microstructure) at equilibrium. This thesis contributes to this research area by evaluating the robustness of this approach and providing results to compare existing trading mechanisms and propose more advanced designs.In Chapter 4, we investigate the equilibrium strategy profiles, including their induced market performance, and their robustness to different simulation parameters. For two mainstream trading mechanisms, continuous double auctions (CDAs) and frequent call markets (FCMs), we find that EGTA is needed for solving the game as pure strategies are not a good approximation of the equilibrium. Moreover, EGTA gives generally sound and robust solutions regarding different market and model setups, with the notable exception of agents’ risk attitudes. We also consider heterogeneous EGTA, a more realistic generalization of EGTA whereby traders can modify their strategies during the simulation, and show that fixed strategies lead to sufficiently good analyses, especially taking the computation cost into consideration.After verifying the reliability of the ABM and EGTA methods, we follow this research methodology to study the impact of two widely adopted and potentially malicious trading strategies: spoofing and submission of iceberg orders. In Chapter 5, we study the effects of spoofing attacks on CDA and FCM markets. We let one spoofer (agent playing the spoofing strategy) play with other strategic agents and demonstrate that while spoofing may be profitable in both market models, it has less impact on FCMs than on CDAs. We also explore several FCM mechanism designs to help curb this type of market manipulation even further. In Chapter 6, we study the impact of iceberg orders on the price and order flow dynamics in financial markets. We find that the volume of submitted orders significantly affects the strategy choice of the other agents and the market performance. In general, when agents observe a large volume order, they tend to speculate instead of providing liquidity. In terms of market performance, both efficiency and liquidity will be harmed. We show that while playing the iceberg-order strategy can alleviate the problem caused by the high-volume orders, submitting a large enough order and attracting speculators is better than taking the risk of having fewer trades executed with iceberg orders.We conclude from Chapters 5 and 6 that FCMs have some exciting features when compared with CDAs and focus on the design of trading mechanisms in Chapter 7. We verify that CDAs constitute fertile soil for predatory behavior and toxic order flows and that FCMs address the latency arbitrage opportunities built in those markets. This chapter studies the extent to which adaptive rules to define the length of the clearing intervals — that might move in sync with the market fundamentals — affect the performance of frequent call markets. We show that matching orders in accordance with these rules can increase efficiency and selfish traders’ surplus in a variety of market conditions. In so doing, our work paves the way for a deeper understanding of the flexibility granted by adaptive call markets
