Learning to Crawl

Web crawling is the problem of keeping a cache of webpages fresh, i.e., having the most recent copy available when a page is requested. This problem is usually coupled with the natural restriction that the bandwidth available to the web crawler is limited. The corresponding optimization problem was solved optimally by Azar et al. [2018] under the assumption that, for each webpage, both the elapsed time between two changes and the elapsed time between two requests follow a Poisson distribution with known parameters. In this paper, we study the same control problem but under the assumption that the change rates are unknown a priori, and thus we need to estimate them in an online fashion using only partial observations (i.e., single-bit signals indicating whether the page has changed since the last refresh). As a point of departure, we characterise the conditions under which one can solve the problem with such partial observability. Next, we propose a practical estimator and compute confidence intervals for it in terms of the elapsed time between the observations. Finally, we show that the explore-and-commit algorithm achieves an $\mathcal{O}(\sqrt{T})$ regret with a carefully chosen exploration horizon. Our simulation study shows that our online policy scales well and achieves close to optimal performance for a wide range of the parameters.Comment: Published at AAAI 202

Change Rate Estimation and Optimal Freshness in Web Page Crawling

For providing quick and accurate results, a search engine maintains a local snapshot of the entire web. And, to keep this local cache fresh, it employs a crawler for tracking changes across various web pages. However, finite bandwidth availability and server restrictions impose some constraints on the crawling frequency. Consequently, the ideal crawling rates are the ones that maximise the freshness of the local cache and also respect the above constraints. Azar et al. 2018 recently proposed a tractable algorithm to solve this optimisation problem. However, they assume the knowledge of the exact page change rates, which is unrealistic in practice. We address this issue here. Specifically, we provide two novel schemes for online estimation of page change rates. Both schemes only need partial information about the page change process, i.e., they only need to know if the page has changed or not since the last crawled instance. For both these schemes, we prove convergence and, also, derive their convergence rates. Finally, we provide some numerical experiments to compare the performance of our proposed estimators with the existing ones (e.g., MLE).Comment: This paper has been accepted to the 13th EAI International Conference on Performance Evaluation Methodologies and Tools, VALUETOOLS'20, May 18--20, 2020, Tsukuba, Japan. This is the author version of the pape

Robust determinants of bilateral trade

What are the policies and country-level conditions which best explain bilateral trade flows between countries? As databases expand, an increasing number of possible explanatory variables are proposed that influence bilateral trade without a clear indication of which variables are robustly important across contexts, time periods, and which are not sensitive to inclusion of other control variables. To shed light on this problem, we apply three model selection methods – Lasso reguarlized regression, Bayesian Model Averaging, and Extreme Bound Analysis -- to candidate variables in a gravity models of trade. Using a panel of 198 countries covering the years 1970 to 2000, we find model selection methods suggest many fewer variables are robust that those suggested by the null hypothesis rejection methodology from ordinary least squares

Sources of inflation and output fluctuations in Poland and Hungary: Implications for full membership in the European Union

This paper examines the sources of fluctuations in inflation and output in two leading transitioneconomy candidates for admission to the European Union (EU), Poland and Hungary. Using a rational expectations, dynamic open economy aggregate supply- aggregate demand model, we consider real oil price, supply, balance of payments, demand, and monetary disturbances incorporating important features of transition economies such as balance of payments disturbances and finite capital mobility. Evidence indicates that supply shocks explain a sizable portion of price level movements in Hungary while demand shocks are dominant in price level movements in Poland. Monetary shocks are an important source of output fluctuations in the short run in Hungary suggesting nominal inertia. In Poland, real demand shocks affect output in the short run (up to one year), while monetary shocks are negligible. Estimates of “core inflation” based on historical realizations of the shocks suggest that a major component of inflation has been demand driven, “core” inflation. Finally, policy implications of these findings regarding EU membership are evaluated. --Business Cycles,Inflation,Transition Economies,Time Series Models

Speculation and the Decision to Abandon a Fixed Exchange Rate Regime

This paper investigates the extent to which it is possible for speculative attacks to be predictable given information on economic fundamentals. A standard model of predictable attacks is extended to incorporate an optimizing monetary authority. It is shown that while incorporating a forward-looking monetary authority improves our understanding of many observed phenomena, it also implies that the branch of the literature that places emphasis on predictable movements in fundamentals cannot generate predictable speculative attacks. In addition, the model provides useful insights into the viability of temporary nominal anchor policies, and a theoretical foundation for an important empirical methodology.

Management of Exchange Rate Regimes in Emerging Asia

This paper revisits the issue of exchange rate regimes in emerging Asia. It is divided into two main parts. The first part compares de jure and de facto exchange rate regimes in Asia over the decade 1999–2009. The second part focuses on the sustained stockpiling of reserves in developing and emerging Asian economies since 2000 (interrupted only briefly by the global financial crisis). The paper concludes with some observations on the management of Asian currencies in light of the global financial crisis and concerns about global imbalances.exchange rate regimes; exchange rate management; emerging asia; global financial crisis; global imbalances