15 research outputs found
Burden of disease scenarios for 204 countries and territories, 2022â2050: a forecasting analysis for the Global Burden of Disease Study 2021
Background: Future trends in disease burden and drivers of health are of great interest to policy makers and the public at large. This information can be used for policy and long-term health investment, planning, and prioritisation. We have expanded and improved upon previous forecasts produced as part of the Global Burden of Diseases, Injuries, and Risk Factors Study (GBD) and provide a reference forecast (the most likely future), and alternative scenarios assessing disease burden trajectories if selected sets of risk factors were eliminated from current levels by 2050. Methods: Using forecasts of major drivers of health such as the Socio-demographic Index (SDI; a composite measure of lag-distributed income per capita, mean years of education, and total fertility under 25 years of age) and the full set of risk factor exposures captured by GBD, we provide cause-specific forecasts of mortality, years of life lost (YLLs), years lived with disability (YLDs), and disability-adjusted life-years (DALYs) by age and sex from 2022 to 2050 for 204 countries and territories, 21 GBD regions, seven super-regions, and the world. All analyses were done at the cause-specific level so that only risk factors deemed causal by the GBD comparative risk assessment influenced future trajectories of mortality for each disease. Cause-specific mortality was modelled using mixed-effects models with SDI and time as the main covariates, and the combined impact of causal risk factors as an offset in the model. At the all-cause mortality level, we captured unexplained variation by modelling residuals with an autoregressive integrated moving average model with drift attenuation. These all-cause forecasts constrained the cause-specific forecasts at successively deeper levels of the GBD cause hierarchy using cascading mortality models, thus ensuring a robust estimate of cause-specific mortality. For non-fatal measures (eg, low back pain), incidence and prevalence were forecasted from mixed-effects models with SDI as the main covariate, and YLDs were computed from the resulting prevalence forecasts and average disability weights from GBD. Alternative future scenarios were constructed by replacing appropriate reference trajectories for risk factors with hypothetical trajectories of gradual elimination of risk factor exposure from current levels to 2050. The scenarios were constructed from various sets of risk factors: environmental risks (Safer Environment scenario), risks associated with communicable, maternal, neonatal, and nutritional diseases (CMNNs; Improved Childhood Nutrition and Vaccination scenario), risks associated with major non-communicable diseases (NCDs; Improved Behavioural and Metabolic Risks scenario), and the combined effects of these three scenarios. Using the Shared Socioeconomic Pathways climate scenarios SSP2-4.5 as reference and SSP1-1.9 as an optimistic alternative in the Safer Environment scenario, we accounted for climate change impact on health by using the most recent Intergovernmental Panel on Climate Change temperature forecasts and published trajectories of ambient air pollution for the same two scenarios. Life expectancy and healthy life expectancy were computed using standard methods. The forecasting framework includes computing the age-sex-specific future population for each location and separately for each scenario. 95% uncertainty intervals (UIs) for each individual future estimate were derived from the 2·5th and 97·5th percentiles of distributions generated from propagating 500 draws through the multistage computational pipeline. Findings: In the reference scenario forecast, global and super-regional life expectancy increased from 2022 to 2050, but improvement was at a slower pace than in the three decades preceding the COVID-19 pandemic (beginning in 2020). Gains in future life expectancy were forecasted to be greatest in super-regions with comparatively low life expectancies (such as sub-Saharan Africa) compared with super-regions with higher life expectancies (such as the high-income super-region), leading to a trend towards convergence in life expectancy across locations between now and 2050. At the super-region level, forecasted healthy life expectancy patterns were similar to those of life expectancies. Forecasts for the reference scenario found that health will improve in the coming decades, with all-cause age-standardised DALY rates decreasing in every GBD super-region. The total DALY burden measured in counts, however, will increase in every super-region, largely a function of population ageing and growth. We also forecasted that both DALY counts and age-standardised DALY rates will continue to shift from CMNNs to NCDs, with the most pronounced shifts occurring in sub-Saharan Africa (60·1% [95% UI 56·8â63·1] of DALYs were from CMNNs in 2022 compared with 35·8% [31·0â45·0] in 2050) and south Asia (31·7% [29·2â34·1] to 15·5% [13·7â17·5]). This shift is reflected in the leading global causes of DALYs, with the top four causes in 2050 being ischaemic heart disease, stroke, diabetes, and chronic obstructive pulmonary disease, compared with 2022, with ischaemic heart disease, neonatal disorders, stroke, and lower respiratory infections at the top. The global proportion of DALYs due to YLDs likewise increased from 33·8% (27·4â40·3) to 41·1% (33·9â48·1) from 2022 to 2050, demonstrating an important shift in overall disease burden towards morbidity and away from premature death. The largest shift of this kind was forecasted for sub-Saharan Africa, from 20·1% (15·6â25·3) of DALYs due to YLDs in 2022 to 35·6% (26·5â43·0) in 2050. In the assessment of alternative future scenarios, the combined effects of the scenarios (Safer Environment, Improved Childhood Nutrition and Vaccination, and Improved Behavioural and Metabolic Risks scenarios) demonstrated an important decrease in the global burden of DALYs in 2050 of 15·4% (13·5â17·5) compared with the reference scenario, with decreases across super-regions ranging from 10·4% (9·7â11·3) in the high-income super-region to 23·9% (20·7â27·3) in north Africa and the Middle East. The Safer Environment scenario had its largest decrease in sub-Saharan Africa (5·2% [3·5â6·8]), the Improved Behavioural and Metabolic Risks scenario in north Africa and the Middle East (23·2% [20·2â26·5]), and the Improved Nutrition and Vaccination scenario in sub-Saharan Africa (2·0% [â0·6 to 3·6]). Interpretation: Globally, life expectancy and age-standardised disease burden were forecasted to improve between 2022 and 2050, with the majority of the burden continuing to shift from CMNNs to NCDs. That said, continued progress on reducing the CMNN disease burden will be dependent on maintaining investment in and policy emphasis on CMNN disease prevention and treatment. Mostly due to growth and ageing of populations, the number of deaths and DALYs due to all causes combined will generally increase. By constructing alternative future scenarios wherein certain risk exposures are eliminated by 2050, we have shown that opportunities exist to substantially improve health outcomes in the future through concerted efforts to prevent exposure to well established risk factors and to expand access to key health interventions
The global burden of cancer attributable to risk factors, 2010-19 : a systematic analysis for the Global Burden of Disease Study 2019
Background Understanding the magnitude of cancer burden attributable to potentially modifiable risk factors is crucial for development of effective prevention and mitigation strategies. We analysed results from the Global Burden of Diseases, Injuries, and Risk Factors Study (GBD) 2019 to inform cancer control planning efforts globally. Methods The GBD 2019 comparative risk assessment framework was used to estimate cancer burden attributable to behavioural, environmental and occupational, and metabolic risk factors. A total of 82 risk-outcome pairs were included on the basis of the World Cancer Research Fund criteria. Estimated cancer deaths and disability-adjusted life-years (DALYs) in 2019 and change in these measures between 2010 and 2019 are presented. Findings Globally, in 2019, the risk factors included in this analysis accounted for 4.45 million (95% uncertainty interval 4.01-4.94) deaths and 105 million (95.0-116) DALYs for both sexes combined, representing 44.4% (41.3-48.4) of all cancer deaths and 42.0% (39.1-45.6) of all DALYs. There were 2.88 million (2.60-3.18) risk-attributable cancer deaths in males (50.6% [47.8-54.1] of all male cancer deaths) and 1.58 million (1.36-1.84) risk-attributable cancer deaths in females (36.3% [32.5-41.3] of all female cancer deaths). The leading risk factors at the most detailed level globally for risk-attributable cancer deaths and DALYs in 2019 for both sexes combined were smoking, followed by alcohol use and high BMI. Risk-attributable cancer burden varied by world region and Socio-demographic Index (SDI), with smoking, unsafe sex, and alcohol use being the three leading risk factors for risk-attributable cancer DALYs in low SDI locations in 2019, whereas DALYs in high SDI locations mirrored the top three global risk factor rankings. From 2010 to 2019, global risk-attributable cancer deaths increased by 20.4% (12.6-28.4) and DALYs by 16.8% (8.8-25.0), with the greatest percentage increase in metabolic risks (34.7% [27.9-42.8] and 33.3% [25.8-42.0]). Interpretation The leading risk factors contributing to global cancer burden in 2019 were behavioural, whereas metabolic risk factors saw the largest increases between 2010 and 2019. Reducing exposure to these modifiable risk factors would decrease cancer mortality and DALY rates worldwide, and policies should be tailored appropriately to local cancer risk factor burden. Copyright (C) 2022 The Author(s). Published by Elsevier Ltd. This is an Open Access article under the CC BY 4.0 license.Peer reviewe
Global age-sex-specific mortality, life expectancy, and population estimates in 204 countries and territories and 811 subnational locations, 1950â2021, and the impact of the COVID-19 pandemic: a comprehensive demographic analysis for the Global Burden of Disease Study 2021
Background: Estimates of demographic metrics are crucial to assess levels and trends of population health outcomes. The profound impact of the COVID-19 pandemic on populations worldwide has underscored the need for timely estimates to understand this unprecedented event within the context of long-term population health trends. The Global Burden of Diseases, Injuries, and Risk Factors Study (GBD) 2021 provides new demographic estimates for 204 countries and territories and 811 additional subnational locations from 1950 to 2021, with a particular emphasis on changes in mortality and life expectancy that occurred during the 2020â21 COVID-19 pandemic period. Methods: 22 223 data sources from vital registration, sample registration, surveys, censuses, and other sources were used to estimate mortality, with a subset of these sources used exclusively to estimate excess mortality due to the COVID-19 pandemic. 2026 data sources were used for population estimation. Additional sources were used to estimate migration; the effects of the HIV epidemic; and demographic discontinuities due to conflicts, famines, natural disasters, and pandemics, which are used as inputs for estimating mortality and population. Spatiotemporal Gaussian process regression (ST-GPR) was used to generate under-5 mortality rates, which synthesised 30 763 location-years of vital registration and sample registration data, 1365 surveys and censuses, and 80 other sources. ST-GPR was also used to estimate adult mortality (between ages 15 and 59 years) based on information from 31 642 location-years of vital registration and sample registration data, 355 surveys and censuses, and 24 other sources. Estimates of child and adult mortality rates were then used to generate life tables with a relational model life table system. For countries with large HIV epidemics, life tables were adjusted using independent estimates of HIV-specific mortality generated via an epidemiological analysis of HIV prevalence surveys, antenatal clinic serosurveillance, and other data sources. Excess mortality due to the COVID-19 pandemic in 2020 and 2021 was determined by subtracting observed all-cause mortality (adjusted for late registration and mortality anomalies) from the mortality expected in the absence of the pandemic. Expected mortality was calculated based on historical trends using an ensemble of models. In location-years where all-cause mortality data were unavailable, we estimated excess mortality rates using a regression model with covariates pertaining to the pandemic. Population size was computed using a Bayesian hierarchical cohort component model. Life expectancy was calculated using age-specific mortality rates and standard demographic methods. Uncertainty intervals (UIs) were calculated for every metric using the 25th and 975th ordered values from a 1000-draw posterior distribution. Findings: Global all-cause mortality followed two distinct patterns over the study period: age-standardised mortality rates declined between 1950 and 2019 (a 62·8% [95% UI 60·5â65·1] decline), and increased during the COVID-19 pandemic period (2020â21; 5·1% [0·9â9·6] increase). In contrast with the overall reverse in mortality trends during the pandemic period, child mortality continued to decline, with 4·66 million (3·98â5·50) global deaths in children younger than 5 years in 2021 compared with 5·21 million (4·50â6·01) in 2019. An estimated 131 million (126â137) people died globally from all causes in 2020 and 2021 combined, of which 15·9 million (14·7â17·2) were due to the COVID-19 pandemic (measured by excess mortality, which includes deaths directly due to SARS-CoV-2 infection and those indirectly due to other social, economic, or behavioural changes associated with the pandemic). Excess mortality rates exceeded 150 deaths per 100 000 population during at least one year of the pandemic in 80 countries and territories, whereas 20 nations had a negative excess mortality rate in 2020 or 2021, indicating that all-cause mortality in these countries was lower during the pandemic than expected based on historical trends. Between 1950 and 2021, global life expectancy at birth increased by 22·7 years (20·8â24·8), from 49·0 years (46·7â51·3) to 71·7 years (70·9â72·5). Global life expectancy at birth declined by 1·6 years (1·0â2·2) between 2019 and 2021, reversing historical trends. An increase in life expectancy was only observed in 32 (15·7%) of 204 countries and territories between 2019 and 2021. The global population reached 7·89 billion (7·67â8·13) people in 2021, by which time 56 of 204 countries and territories had peaked and subsequently populations have declined. The largest proportion of population growth between 2020 and 2021 was in sub-Saharan Africa (39·5% [28·4â52·7]) and south Asia (26·3% [9·0â44·7]). From 2000 to 2021, the ratio of the population aged 65 years and older to the population aged younger than 15 years increased in 188 (92·2%) of 204 nations. Interpretation: Global adult mortality rates markedly increased during the COVID-19 pandemic in 2020 and 2021, reversing past decreasing trends, while child mortality rates continued to decline, albeit more slowly than in earlier years. Although COVID-19 had a substantial impact on many demographic indicators during the first 2 years of the pandemic, overall global health progress over the 72 years evaluated has been profound, with considerable improvements in mortality and life expectancy. Additionally, we observed a deceleration of global population growth since 2017, despite steady or increasing growth in lower-income countries, combined with a continued global shift of population age structures towards older ages. These demographic changes will likely present future challenges to health systems, economies, and societies. The comprehensive demographic estimates reported here will enable researchers, policy makers, health practitioners, and other key stakeholders to better understand and address the profound changes that have occurred in the global health landscape following the first 2 years of the COVID-19 pandemic, and longer-term trends beyond the pandemic
Global burden and strength of evidence for 88 risk factors in 204 countries and 811 subnational locations, 1990â2021: a systematic analysis for the Global Burden of Disease Study 2021
Background: Understanding the health consequences associated with exposure to risk factors is necessary to inform public health policy and practice. To systematically quantify the contributions of risk factor exposures to specific health outcomes, the Global Burden of Diseases, Injuries, and Risk Factors Study (GBD) 2021 aims to provide comprehensive estimates of exposure levels, relative health risks, and attributable burden of disease for 88 risk factors in 204 countries and territories and 811 subnational locations, from 1990 to 2021. Methods: The GBD 2021 risk factor analysis used data from 54 561 total distinct sources to produce epidemiological estimates for 88 risk factors and their associated health outcomes for a total of 631 riskâoutcome pairs. Pairs were included on the basis of data-driven determination of a riskâoutcome association. Age-sex-location-year-specific estimates were generated at global, regional, and national levels. Our approach followed the comparative risk assessment framework predicated on a causal web of hierarchically organised, potentially combinative, modifiable risks. Relative risks (RRs) of a given outcome occurring as a function of risk factor exposure were estimated separately for each riskâoutcome pair, and summary exposure values (SEVs), representing risk-weighted exposure prevalence, and theoretical minimum risk exposure levels (TMRELs) were estimated for each risk factor. These estimates were used to calculate the population attributable fraction (PAF; ie, the proportional change in health risk that would occur if exposure to a risk factor were reduced to the TMREL). The product of PAFs and disease burden associated with a given outcome, measured in disability-adjusted life-years (DALYs), yielded measures of attributable burden (ie, the proportion of total disease burden attributable to a particular risk factor or combination of risk factors). Adjustments for mediation were applied to account for relationships involving risk factors that act indirectly on outcomes via intermediate risks. Attributable burden estimates were stratified by Socio-demographic Index (SDI) quintile and presented as counts, age-standardised rates, and rankings. To complement estimates of RR and attributable burden, newly developed burden of proof risk function (BPRF) methods were applied to yield supplementary, conservative interpretations of riskâoutcome associations based on the consistency of underlying evidence, accounting for unexplained heterogeneity between input data from different studies. Estimates reported represent the mean value across 500 draws from the estimate's distribution, with 95% uncertainty intervals (UIs) calculated as the 2·5th and 97·5th percentile values across the draws. Findings: Among the specific risk factors analysed for this study, particulate matter air pollution was the leading contributor to the global disease burden in 2021, contributing 8·0% (95% UI 6·7â9·4) of total DALYs, followed by high systolic blood pressure (SBP; 7·8% [6·4â9·2]), smoking (5·7% [4·7â6·8]), low birthweight and short gestation (5·6% [4·8â6·3]), and high fasting plasma glucose (FPG; 5·4% [4·8â6·0]). For younger demographics (ie, those aged 0â4 years and 5â14 years), risks such as low birthweight and short gestation and unsafe water, sanitation, and handwashing (WaSH) were among the leading risk factors, while for older age groups, metabolic risks such as high SBP, high body-mass index (BMI), high FPG, and high LDL cholesterol had a greater impact. From 2000 to 2021, there was an observable shift in global health challenges, marked by a decline in the number of all-age DALYs broadly attributable to behavioural risks (decrease of 20·7% [13·9â27·7]) and environmental and occupational risks (decrease of 22·0% [15·5â28·8]), coupled with a 49·4% (42·3â56·9) increase in DALYs attributable to metabolic risks, all reflecting ageing populations and changing lifestyles on a global scale. Age-standardised global DALY rates attributable to high BMI and high FPG rose considerably (15·7% [9·9â21·7] for high BMI and 7·9% [3·3â12·9] for high FPG) over this period, with exposure to these risks increasing annually at rates of 1·8% (1·6â1·9) for high BMI and 1·3% (1·1â1·5) for high FPG. By contrast, the global risk-attributable burden and exposure to many other risk factors declined, notably for risks such as child growth failure and unsafe water source, with age-standardised attributable DALYs decreasing by 71·5% (64·4â78·8) for child growth failure and 66·3% (60·2â72·0) for unsafe water source. We separated risk factors into three groups according to trajectory over time: those with a decreasing attributable burden, due largely to declining risk exposure (eg, diet high in trans-fat and household air pollution) but also to proportionally smaller child and youth populations (eg, child and maternal malnutrition); those for which the burden increased moderately in spite of declining risk exposure, due largely to population ageing (eg, smoking); and those for which the burden increased considerably due to both increasing risk exposure and population ageing (eg, ambient particulate matter air pollution, high BMI, high FPG, and high SBP). Interpretation: Substantial progress has been made in reducing the global disease burden attributable to a range of risk factors, particularly those related to maternal and child health, WaSH, and household air pollution. Maintaining efforts to minimise the impact of these risk factors, especially in low SDI locations, is necessary to sustain progress. Successes in moderating the smoking-related burden by reducing risk exposure highlight the need to advance policies that reduce exposure to other leading risk factors such as ambient particulate matter air pollution and high SBP. Troubling increases in high FPG, high BMI, and other risk factors related to obesity and metabolic syndrome indicate an urgent need to identify and implement interventions
Parallélisme et équilibrage de charges dans le traitement de la jointure sur des architectures distribuées.
The appeal of parallel processing becomes very strong in applications which require ever higher performance and particularly in applications such as: data-warehousing, decision support, On-Line Analytical Processing (OLAP) and more generally DBMS. A linear speed-up is the main objective of parallel algorithms. However, in real applications, it's not obvious to reach this objective due to the high communication cost in parallel and distributed systems and to the possible skew in the charge of different processors. In addition, on heterogeneous multi-user architectures, the load of each processor may highly vary in a dynamic and unpredictable way. In this thesis, we are interested in treating the join and multi-join queries on distributed multi-user heteregeneous systems, grid systems and distributed file systems. We have proposed several algorithms based on using distributed histograms. These algorithms are based on a dynamic data distribution and task allocation which makes them insensitive to data skew and ensure perfect balancing properties during all stages of join computation even on heteregeneous multi-user environment. The complexity analysis of our algorithms and the experimental results show that they have a near-linear speedup.L'Ă©mergence des applications de bases de donnĂ©es dans les domaines tels que le data warehousing, le data mining et l'aide Ă la dĂ©cision qui font gĂ©nĂ©ralement appel Ă de trĂšs grands volumes de donnĂ©es rend la parallĂ©lisation des algorithmes des jointures nĂ©cessaire pour avoir un temps de rĂ©ponse acceptable. Une accĂ©lĂ©ration linĂ©aire est l'objectif principal des algorithmes parallĂšles, cependant dans les applications rĂ©elles, elle est difficilement atteignable : ceci est dĂ» gĂ©nĂ©ralement d'une part aux coĂ»ts de communications inhĂ©rents aux systĂšmes multi-processeur et d'autre part au dĂ©sĂ©quilibre des charges des diffĂ©rents processeurs. En plus, dans un environnement hĂ©tĂ©rogĂšne multi-utilisateur, la charge des diffĂ©rents processeurs peut varier de maniĂšre dynamique et imprĂ©visible. Dans le cadre de cette thĂšse, nous nous intĂ©ressons au traitement de la jointure et de la multi-jointure sur les architectures distribuĂ©es hĂ©tĂ©rogĂšnes, les grilles de calcul et les systĂšmes de fichiers distribuĂ©s. Nous avons proposĂ© une variĂ©tĂ© d'algorithmes, basĂ©s sur l'utilisation des histogrammes distribuĂ©s, pour traiter de maniĂšre efficace le dĂ©sĂ©quilibre des donnĂ©es, tout en garantissant un Ă©quilibrage presque parfait de la charge des diffĂ©rents processeurs mĂȘme dans un environnement hĂ©tĂ©rogĂšne et multi-utilisateur. Ces algorithmes sont basĂ©s sur une approche dynamique de redistribution des donnĂ©es permettant de rĂ©duire les coĂ»ts de communication Ă un minimum tout en traitant de maniĂšre trĂšs efficace le problĂšme de dĂ©sĂ©quilibre des valeurs de l'attribut de jointure. L'analyse de complexitĂ© de nos algorithmes et les rĂ©sultats expĂ©rimentaux obtenus montrent que ces algorithmes possĂšdent une accĂ©lĂ©ration presque linĂ©aire
Parallelism and load balancing in the treatment of the join on distributed architectures
LâĂ©mergence des applications de bases de donnĂ©es dans les domaines tels que le data warehousing,le data mining et lâaide Ă la dĂ©cision qui font gĂ©nĂ©ralement appel Ă de trĂšs grands volumes de donnĂ©esrend la parallĂ©lisation des algorithmes des jointures nĂ©cessaire pour avoir un temps de rĂ©ponse acceptable.Une accĂ©lĂ©ration linĂ©aire est lâobjectif principal des algorithmes parallĂšles, cependant dans les applicationsrĂ©elles, elle est difficilement atteignable : ceci est dĂ» gĂ©nĂ©ralement dâune part aux coĂ»ts de communicationsinhĂ©rents aux systĂšmes multi-processeurs et dâautre part au dĂ©sĂ©quilibre des charges des diffĂ©rents processeurs.En plus, dans un environnement hĂ©tĂ©rogĂšne multi-utilisateur, la charge des diffĂ©rents processeurspeut varier de maniĂšre dynamique et imprĂ©visible.Dans le cadre de cette thĂšse, nous nous intĂ©ressons au traitement de la jointure et de la multi-jointure surles architectures distribuĂ©es hĂ©tĂ©rogĂšnes, les grilles de calcul et les systĂšmes de fichiers distribuĂ©s. Nousavons proposĂ© une variĂ©tĂ© dâalgorithmes, basĂ©s sur lâutilisation des histogrammes distribuĂ©s, pour traiterde maniĂšre efficace le dĂ©sĂ©quilibre des donnĂ©es, tout en garantissant un Ă©quilibrage presque parfait dela charge des diffĂ©rents processeurs mĂȘme dans un environnement hĂ©tĂ©rogĂšne et multi-utilisateur. Cesalgorithmes sont basĂ©s sur une approche dynamique de redistribution des donnĂ©es permettant de rĂ©duire lescoĂ»ts de communication Ă un minimum tout en traitant de maniĂšre trĂšs efficace le problĂšme de dĂ©sĂ©quilibredes valeurs de lâattribut de jointure.Lâanalyse de complexitĂ© de nos algorithmes et les rĂ©sultats expĂ©rimentaux obtenus montrent que cesalgorithmes possĂšdent une accĂ©lĂ©ration presque linĂ©aire.The appeal of parallel processing becomes very strong in applications which require ever higher performanceand particularly in applications such as : data-warehousing, decision support, On-Line Analytical Processing(OLAP) and more generally DBMS. A linear speed-up is the main objective of parallel algorithms. However,in real applications, itâs not obvious to reach this objective due to the high communication cost in parallel anddistributed systems and to the possible skew in the charge of different processors. In addition, on heterogeneousmulti-user architectures, the load of each processor may highly vary in a dynamic and unpredictableway.In this thesis, we are interested in treating the join and multi-join queries on distributed multi-user heteregeneoussystems, grid systems and distributed file systems. We have proposed several algorithms based onusing distributed histograms. These algorithms are based on a dynamic data distribution and task allocationwhich makes them insensitive to data skew and ensure perfect balancing properties during all stages of joincomputation even on heteregeneous multi-user environment. The complexity analysis of our algorithms andthe experimental results show that they have a near-linear speedup
Parallélisme et équilibrage de charges dans le traitement de la jointure sur des architectures distribuées
L Ă©mergence des applications de bases de donnĂ©es dans les domaines tels que le data warehousing,le data mining et l aide Ă la dĂ©cision qui font gĂ©nĂ©ralement appel Ă de trĂšs grands volumes de donnĂ©esrend la parallĂ©lisation des algorithmes des jointures nĂ©cessaire pour avoir un temps de rĂ©ponse acceptable.Une accĂ©lĂ©ration linĂ©aire est l objectif principal des algorithmes parallĂšles, cependant dans les applicationsrĂ©elles, elle est difficilement atteignable : ceci est dĂ» gĂ©nĂ©ralement d une part aux coĂ»ts de communicationsinhĂ©rents aux systĂšmes multi-processeurs et d autre part au dĂ©sĂ©quilibre des charges des diffĂ©rents processeurs.En plus, dans un environnement hĂ©tĂ©rogĂšne multi-utilisateur, la charge des diffĂ©rents processeurspeut varier de maniĂšre dynamique et imprĂ©visible.Dans le cadre de cette thĂšse, nous nous intĂ©ressons au traitement de la jointure et de la multi-jointure surles architectures distribuĂ©es hĂ©tĂ©rogĂšnes, les grilles de calcul et les systĂšmes de fichiers distribuĂ©s. Nousavons proposĂ© une variĂ©tĂ© d algorithmes, basĂ©s sur l utilisation des histogrammes distribuĂ©s, pour traiterde maniĂšre efficace le dĂ©sĂ©quilibre des donnĂ©es, tout en garantissant un Ă©quilibrage presque parfait dela charge des diffĂ©rents processeurs mĂȘme dans un environnement hĂ©tĂ©rogĂšne et multi-utilisateur. Cesalgorithmes sont basĂ©s sur une approche dynamique de redistribution des donnĂ©es permettant de rĂ©duire lescoĂ»ts de communication Ă un minimum tout en traitant de maniĂšre trĂšs efficace le problĂšme de dĂ©sĂ©quilibredes valeurs de l attribut de jointure.L analyse de complexitĂ© de nos algorithmes et les rĂ©sultats expĂ©rimentaux obtenus montrent que cesalgorithmes possĂšdent une accĂ©lĂ©ration presque linĂ©aire.The appeal of parallel processing becomes very strong in applications which require ever higher performanceand particularly in applications such as : data-warehousing, decision support, On-Line Analytical Processing(OLAP) and more generally DBMS. A linear speed-up is the main objective of parallel algorithms. However,in real applications, it s not obvious to reach this objective due to the high communication cost in parallel anddistributed systems and to the possible skew in the charge of different processors. In addition, on heterogeneousmulti-user architectures, the load of each processor may highly vary in a dynamic and unpredictableway.In this thesis, we are interested in treating the join and multi-join queries on distributed multi-user heteregeneoussystems, grid systems and distributed file systems. We have proposed several algorithms based onusing distributed histograms. These algorithms are based on a dynamic data distribution and task allocationwhich makes them insensitive to data skew and ensure perfect balancing properties during all stages of joincomputation even on heteregeneous multi-user environment. The complexity analysis of our algorithms andthe experimental results show that they have a near-linear speedup.ORLEANS-SCD-Bib. electronique (452349901) / SudocSudocFranceF
Handling limits of high degree vertices in graph processing using MapReduce and Pregel
Even if Pregel scales better than MapReduce in graph processing by reducing iteration's disk I/O, while offering an easy programming model using " think like vertex " approach, large scale graph processing is still challenging in the presence of high degree vertices: Communication and load imbalance among processing nodes can have disastrous effects on performance. In this paper, we introduce a scalable MapReduce graph partitioning approach for high degree vertices using a master/slave partitioning allowing to balance communication and computation among processing nodes during all the stages of graph processing. Cost analysis and performance tests of this partitioning are given to show the effectiveness and the scalability of this approach in large scale systems
Scalability and Optimisation of GroupBy-Joins in MapReduce Scalability and Optimisation of GroupBy-Joins in MapReduce
For over a decade, MapReduce has become the leading programming model for parallel and massive processing of large volumes of data. This has been driven by the development of many frameworks such as Spark, Pig and Hive, facilitating data analysis on large-scale systems. However, these frameworks still remain vulnerable to communication costs, data skew and tasks imbalance problems. This can have a devastating effect on the performance and on the scalability of these systems, more particularly when treating GroupBy-Join queries of large datasets. In this paper, we present a new GroupBy-Join algorithm allowing to reduce communication costs considerably while avoiding data skew effects. A cost analysis of this algorithm shows that our approach is insensitive to data skew and ensures perfect balancing properties during all stages of GroupBy-Join computation even for highly skewed data. These performances have been confirmed by a series of experimentations