Appendix for Crescenzi, Ward, Li, & Capra, 2021.

Anita Crescenzi, Austin R. Ward, Yuan Li, Rob Capra. 2021. Supporting metacognition during exploratory search with the OrgBox. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’21), July 11–15, 2021, Virtual Event, Canada. ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/3404835.346295

Capra, Rob

Crescenzi, Anita

Li, Yuan

Ward, Austin R.

Carolina Digital Repository

Appendix A: Exploratory Factor Analysis   Anita Crescenzi, Austin R. Ward, Yuan Li, Rob Capra. 2021. Supporting metacognition during exploratory search with the OrgBox. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’21), July 11–15, 2021, Virtual Event, Canada. ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/3404835.3462955  This appendix contains  • the full text of the 13 metacognition items on the post-task questionnaire (also shown in Table 4 in the paper),  • overall descriptive statistics (M, SD, min, max) and correlations between items (Pearson’s r) • results of exploratory factor analysis  • a description of the composite variable creations and descriptive statistics   Metacognition items on Post-task Questionnaire Participants indicated their agreement with the following statements on a 1-7 point scale with labeled endpoints (1=Strongly disagree, 7=Strongly agree).  The OrgDoc/OrgBox helped me to… …keep track of how broadly I was covering the topic. (track broad) ...keep track of how deeply I was covering some aspects of the topic. (track deep) ...organize the information about the topic. (org info)  ...keep track of my progress towards completing the task. (track progress) ...decide when I had met the task goal. (met goal) ...evaluate whether my strategy for the task was working. (evaluate strategy) ...change my approach to the task. (change approach) ...ask myself if I had included important facets of the topic. (important facets)  ...figure out how the information about the topic fits together. (info fits together) ....determine facets of the topic. (determine facets)  ...understand what the task was asking me to do. (understand task) ...develop a plan for approaching the task. (plan task) ...decide where to begin my search. (begin search Descriptive statistics Appendix Table 1. Post-task questionnaire metacognition items: descriptive statistics (Mean, SD, Min, Max), pairwise correlations (Pearson’s r with uncorrected statistical significance. ‡ p<.001, † p<.01, * p<.05), and composite variable (after exploratory factor analysis)  Item  Descriptive Statistics  Correlations  Composite # Name  M SD Min Max  (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13)  Name (1) track broad  5.23 1.26 2 7  1              monitor (2) track deep  4.96 1.35 2 7  † .44 1             monitor (3) org info  5.58 1.20 2 7  ‡ .53 ‡ .46 1            monitor (4) track progress  5.08 1.61 1 7  * .30 † .37 .24 1           evaluate (5) met goal  4.42 1.47 1 7  .20 † .46 .11 ‡ .61 1          evaluate (6) eval. strategy  4.73 1.30 1 7  .12 * .34 * .34 † .38 ‡ .47 1         evaluate (7) change approach  4.31 1.49 1 7  .19 .14 .18 .27 * .34 ‡ .63 1        evaluate (8) important facets  5.15 1.37 2 7  † .38 * .34 * .32 ‡ .52 * .34 ‡ .51 † .44 1       evaluate (9) info fits together  4.85 1.53 2 7  † .44 * .33 † .41 † .42 .27 .25 * .32 † .38 1      understand topic (10) determine facets  4.79 1.40 2 7  † .38 .25 † .45 † .38 .21 .27 .19 ‡ .54 ‡ .75 1     understand topic (11) understand task  3.83 1.55 1 7  .18 .04 .10 * .36 .25 .17 .24 .22 ‡ .53 ‡ .50 1    plan (12) plan task  4.65 1.54 1 7  .15 .26 .05 0.27 .25 † .42 * .36 † .44 † .44 † .37 ‡ .48 1   plan (13) begin search  3.31 1.52 1 6  .08 .03 .05 † .37 * .33 .25 .18 * .31 .26 .28 ‡ .54 * .37 1  plan     Exploratory factor analysis Before exploratory factor analysis, two tests were run to ascertain whether or not it was advisable to pursue factor analysis: Barlett’s test of sphericity and the Kaiser-Meyer-Olkin Measure of Sampling Adequacy.  Barlett’s test of sphericity. χ2(78)=265.57, p<.001 H0 items are not correlated.  Kaiser-Meyer-Olkin Measure of Sampling Adequacy. KMO=.729  Then exploratory factor analysis was conducted using principal factor with a pre-specified three factor solution.  Oblique, promax rotation to allow for correlated factors. Pattern matrix of promax-rotated loadings Structural matrix of unrotated loadings after principal factors extraction.  Requirements for item retention: (1) loading >= .4, (2) no crossloadings > .32, (3) communality >=.4  Factor loadings > .4 are shaded.  Appendix Table 2: Pattern matrices after principal factors extraction Item  Rotated loadings    Unrotated loadings Name  F1 F2 F3  Communality  F1 F2 F3 track broad  0.00 0.00 0.68  0.51  0.50 0.44 -0.10 track deep  0.34 -0.23 0.57  0.59  0.51 0.39 0.24 organize info  0.02 -0.13 0.77  0.61  0.49 0.55 -0.03 track progress  0.46 0.24 0.16  0.61  0.66 -0.07 0.13 met goal  0.65 0.06 0.03  0.62  0.57 -0.13 0.35 evaluate strategy  0.78 -0.02 0.04  0.71  0.62 -0.12 0.47 change approach  0.61 0.10 -0.04  0.59  0.53 -0.18 0.32 important facets  0.46 0.18 0.27  0.68  0.71 0.02 0.15 info fits together  -0.06 0.57 0.46  0.76  0.73 0.10 -0.39 determine facets  -0.12 0.58 0.49  0.78  0.71 0.13 -0.44 understand task  0.00 0.82 -0.11  0.65  0.55 -0.38 -0.41 plan task  0.33 0.51 -0.10  0.57  0.58 -0.32 -0.06 begin search  0.24 0.57 -0.23  0.46  0.45 -0.41 -0.12            Variance/Eigenvalues  3.18 3.14 3.03    4.55 1.16 1.08 Variance explained  0.44 0.43 0.42    0.62 0.16 0.15     Appendix Table 3: Correlation matrix of promax rotated factors  F1 F2 F3 F1 1   F2 0.36 1  F3 0.37 0.40 1     Composite variables  Composite variables were created by averaging the values of the items in the factors.  Monitor = (track broad + track deep + organize info ) / 3  Evaluate = (track progress + met goal + evaluate strategy + change approach + important facets) / 5  Plan = (understand task + plan task + begin search) / 3  Understand topic = (info fits together + determine facets) / 2   Appendix Table 4: Descriptive statististics (Mean, SD, min, max, skew, kurtosis), and correlation matrix for composite variables. Note: kurtosis = 3 for normal distribution Item  Descriptive statistics  Correlations # Name  M SD Min Max Skew Kurtosis  (1) (2) (3) (4) (1) Monitor  5.26 1.03 3 7 -.19 2.14  1.00    (2) Evaluate  4.74 1.08 1.4 6.8 -.55 3.77  † .45 1.00   (3) Understand topic  4,82 1.37 2 7 -.19 2.43  ‡ .50 ‡ .46 1.00  (4) Plan  3.93 1.23 1 6.7 -.28 3.05   .16 ‡ .49 ‡ .53 1    

Supporting metacognition during exploratory search with the OrgBox

https://cdr.lib.unc.edu/downloads/t148fr94g

Supporting metacognition during exploratory search with the OrgBox

Abstract

Similar works

Full text

Available Versions

Carolina Digital Repository